# Let's start MANUFACIA 3 ## Evaluate project ### Select AI model 1 Training results of generated AI models will be displayed in a scatter diagram with its X axis for accuracy and Y axis for network size as initial setting. ```{image} media/EvalProject_ScatterDiagram.png :width: 800px :name: img_scatter_diagram ``` Under the scatter diagram, major results of each model will be displayed in a table, **it can be sorted by column**. Clicking a marker(model) in the scatter diagram or a line in the table will show the model detail. **Models whose training was unsuccessful will be noted as "Training failed." and there is no detail result to show.** Whenever any operation was not properly done, for example, **division by zero or the number is infinity, some of the results can be missing in the table and kept blank**. ```{tip} Click axis title to change the axis settings of scatter diagram. ``` ### Select AI model 2 Once having selected a model for model detail, it is possible to browse models by clicking "<>" key on the top left on the both sides of the figure which shows (Model ID / The number of models). If the training failed for the selected model, nothing will be displayed. ```{image} media/Model_Detail_TrainingFailed.png :width: 800px :name: img_model_result_training_failed ``` ```{tip} It takes longer to show the model detail, as the number of validation datasets samples increases. ``` ```{caution} Showing training results of a model from older MANUFACIA version before v2.2, accuracy or threshold/distance shown in the model list and model detail can be different, because threshold/distance was calculated differently from those of v2.2. In such the case, press [automatic] button above the slider below confusion matrix to adjust the threshold/distance to how it is calculated in the v2.2. ``` ### Evaluate AI model ```{image} media/EvalProject_ModelResult.png :width: 800px :name: img_model_result ``` As AI model training detail, following items are displayed. 1. [Performance index of neutral network](#performance-index-neural-network)
2. [Confusion matrix](#confusion-matrix)
3. [Histogram (Unsupervised learning only)](#histogram)
4. [Validation sample list](#validatio-sample-list)
5. [Training loss curve](#training-loss-curve) In the following sections, it will be explained in detail. (performance-index-neural-network)= #### Performance index of neural network ```{image} media/EvalProject_performance_index.png :width: 250px :name: img_performance_index ``` See [Confusion matrix](#confusion-matrix) for the explanation of TP (**T**rue **P**ositive), FP (**F**alse **P**ositive), FN (**F**alse **N**egative), TN (**T**rue **N**egative) in the formulas. ```{tip} Whenever any operation was not properly done, for example, division by zero or the number is infinity, some of the results can be missing and kept blank. ``` - **MCC** >MCC stands for **M**atthews **C**orrelation **C**oefficient. **This index will take values in the rage from -1.0 to 1.0 for binary classification(anomaly detection)** and we can interpret if it is closer to 1.0, then the network has better prediction as same as in the training. If it takes 0.0, then it is as same as random evaluation of normal/anomaly. If it will be close to -1.0, then we can interpret that the network prediction is totally opposite to the training. >The definition of this index for anomaly detection is >$MCC = \frac{TP \times TN – FP \times FN}{\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}}$ >For multi-class classification, it is defined as follows. **For classification with more than two classes, the minimum value is not -1.0. It varies between -1.0 and 0.0 based on the number of classes, while the maximum value is 1.0, unchanged.** >$MCC = \frac{\sum_{k} \sum_{l} \sum_{m} C_{kl} C_{lm} - C_{kl} C_{mk}}{\sqrt{\sum_k (\sum_l C_{kl})(\sum_{k'|k' \neq k} \sum_{l'} C_{k'l'})}\sqrt{\sum_k (\sum_l C_{lk}) (\sum_{k'|k' \neq k} \sum_{l'} C_{l'k'})}}$ ```{note} [Wikipedia page](https://en.wikipedia.org/wiki/Matthews_correlation_coefficient) on Matthews correlation coefficient. ``` - **Accuracy** >It is the accuracy of all predictions and can be calculated by "number of correct predictions / sum of numbers in the matrix". This index will be often used together with recall or precision in the following sections, however, it is quite important that **the network is not necessarily good, even with accuracy value near 100%, especially if the datasets of each label are not well balanced.** Even if the total number of datasets is very big, it is difficult to correctly predict the label with much less number of datasets than others. >$Accuracy = \frac{TP ＋ TN}{TP + FP + FN + TN}$ - **Recall** >The percentage of what was predicted as "normal" from actual "normal" labels. >$Recall = \frac{TP}{TP + FN}$ - **Precision** >The percentage of actual "normal" labels from what was predicted as "normal". >$Precision = \frac{TP}{TP + FP}$ - **F-measure** >Harmonic average of precision and recall. >$F$-$measure = \frac{2}{Recall^{-1} + Precision^{-1}} = \frac{2 \times Precision \times Recall}{Precision + Recall}$ - Training loss >Training loss value displayed here is an average of [training-loss-curve](#training-loss-curve) from training start until training end. - Network size >Network size is the file size of generated network. The more complex the network will be, the bigger it will be and it will take longer to process. (roc-curve)= - ROC curve and AUC >ROC curve is a chart by plotting dividing point of normal and anomaly (cut-off value) and connects them with lines. X axis is false positive rate, Y axis is true positive rate and it will take value from 0.0 to 1.0. AUC is the area defined under the ROC curve, AUC will be bigger for nice models, smaller for bad models. >The highest value of AUC is 1.0, the lowest is 0.0, a model which randomly decides normal from anomaly will have AUC around 0.5. When AUC becomes 1.0, ROC curve will be made by two lines by connecting two points of (0.0, 0.0) and (0.0, 1.0), and then (0.0, 1.0) and (1.0, 1.0), where the coordinates represent (false positive rate, true positive rate). (confusion-matrix)= #### Confusion matrix Confusion Matrix is a table to visualize algorithm performance. In MANUFACIA, **actual labels are in column, predicted labels are in row of the table.** ```{image} media/EvalProject_ConfusionMatrix.png :width: 500px :name: img_confusion_matrix ``` (1) The number of AI's "normal" counts for actual "normal" data. (**T**rue **N**egative = correct）
(2) The number of AI's "normal" counts for actual "anomaly" data. (**F**alse **N**egative = wrong）
(3) The number of AI's "anomaly" counts for actual "normal" data. (**F**alse **P**ositive = wrong）
(4) The number of AI's "anomaly" counts for actual "anomaly" data. (**T**rue **P**ositive = correct） With the slider under confusion matrix, threshold value (anomaly detection, supervised learning) and distance (anomaly detection, unsupervised learning) can be modified. (See also [Histogram](#histogram)). Changing the slider value will change confusion matrix and performance index. The modified value will be used for the model. The default threshold / distance is now calculated as the point on the ROC curve that minimizes the function $FPR^2 + FNR^2$, where FPR stands for **F**alse **P**ositive **R**ate, FNR for **F**alse **N**egative **R**ate. ```{image} media/EvalProject_ConfusionMatrix_Threshold.png :width: 600px :name: img_threshold ``` ```{hint} By pressing [automatic] button above the slider, threshold or distance can be reset to the default value any time. ``` On the right and bottom of the confusion matrix, precision and recall for each predicted label or actual label will be displayed in bar graph. Moreover, the following values will be displayed. - Mean recall ... Average of recall values; normal and anomaly. - Mean precision ... Average of precision values; normal and anomaly. - Mean F-Measure ... F-Measure value by using mean recall and mean precision. ```{caution} For multi-class classification, threshold cannot be modified. ``` (histogram)= #### Histogram Histogram shows distribution of validation samples results and threshold/distance of the labels "normal" and "anomaly" for anomaly detection (supervised/unsupervised learning). For supervised learning, it can be seen how the "normal" and "anomaly" samples have been divided during the training, and for unsupervised learning how far "anomaly" samples are from the center of "normal" samples. Clicking the histogram tab will show the histogram instead of confusion matrix. Above the horizontal line in the middle of the histogram, distribution of validation results of samples with "normal" label is shown, and below the line those of "anomaly" label. As same as confusion matrix, samples with the correct prediction are in green, incorrect ones in red. Automatically calculated threshold or distance value is shown in white dotted line. Move the slider below to adjust the vertical line to separate "normal" (left) and "anomaly" (right) for prediction, while watching the sample distribution. The value after the change will be used for the model. At an ideal model, "normal" sample distribution above the horizontal line and "anomaly" sample distribution below the line have some horizontal distance from each other. By setting threshold or distance between the distributions, samples in the UI will be all green as correctly predicted. SSee also [this figure](./index_1.html#how-to-evaluate-ai-judgement). ```{image} media/EvalProject_Histogram.png :width: 600px :name: img_histogram ``` ```{caution} Histogram is displayed by counting samples within a certain range of threshold/distance value on the scale. The color of the bar can be different from as expected depending on where the automatically calculated threshold/distance value lies on the scale. ``` (validation-sample-list)= #### Validation sample list In this table, prediction results of used validation datasets are shown. Click confusion matrix element will filter the table content. Additionally, by clicking arbitrary line of the table will show visualization of after pre-processing and detected feature in addition to sample file name, predicted label. ```{image} media/EvalProject_Validation_samples.png :width: 600px :name: img_validation_samples ``` - **In case Anomaly detection in supervised learning / multi-class classification** >For anomaly detection in supervised learning and multi-class classification, pre-processed image and feature visualization processed by **SmoothGrad** method will be displayed to tell where the convolutional network (CNN) found specific features in the image. ```{image} media/EvalProject_SmoothGrad.png :width: 600px :name: img_smoothgrad ``` - **In case anomaly detection in unsupervised learning** >For anomaly detection in unsupervised learning, pre-processed image and **reconstructed data** will be displayed. The reconstructed data is far away from the normal data where there is difference from normal label group. ```{image} media/EvalProject_ReconstructedData.png :width: 600px :name: img_reconstructed_data ``` ```{caution} For some models whose training did not proceed well, SmoothGrad can just be black with an error message because of internal operation error. ``` ```{tip} For trainings with time-series or vibration datasets, pre-processed data, SmoothGrad and reconstructed data will be displayed in one line by connecting each selected column in pre-processing one after another from the one in the left. ``` (training-loss-curve)= #### Training loss curve Training loss curve is a chart which shows the trend how the error will be reduced as training proceeds, and the error increases if the training goes too far and overfits. In this chart, **X axis represents the number of iterations(the number of steps)***, Y axis error from the true value. By moving the "Smoothing" slider on the top right of the chart, the curve will be displayed by smoothing top and bottom peaks to show the base line trend. ```{note} The number of iteration will be automatically set from the number of datasets(assets) and [Batch size](./index_3.html#batch). For example, with 1000 datasets and batch size of four, 250 (1000/4) small subsets of dataset will be used in the training. This number 250 is the number of iteration and it means the trainings　with all datasets only once are done. It is defined as 1 epoch of the training. ``` Even if the evaluation of confusion matrix such as MCC or accuracy is good, it is less certain that the model is practicable, if this training loss curve will not continuously get closer to zero and stabilized. Keep your eyes on this curve's trend. ```{image} media/EvalProject_Training_loss_curve.png :width: 600px :name: img_training_loss_curve ``` #### Generate reports Marking favorite models in the model list table and then click [Create a report for selected models] will create report selected models in a new browser tab including model detail, several charts and sample detail. In sample detail UI, besides actual label and predicted label, visualized sample will be displayed: SmoothGrad for supervised learning, reconstructed data for unsupervised learning. Selecting the number of images in [Number of visualized samples] under the button [Create a report for selected models], the number of visualized sample images can be defined. For the model of anomaly detection, the number of samples for following three of matrix elements will be set to 1/3 of the given number in the UI, - Samples with "anomaly" prediction for "anomaly" label. (TN)
- Samples with "anomaly" prediction for "normal" label. (FN)
- Samples with "normal" prediction for "anomaly" label. (FP) and the samples will be selected that match the condition from the sample with smaller ID. If there is still less samples for report after the process above, then the rest will be selected from - Samples with "normal" prediction for "normal" label. (TP) For the multi-class classification, samples will be selected from the ones with wrong predictions up to the given number. ```{image} media/EvalProject_SelectModel_toReport.png :width: 800px :name: img_report_model ``` Report example: ```{image} media/EvalProject_Report_Sample.png :width: 800px :name: img_report_sample ``` ```{hint} Visualized samples are sorted by sample ID. ``` ## Apply AI models In this chapter, necessary steps to infer by using generated AI models trained by MANUFACIA will be explained. ### Deploy AI models Click "Use this model" in the bottom of model detail screen after deciding models to use. The button "Use this model" will not be activated (light blue) if there is no connection to MQTT broker of MANUFACIA server. To establish the connection in this case; - Start AI model tester and then establish communication between MANUFACIA. - Start the sample application of Greenia Embedded SDK, then establish communication between MANUFACIA. In the proceeding screen, list of all devices connected to the server is displayed. Confirm that the target device is in the list and then click "Deploy". ```{image} media/ApplyModel_DeployModel.png :width: 800px :name: img_use_deploy_model ``` ```{warning} The model will be deployed to all devices without cartridge connected to the MANUFACIA server. ``` The status of "installed" means that the model has been installed on the device. Then, the rest will be done on the device. ```{image} media/ApplyModel_ModelInstalled.png :width: 800px :name: img_model_installed ``` ### Test AI models Created AI models by MANUFACIA can be used to infer with newly aquired test data. For further information, see "AI model tester users manual". ### Integrate AI models in a system To use AI models in a real production line/system, a system should be developed with the help of Greenia Embedded SDK or Python API. ### Retrain AI models In MANUFACIA, created AI models can be retrained by adding new assets. For the re-training, pre-processing is fixed and only assets can be changed. To avoid overfitting, **it is not recommended to use assets that were used for the first training**. Therefore, additional assets for training for all classes are necessary. Re-training will be carried out per lab, all the models that are marked as favorite will be retrained. Follow the procedure below to set up retraining. #### Add/remove models to/from favorite Select models to retrain in the model list and click on the star on the left. If it is properly added, then it will be marked with color. Click on the added model will remove it. In the model detail UI, add to / remove from the favorite list can be done on the top left. ```{image} media/ApplyModel_SelectModel_Retrain.png :width: 800px :name: img_select_model_retrain ``` #### Add assets to the lab Add assets to the lab where there is the project to be retrained. See [Resister assets](./index_2.html#register-assets) for how to add assets. Remember to add assets of **all classes**. #### Create a project A new project is necessary to retrain the model. Check [Retrain existing model] and click on [+ Create new]. With this check, the new project will be created with all preprocessing settings being locked. ```{image} media/ApplyModel_NewProject_Retrain.png :width: 600px :name: img_new_project_retrain ``` A list of the projects marked as favorite will be displayed and then click [Selection] on the project to be retrained. ```{image} media/ApplyModel_SelectFavorite_Retrain.png :width: 800px :name: img_select_favorite_retrain ``` The project creation screen appears. Modify the project name which is given as "{Selected Project Name}_Copy" then click [Rename and create]. Clicking [Create with original name], then a project in the original name will be created. ```{image} media/ApplyModel_EditProjectName_Retrain.png :width: 400px :name: img_edit_project_name_retrain ``` #### Add assets The asset selection screen appears. **Prepare assets of all classes that were used in previous time.** The assets that were used last time are already selected, uncheck them first and then select newly prepared assets. If newly added assets do not cover all classes, **it is possible to use the old ones for missing classes. However, it is not recommended**. Click [OK] after selected all assets. ```{image} media/ApplyModel_SelectAsset_Retrain.png :width: 800px :name: img_select_asset_retrain ``` In the label assets screen, **delete all previous labels and newly label assets**. **For the retraining, pre-process setting cannot be modified.** The rest is as same as a usual training setting.