# What shall I do if I get all results with N/A after training?

## Check if tag setting to upload assets/ labeling to assets were correctly done

See explanations of following chapters in the users manual.
- [Users manual -> Set tags](../usermanual/index_2.html#set-tags)
 
    -> **If any tag is missing to upload assets?**

- [Users manual -> Label assets](../usermanual/index_3.html#label-assets)

    -> **If independent assets are labeled together?**

## For time-series/vibration datasets: Check if CSV file format is supported

MANUFACIA may not support your CSV file format, the one without csv header row or partly missing entry. You may compare your file with the one delivered as test data or it is recommended to use [CSV format checker](../csv_format_checker/index_1.html#what-is-csv-format-checker).

## Check if memory ran out when MANUFACIA starts

Run ```free``` in a terminal.

## Check if other applications are using GPU

Check GPU_RAM consumption. Run ```nvidia-smi``` in a terminal.

## Check if the access to MANUFACIA server is via proxy server

Ask a person in your company responsible for IT issues. MANUFACIA does not support the access via proxy server. Retry after switching off proxy server setting.

## Check Nvidia driver version

MANUFACIA may not work with the Nvidia driver installed in the environment. Please refer to [Install manual -> Recommended system requirements](../installation_manual_gpu/index_1.html#recommended-system-requirements) to confirm the version.

For all other cases, please zip log files generated by ```manufacia-ctrl logs --all``` and send it to our support. Log files are saved in the folder,

>var/log/supervisor

# What shall I do if MANUFACIA will not start by manufacia-ctrl start command?

Check first where error occurs in the console.

a. After "Starting containers..."<br>
+ docker-compose_up.log

b. After "Initializing database..."<br>
c. After "This Host is Linux System, tuning start ... "
+ docker-stop.log
+ docker-update.log
+ docker-start.log

Find lines with ERROR in the log file above under the directory .manufacia/versions/x.x.x-gpu/bin/error_logs/

## Check if the MANUFACIA executable path is correctly set

```
$ export PATH=$HOME/.manufacia/bin:$PATH
```

## Check if host setting is correct

Run ```hostname -I``` in a terminal and check if the IP address has been set as below.

```
$ export THIS_HOST=192.168.100.1
```

## Check if there are error messages in log files

```
$ manufacia-ctrl logs --all
```

## Check driver/library versions are supported

Please refer to [Recommended system requirements](../installation_manual_gpu/index_1.html#recommended-system-requirements) to confirm which version of drivers or libraries are supported　by MANUFACIA.


# Assets cannot be partly/fully uploaded. What shall I do？

In the case that dataset files cannot be partly/fully uploaded after pressing the button [Add files], although the thumbnails or icons were properly displayed in the preview of assets registration, the storage can have a problem to read data.

Run the following command in the terminal,
```
$ docker logs manufacia_db_1
```

then check if you can find a line like the one below.

```
 :
 :
2021-07-28 03:22:30.426 UTC [5628] ERROR:  could not read block 1 in file "base/16384/16506": read only 0 of 8192 bytes
 :
 :
```

If you could find the read error, it means that the storage has some problems and some parts of the disk cannot be used. Stop using the server, back up the data and then restore it after replacing the storage with the new one.


(disk-full-what-to-do)=
# Local disk is full. Which files may I delete?

If you continue to use the application, the local disk may become full.

```
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 8189636 0 8189636 0% /dev
tmpfs 1642520 1188 1641332 1% /run
/dev/sda1 522541784 514764840 7776944 99% /
...
```

Disk is full if the root directory (/) Use% is 100. At this time, it is possible to free disk space to some extent by deleting the temporary file in the application container.
You may delete the following two types of files.

- Cache files created for preprocessing
- Application log files

## Deleting the cache files

The cache files created by the application are stored in the path /root/.cache/draupnir in the application container.

```
$ docker exec -it manufacia_app_1 /bin/bash
app$ cd /root/.cache/draupnir
app$ ls
168ea5becaff058a130761e029e512dafdb733fba3bf3a4a477e426a5b892cb2
168ea5becaff058a130761e029e512dafdb733fba3bf3a4a477e426a5b892cb2-meta
c9d1526a96cca5656842b43b3bead16fd4857f3fbc9d3b287d1a8c48092dbd69
c9d1526a96cca5656842b43b3bead16fd4857f3fbc9d3b287d1a8c48092dbd69-meta
...
```

Since these files are automatically created again if necessary, there is no problem in removing them.

## Removing the application's log files

The application's server log files are stored in the application container.

```
$ docker exec -it manufacia_app_1 /bin/bash
```

It is stored in the following directory here.

- /root/rb/log
- /var/log/supervisor
- supervisord.log
- web.log
- worker.log

```
app$ cd /root/rb/log
app$ rm *
```

While there is no problem in removing log files, it should be noted that, once these files are removed, it is no longer possible to see the previous log data.

# How can I check if application containers are running?

If the app container is restarted, the ssh may not start automatically.

Look into the application containers, and check whether these processes are running. Results like those shown below will be displayed if all the expected processes are running. 

```
$ docker exec -it manufacia_app_1 supervisorctl status
sshd RUNNING pid 804, uptime 0:03:02
web RUNNING pid 803, uptime 0:03:02
worker RUNNING pid 805, uptime 0:03:02
```

If **sshd** is not running, you will see results like below.

```
$ docker exec -it manufacia_app_1 supervisorctl status
sshd STOPPED Jul 17 03:23 PM
web RUNNING pid 803, uptime 0:05:25
worker RUNNING pid 805, uptime 0:05:25
```

**To start sshd**, execute the following command.

```
$ docker exec -it manufacia_app_1 supervisorctl restart sshd
```

Check the processes. There is no problem if you see results like below.

```
$ docker exec -it manufacia_app_1 supervisorctl status
sshd RUNNING pid 803, uptime 0:22:46
web RUNNING pid 802, uptime 0:22:46
worker RUNNING pid 804, uptime 0:22:46
```

# How to start MANUFACIA after a server reboot?

After the server reboot, the docker container is in a stopped state. It is necessary to start it.

To start a docker container, execute the following command (use the following command as it is because it specifies the order of the container's startup):

```
$ docker start manufacia_db_1 manufacia_redis_1 manufacia_jarngreipr_1 manufacia_broker_1 manufacia_frontend_1 manufacia_app_1
```

After starting the docker container, check the application for startup.

If you do not have access to MANUFACIA, you may need to contact us because the IP address may have changed.

# How to get and restore backups?

Procedure to get and restore backups.

The assumption to restore is, MANUFACIA major version is the same. If it is different, in principle it cannot be　restored.

Cases:
| Restore Availability | Version at the time of backup | Version at restore time |
| :------------------- | :---------------------------- | :---------------------- |
| Restorable           | MANUFACIA_v2.0                | MANUFACIA_v2.1          |
| Cannot be restored   | MANUFACIA_v2.0                | MANUFACIA_v3.0          |

The ability to retrieve and restore backups was added from MANUFACIA_v2.0. If you are using MANUFACIA_v1.x and want to use the backup acquisition and restore features, you must uninstall MANUFACIA_v1.x and then install version MANUFACIA_v2.0 or higher.

## Get a backup

After switching to root with ```sudo su -```, confirm what is being backed up.

```console
# cd /var/lib/docker/volumes/
# ls
32a1abc7f27d171f41cadebb6853628f057a9f7fb6ca5a8374588a88eb125b4f
4a932f10b366e924a3f5639dd772f6978ea3f181ad2468801e55ba067ce197aa
cdb73458638f40c497397ea38eb404d34309cbbba9a17ecb95994d19e3d1e37e
manufacia_app_storage
370dfb363cdb4d59a9340498575a7dbd355c4115f639ce07960740e662e77c99
cc48cb9062e364259d803487b1c2298fee62b22d0f8b08eb255546bfc50d3a00
ede73affe0e762d339273231ae0bb04d5097b019fe3514a9bd587cb2fd3de16b
manufacia_ps_data
```

Create a directory where you want to store backup data.

As an example, create a /root/backup directory.

```console
# mkdir /root/backup
```

Create a backup.

```console
# cd /var/lib/docker/volumes/
# cp -rp manufacia_app_storage manufacia_ps_data /root/backup
```

Verify that the backup has been created.

```console
# ls /root/backup
manufacia_app_storage manufacia_ps_data
```

## Restore

Restoration requires the following conditions.

- There must be a backup file to be restored. (For example, it must be stored under /root/backup.)
- Delete or move the "manufacia_app_storage" and "manufacia_ps_data" folders under /var/lib/docker/volumes (move them if you want to keep them as backups).

Copy the backup file to /var/lib/docker/volumes.

```console
# cd /root/backup
# ls
manufacia_app_storage manufacia_ps_data
# cp -rp manufacia_app_storage manufacia_ps_data /var/lib/docker/volumes/
```

Remove docker containers.

```console
# docker ps -aq | xargs docker rm -f
9daa4c1b44a8
5aa2c20db00f
f0c4c7424380
144b976334d6
888b72a7fd8f
7858f80a8926
```

Start MANUFACIA.

```console
# manufacia-ctrl start
Starting containers...
Starting containers...
Creating manufacia_redis_1 ... done
Creating manufacia_frontend_1 ... done
Creating manufacia_broker_1 ... done
Creating manufacia_db_1 ... done
Creating manufacia_app_1 ... done
Creating manufacia_jarngreipr_1 ... done
Initializing container infos...
Initializing core...
Initializing database...
manufacia-0.10.0.1.3 ... ok
manufacia-0.10.0.1.3 ... ok
Cleaning up...
Done.
```

Access MANUFACIA in your browser and verify that you are back to the data at the time of the backup.