When updating a device with a new docker-compose.yml through the TorizonOTA WebUI, we would like to reference container images from a private container registry.
The way we tried to solve this, is to write a systemd service executing a login to our registry. Once the service was executed (e. g. on boot, or when the docker service has been started), pulling manually directly from the device works with every user the service was executed for.
Whenever we reference images from private registries in a docker-compose file, initiating an update through the WebUI seems to fail after being in the “updating” state for a while and just reverts to the prior docker-compose.yml.
Question
Which user executes the docker-compose up when an update was initiated through the TorizonOTA WebUI?
OR
What is the suggested approach to enable pulling images from private container registries for updates through the TorizonOTA WebUI?
Thank you for reporting this issue. It seems we have overlooked private registry compatibility with docker-compose updates. However this is now on our radar.
In the meantime I’ve come up with a sort of workaround that should suffice until we have a more long term solution planned out.
So in short docker-compose up is ran by systemd so the user here is root unless specified differently. When a docker-compose update is pushed by OTA the compose file is written onto the system by aktualizr, as a tmp file. Then systemd runs checks on the tmp file. If this process fails then the tmp file is removed from the system and the previous docker-compose.yml is retained. This is intended as part of our rollback feature. I assume this is what you’re experiencing since the check fails with docker not being able to access your private registry.
As for the workaround we need to perform a docker login at some point so that your credentials can be used. However we need to do docker login as the root user since that is who is performing the docker-compose up. There’s two options here:
First option is to docker login once as root, this can be done as follows:
1: run sudo -i on the device to switch to root user
2: once you’re root just run docker login this will create a file at /home/root/.docker/config.json with your credentials that are read by docker.
Second option is the systemd approach you were trying:
1: This was the systemd service I created that seems to work:
It’s important to remember to run the above service at least once before you perform a docker-compose update. Also make sure the config.json is created from docker login at /home/root/.docker/config.json otherwise docker won’t be able to find it.
Try this workaround and let me know how it works out for you. On my setup I was able to pull an image from a private registry and my resulting docker-compose.yml was properly replaced by the update. If you’re still having difficult let me know and also provide the output of sudo journalctl -u aktualizr. The logs from aktualizr should let us know what is happening during the update process.
I have tried all your suggested approaches but none of them seem to work. I have executed the following steps to get the logs:
Stopped Aktualizr with sudo systemctl stop aktualizr
Started Aktualizr manually with sudo aktualizr --loglevel 1 for more verbose logs
Initiated a custom package update via the TorizonOTA WebUI
Copied the log output here: (sorry, it doesn’t want to format it as code…)
user:~$ sudo systemctl stop aktualizr
Password:
user:~$ sudo aktualizr --loglevel 1
Aktualizr version 2020.10-0-g1255aa24f starting
Reading config: “/usr/lib/sota/conf.d/20-sota-device-cred.toml”
Reading config: “/usr/lib/sota/conf.d/30-rollback.toml”
Reading config: “/usr/lib/sota/conf.d/40-hardware-id.toml”
Reading config: “/usr/lib/sota/conf.d/50-docker-compose.toml”
Reading config: “/usr/lib/sota/conf.d/60-polling-interval.toml”
Current directory: /var/rootdirs/home/torizon
Use existing SQL storage: “/var/sota/sql.db”
Initializing docker-compose Secondaries…
Adding Secondary with ECU serial: 60284352aebc54f1b755a28ec8ef0fe576891c4680d4458f99e22cb930cae4e6 with hardware ID: docker-compose
Checking if device is provisioned…
All ECUs are already registered with the server.
Primary ECU serial: 28030e0e70f3ffb44740743007af2c331d77bc2771903bca13582f3454fd27de with hardware ID: apalis-imx8
Device ID: 832bae7f-0a1d-4297-8ff7-b3a2ce6cff28
Device Gateway URL: https://ota-ce.torizon.io:8443
Certificate subject: CN=832bae7f-0a1d-4297-8ff7-b3a2ce6cff28
Certificate issuer: CN=ota-devices-CA
Certificate valid from: Jan 26 12:59:55 2021 GMT until: Jan 3 12:59:55 2120 GMT
… provisioned OK
No pending updates, continuing with initialization
Reporting default hardware information
Reporting libaktualizr configuration
Event: SendDeviceDataComplete
Current versions in storage and reported by OSTree do not match
No installation result to report in manifest
GET https://ota-ce.torizon.io:8443/director/2.root.json
GET https://ota-ce.torizon.io:8443/director/targets.json
New updates found in Director metadata. Checking Image repo metadata…
GET https://ota-ce.torizon.io:8443/repo/2.root.json
GET https://ota-ce.torizon.io:8443/repo/timestamp.json
Skipping Image repo Snapshot download; stored version is still current.
Skipping Image repo Targets download; stored version is still current.
1 new update found in both Director and Image repo metadata.
Event: UpdateCheckComplete
Update available. Acquiring the update lock…
New updates found in stored Director metadata. Checking stored Image repo metadata…
File test02_925fdc5e12e841debf091cc2ec147024-925fdc5e12e841debf091cc2ec147024 was found in the database, but is incomplete.
Continuing incomplete download of file test02_925fdc5e12e841debf091cc2ec147024-925fdc5e12e841debf091cc2ec147024
Event: DownloadProgressReport
The image server doesn’t support byte range requests, try to download the image from the beginning: https://ota-ce.torizon.io:8443/repo/targets/test02_925fdc5e12e841debf091cc2ec147024-925fdc5e12e841debf091cc2ec147024
Event: DownloadProgressReport
Error while downloading a target: Could not download file, error:
File test02_925fdc5e12e841debf091cc2ec147024-925fdc5e12e841debf091cc2ec147024 was found in the database, but is incomplete.
Continuing incomplete download of file test02_925fdc5e12e841debf091cc2ec147024-925fdc5e12e841debf091cc2ec147024
Event: DownloadProgressReport
The image server doesn’t support byte range requests, try to download the image from the beginning: https://ota-ce.torizon.io:8443/repo/targets/test02_925fdc5e12e841debf091cc2ec147024-925fdc5e12e841debf091cc2ec147024
Event: DownloadProgressReport
Error while downloading a target: Could not download file, error:
File test02_925fdc5e12e841debf091cc2ec147024-925fdc5e12e841debf091cc2ec147024 was found in the database, but is incomplete.
Continuing incomplete download of file test02_925fdc5e12e841debf091cc2ec147024-925fdc5e12e841debf091cc2ec147024
Event: DownloadProgressReport
The image server doesn’t support byte range requests, try to download the image from the beginning: https://ota-ce.torizon.io:8443/repo/targets/test02_925fdc5e12e841debf091cc2ec147024-925fdc5e12e841debf091cc2ec147024
Event: DownloadProgressReport
Error while downloading a target: Could not download file, error:
Download unsuccessful after 3 attempts.
Error downloading image: The target’s calculated hash did not match the hash in the metadata.
Event: DownloadTargetComplete
0 of 1 targets were successfully downloaded.
Event: AllDownloadsComplete
Current versions in storage and reported by OSTree do not match
Event: PutManifestComplete
Current versions in storage and reported by OSTree do not match
No installation result to report in manifest
GET https://ota-ce.torizon.io:8443/director/2.root.json
GET https://ota-ce.torizon.io:8443/director/targets.json
No new updates found in Uptane metadata.
Event: UpdateCheckComplete
Current versions in storage and reported by OSTree do not match
No installation result to report in manifest
GET https://ota-ce.torizon.io:8443/director/2.root.json
GET https://ota-ce.torizon.io:8443/director/targets.json
No new updates found in Uptane metadata.
Event: UpdateCheckComplete
INFO: It seems like it is trying to update the package test02_925fdc5e12e841debf091cc2ec147024-925fdc5e12e841debf091cc2ec147024 which is indeed a package I uploaded a few days ago, for testing purposes BUT NOT the package I’m trying to update the device with. This may be part of the problem. Sadly, I do not see an option to delete custom packages in the TorizonOTA WebUI.
Then I pushed another package via the WebUI (just a mongo container from docker hub in the docker-compose file)
Image repo Snapshot verification failed: Snapshot metadata hash verification failed
GET https://ota-ce.torizon.io:8443/repo/snapshot.json
Signature verification for Image repo Targets metadata failed
Image repo Target verification failed: Hash metadata mismatch
GET https://ota-ce.torizon.io:8443/repo/targets.json
1 new update found in both Director and Image repo metadata.
Event: UpdateCheckComplete
Update available. Acquiring the update lock...
New updates found in stored Director metadata. Checking stored Image repo metadata...
File just_mongo-46eaee8298614f80100eff2cb537d1f3 with expected hash not found in the database.
Initiating download of file just_mongo-46eaee8298614f80100eff2cb537d1f3
Error while downloading a target: The target's size was greater than the size in the metadata.
File just_mongo-46eaee8298614f80100eff2cb537d1f3 was found in the database, but is incomplete.
Continuing incomplete download of file just_mongo-46eaee8298614f80100eff2cb537d1f3
Error while downloading a target: The target's size was greater than the size in the metadata.
File just_mongo-46eaee8298614f80100eff2cb537d1f3 was found in the database, but is incomplete.
Continuing incomplete download of file just_mongo-46eaee8298614f80100eff2cb537d1f3
Error while downloading a target: The target's size was greater than the size in the metadata.
Download unsuccessful after 3 attempts.
Error downloading image: The target's calculated hash did not match the hash in the metadata.
Event: DownloadTargetComplete
0 of 1 targets were successfully downloaded.
Event: AllDownloadsComplete
Current versions in storage and reported by OSTree do not match
Event: PutManifestComplete
Now this time, it picked the right package, but still ran into errors.
Other than the initial private registry issue, there seems to be another issue here according to your provided logs.
Actually there may be two issues. First as you said it seems the system is “confused” about what update to pull from the server and is attempting to pull the wrong one. The other issue is once you got the system to pull the right update there was a mismatch in the metadata somehow. In any case it seems like the metadata was messed up on some way or another.
We’re going to investigate this internally. But I would appreciate if you could try and reproduce this from your side. If you’re able to reproduce this and provide reliable steps to do so, it would help in our investigation.
I have reproduced it with 2 different Apalis IMX8QM boards.
These are the steps I executed:
Install VERSION=“5.1.0-devel-20210110+build.173 (dunfell)” on each edge device
Provision edge device according to the TorizonOTA WebUI (works)
Set up systemd service to login to private docker registry (works → can pull images manually when on device, for every logged in user; e.g. root/torizon)
Upload custom package to TorizonOTA WebUI
Initiate update of custom package via TorizonOTA WebUI
That’s it.
I guess prerequisites are to have a lot of random custom packages uploaded to the OTA server. I uploaded a lot and experimented with different naming schemes, docker compose versions, version numbers for the custom packages etc. so it’s very possible something got somehow twisted that way.
Thank you for the additional information, this will help with our testing. The team has been working on a fix that should address this. We’ll update you when this fix is available publicly.
