Offline Docker Compose updates do not tag images properly, docker-compose fails after successful update

Ernest · October 14, 2022, 1:19pm

Hi,

I got our update system working and can create functioning docker compose lockboxes and generate an update package from them. When plugging my USB stick into our device, the update starts (as seen from journalctl -f -u aktualizr*) and eventually reboots the system, notifying us that the update succeeded with the message Event: InstallTargetComplete, Result - Success.

The new containers now run properly on the system, until you reboot. After a reboot, the following error messages appear in journalctl:

Oct 14 12:43:45 verdin-imx8mm-07276307 systemd[1]: Starting Docker Compose service with docker compose...
Oct 14 12:43:45 verdin-imx8mm-07276307 systemd[1]: Started Docker Compose service with docker compose.
Oct 14 12:43:45 verdin-imx8mm-07276307 systemd[1]: docker-integrity-checker.service: Main process exited, code=exited, status=1/FAILURE
Oct 14 12:43:45 verdin-imx8mm-07276307 systemd[1]: docker-integrity-checker.service: Failed with result 'exit-code'.
Oct 14 12:43:45 verdin-imx8mm-07276307 systemd[1]: Failed to start Verify Docker images integrity and recover from Docker data corruption.
Oct 14 12:43:48 verdin-imx8mm-07276307 docker-compose[1019]: Pulling mycontainer (mycompany/mycontainer:digest_sha256_04c2e1d462f7e4a57f108a5c758e7f7905bd517797f8f8d439d37d929c73c270)...
Oct 14 12:43:54 verdin-imx8mm-07276307 dockerd[687]: time="2022-10-14T12:43:54.441252375Z" level=info msg="Attempting next endpoint for pull after error: manifest unknown: manifest unknown"
Oct 14 12:43:54 verdin-imx8mm-07276307 docker-compose[1019]: manifest for mycompany/mycontainer :digest_sha256_04c2e1d462f7e4a57f108a5c758e7f7905bd517797f8f8d439d37d929c73c270 not found: manifest unknown: manifest unknown
Oct 14 12:43:54 verdin-imx8mm-07276307 systemd[1]: docker-compose.service: Main process exited, code=exited, status=1/FAILURE
Oct 14 12:43:54 verdin-imx8mm-07276307 systemd[1]: docker-compose.service: Failed with result 'exit-code'.
Oct 14 12:43:54 verdin-imx8mm-07276307 systemd[1]: docker-compose.service: Triggering OnFailure= dependencies.
Oct 14 12:43:54 verdin-imx8mm-07276307 systemd[1]: Starting Verify Docker images integrity and recover from Docker data corruption...
Oct 14 12:43:54 verdin-imx8mm-07276307 docker-integrity-checker.sh[1037]: docker-compose service has failed.
Oct 14 12:43:54 verdin-imx8mm-07276307 docker-integrity-checker.sh[1037]: Docker integrity checker is disabled. Create /etc/docker/enable-integrity-checker file to enable it.
Oct 14 12:43:54 verdin-imx8mm-07276307 docker-integrity-checker.sh[1037]: Restarting docker-compose in 10 seconds...

So seemingly, docker-compose cannot find the image mycompany/mycontainer with the new tag (digest_sha256_04c2e1d462f7e4a57f108a5c758e7f7905bd517797f8f8d439d37d929c73c270 ). This digest tag is correct. It’s no surprise that docker can’t find this image, because when I run docker images, there only exists mycompany/mycontainer:latest and there is no mycompany/mycontainer:digest_sha256_04c2e1d462f7e4a57f108a5c758e7f7905bd517797f8f8d439d37d929c73c270 . I’ve edited all container/repository names in the logs in this post.

Consulting the logs, I see that after aktualizr-torizon rebooted the system during the update, it brings up docker-compose, and my new containers start succesfully, explaining why right after the update everything is fine. But after bringing up the compose file, it runs the command docker system prune -a --force, which untags some containers!

Oct 14 12:28:29 verdin-imx8mm-07276307 aktualizr-torizon[868]: Updating containers via docker-compose
Oct 14 12:28:29 verdin-imx8mm-07276307 aktualizr-torizon[868]: Running docker-compose down
Oct 14 12:28:29 verdin-imx8mm-07276307 aktualizr-torizon[868]: Running command: /usr/bin/docker-compose --file /var/sota/storage/docker-compose/docker-compose.yml -p torizon down
Oct 14 12:28:32 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping torizon_kiosk_1 ...
Oct 14 12:28:32 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping mydevice-hw      ...
Oct 14 12:28:32 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping mydevice-speech  ...
Oct 14 12:28:32 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping mydevice-web     ...
Oct 14 12:28:32 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping mydevice-bt      ...
Oct 14 12:28:32 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping siab            ...
Oct 14 12:28:32 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping mydevice-skippy  ...
Oct 14 12:28:32 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping mydevice-mqtt    ...
Oct 14 12:28:32 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping weston          ...
Oct 14 12:28:34 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping siab            ... done
Oct 14 12:28:34 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping torizon_kiosk_1 ... done
Oct 14 12:28:34 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping mydevice-skippy  ... done
Oct 14 12:28:35 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping weston          ... done
Oct 14 12:28:43 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping mydevice-bt      ... done
Oct 14 12:28:43 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping mydevice-hw      ... done
Oct 14 12:28:45 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping mydevice-speech  ... done
Oct 14 12:28:45 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping mydevice-web     ... done
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Stopping mydevice-mqtt    ... done
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing torizon_kiosk_1 ...
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing mydevice-hw      ...
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing mydevice-speech  ...
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing mydevice-web     ...
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing mydevice-bt      ...
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing siab            ...
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing mydevice-skippy  ...
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing mydevice-mqtt    ...
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing weston          ...
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing mydevice-bt      ... done
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing mydevice-speech  ... done
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing mydevice-skippy  ... done
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing mydevice-web     ... done
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing siab            ... done
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing mydevice-mqtt    ... done
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing torizon_kiosk_1 ... done
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing mydevice-hw      ... done
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing weston          ... done
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[2051]: Removing network torizon_default
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[868]: Running docker-compose up
Oct 14 12:28:46 verdin-imx8mm-07276307 aktualizr-torizon[868]: Running command: /usr/bin/docker-compose --file /var/sota/storage/docker-compose/docker-compose.yml.tmp -p torizon up --detach --re>
Oct 14 12:28:48 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating network "torizon_default" with the default driver
Oct 14 12:28:49 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating mydevice-skippy ...
Oct 14 12:28:49 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating siab           ...
Oct 14 12:28:49 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating mydevice-mqtt   ...
Oct 14 12:28:49 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating weston         ...
Oct 14 12:28:51 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating weston         ... done
Oct 14 12:28:52 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating siab           ... done
Oct 14 12:28:53 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating mydevice-skippy ... done
Oct 14 12:28:53 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating mydevice-mqtt   ... done
Oct 14 12:28:53 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating mydevice-web    ...
Oct 14 12:28:53 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating mydevice-speech ...
Oct 14 12:28:53 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating mydevice-bt     ...
Oct 14 12:28:53 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating mydevice-hw     ...
Oct 14 12:28:55 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating mydevice-bt     ... done
Oct 14 12:28:58 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating mydevice-speech ... done
Oct 14 12:28:58 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating mydevice-hw     ... done
Oct 14 12:28:59 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating mydevice-web    ... done
Oct 14 12:28:59 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating torizon_kiosk_1 ...
Oct 14 12:29:01 verdin-imx8mm-07276307 aktualizr-torizon[2359]: Creating torizon_kiosk_1 ... done
Oct 14 12:29:01 verdin-imx8mm-07276307 aktualizr-torizon[868]: Removing not used containers, networks and images
Oct 14 12:29:01 verdin-imx8mm-07276307 aktualizr-torizon[868]: Running command: docker system prune -a --force
Oct 14 12:29:01 verdin-imx8mm-07276307 aktualizr-torizon[3112]: Deleted Containers:
Oct 14 12:29:01 verdin-imx8mm-07276307 aktualizr-torizon[3112]: c0bdc1753d1278d3d7bb942ca45de06d74b54360a16b715cbd5920ffaea46df8
Oct 14 12:29:01 verdin-imx8mm-07276307 aktualizr-torizon[3112]: 3f83539e0978b78e1df55fc8400d22a65d553b319267006e3f2dd61cc6824724
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[3112]: Deleted Images:
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[3112]: untagged: mycompany/mydevice-web:digest_sha256_cf3d22dc245aa759199fe8de589fdedc4b7fa19c9aca0678d844afd81effb682
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[3112]: untagged: mycompany/mydevice-skippy:digest_sha256_04c2e1d462f7e4a57f108a5c758e7f7905bd517797f8f8d439d37d929c73c270
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[3112]: untagged: mycompany/mydevice-weston:digest_sha256_73aa9327063a85d943cf31471bb123a1b61b7fc4585111cd29f034f74db9026a
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[3112]: untagged: mycompany/mydevice-bt:latest
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[3112]: untagged: mycompany/mydevice-bt@sha256:1d53435366f9ba57cedac4cd62a137aecd608613fcc9730c0df2f52abac33c44
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[3112]: deleted: sha256:8e9d5ceb8c534430580fb224031dd8e81bb75e8ee5ec6c7d462b2ed7753ba6c8
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[3112]: deleted: sha256:f1494f9182132b688dfe8d7db2df8fba8859b0f8915022d97fad77643b1da978
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[3112]: deleted: sha256:12709dfceeebf0f1cc36e9f74fd094483083dd085b0deed86a9b27d57e75d863
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[3112]: deleted: sha256:00e9dcc18d1813a9473846e00692ea085ee335e978398cd35675fa2a42a2435a
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[3112]: untagged: mycompany/mydevice-speech:digest_sha256_a215ef370d5f7118af718fb04746073f7de90c30acf2b11fd6703d95228d091f
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[3112]: untagged: mycompany/mydevice-kiosk:digest_sha256_5e0920db77bb1383e2a503a35e84f43c1720353921a5ffee8db7558f4f3a405d
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[3112]: untagged: mycompany/mydevice-hw:digest_sha256_fc505c46824ef5d8da8790174de546022db35c32e0747f5f0bf144f908edefff
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[3112]: Total reclaimed space: 608.5kB
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[868]: Event: InstallTargetComplete, Result - Success
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[868]: Event: AllInstallsComplete, Result - INTERNAL_ERROR
Oct 14 12:29:02 verdin-imx8mm-07276307 aktualizr-torizon[868]: Update install completed. Releasing the update lock...
Oct 14 12:36:08 verdin-imx8mm-07276307 systemd[1]: Stopping Aktualizr SOTA Client...
Oct 14 12:36:08 verdin-imx8mm-07276307 aktualizr-torizon[868]: PROXY: command received to stop execution.
Oct 14 12:36:08 verdin-imx8mm-07276307 aktualizr-torizon[868]: PROXY: stopping thread.
Oct 14 12:36:08 verdin-imx8mm-07276307 aktualizr-torizon[868]: PROXY: sending device data to Torizon OTA.
Oct 14 12:36:09 verdin-imx8mm-07276307 aktualizr-torizon[868]: terminate called after throwing an instance of 'SotaUptaneClient::ProvisioningFailed'
Oct 14 12:36:09 verdin-imx8mm-07276307 aktualizr-torizon[868]:   what():  Device was not able provision on-line
Oct 14 12:36:09 verdin-imx8mm-07276307 systemd[1]: aktualizr-torizon.service: Main process exited, code=killed, status=6/ABRT
Oct 14 12:36:09 verdin-imx8mm-07276307 systemd[1]: aktualizr-torizon.service: Failed with result 'signal'.
Oct 14 12:36:09 verdin-imx8mm-07276307 systemd[1]: Stopped Aktualizr SOTA Client.

It seems that the containers which did NOT receive an update (which already had the same digest_… tag before the update) are untagged because the ‘old’ tags get pruned, but these old tags are the same as the new tags, causing only a ‘latest’ tag to remain, such that docker-compose can no longer find the image the next time it runs. If I manually tag these images again, everything is fine. I suspect this is a bug?

cat /proc/version: Linux version 5.4.193-5.7.1-devel+git.f78299297185 (oe-user@oe-host) (gcc version 9.3.0 (GCC)) #1-TorizonCore SMP PREEMPT Fri Sep 16 14:24:52 UTC 2022

This issue is blocking a release since the update system is important for us. As I already monitor the logs to signal a successful update I could write a quick workaround to retag the unchanged images after an update, unless you already have a fix ready?

Kind regards,

Ernest Van Hoecke

Ernest · October 14, 2022, 3:26pm

I started wondering why docker system prune -a --force would be removing tags that are actively in use, but apparently this is known (weird) behaviour:

github.com/moby/moby

docker image prune/rm removes in-use image tags

opened 12:36AM - 13 Feb 18 UTC

githubsaturn

area/images

--------------------------------------------------- BUG REPORT INFORMATION ---…------------------------------------------------ **Describe the results you received:** Running `docker image prune --all --force` removes IN-USE images causing locally created images to get lost. **Describe the results you expected:** If a service is running via `locally-create-image:1`, running `docker image prune --all --force` SHOULD NOT remove `locally-create-image:1` **Steps to reproduce the issue:** 1. `docker swarm init` # simply create a swarm 2. `mkdir /test-bug` 3. `cd /test-bug` 4. `echo "FROM nginx:1.13.8" > Dockerfile` # this is an example, it can be anything 5. `docker build -t myimage:1 . ` 6. `docker service create --name my-service myimage:1` 7. wait and make sure that your container is running. Check using `docker ps -a`, you will see image is listed as `myimage:1` 8. `docker image prune --all --force` # this is supposed to remove all "unused" images 9. `docker service update my-service --force` # this can be an update of environmental variables or anything else that causes a container restart. 10. You will see that the service does not get restarted as there is no `myimage:1` image! **Additional information you deem important (e.g. issue happens only occasionally):** **Output of `docker version`:** ``` Client: Version: 17.12.0-ce API version: 1.35 Go version: go1.9.2 Git commit: c97c6d6 Built: Wed Dec 27 20:11:19 2017 OS/Arch: linux/amd64 Server: Engine: Version: 17.12.0-ce API version: 1.35 (minimum version 1.12) Go version: go1.9.2 Git commit: c97c6d6 Built: Wed Dec 27 20:09:53 2017 OS/Arch: linux/amd64 Experimental: false ``` **Output of `docker info`:** ``` Containers: 7 Running: 4 Paused: 0 Stopped: 3 Images: 66 Server Version: 17.12.0-ce Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 113 Dirperm1 Supported: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: active NodeID: hl08z3yjk34msargicyuklvzy Is Manager: true ClusterID: k883g1werpi8dj207nr09kj3h Managers: 1 Nodes: 1 Orchestration: Task History Retention Limit: 5 Raft: Snapshot Interval: 10000 Number of Old Snapshots to Retain: 0 Heartbeat Tick: 1 Election Tick: 3 Dispatcher: Heartbeat Period: 5 seconds CA Configuration: Expiry Duration: 3 months Force Rotate: 0 Autolock Managers: false Root Rotation In Progress: false Node Address: 174.7.187.204 Manager Addresses: 174.7.187.204:2377 Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 89623f28b87a6004d4b785663257362d1658a729 runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f init version: 949e6fa Security Options: apparmor seccomp Profile: default Kernel Version: 4.4.0-112-generic Operating System: Ubuntu 16.04.3 LTS OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 7.682GiB Name: REMOVED ID: REMOVED Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Username: dockersaturn Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false WARNING: No swap limit support ```

I suspect that the developers of aktualizr-torizon expect prune not to delete any currently in use tags, so after running the docker-compose it would be a great way to get rid of the old tags and images, but it seems that this is counter-intuitive and prune DOES delete tags that are in use, if there are multiple tags for an image.

jeremias.tx · October 14, 2022, 6:56pm

Greetings @Ernest,

This is a very strange sounding issue and currently not a known one to us. I also wasn’t aware of this open issue on the Docker side regarding the misbehavior with prune.

I’ll need to try and reproduce this issue myself so I can further investigate. Can you please share your docker-compose file from before the update, and the docker-compose file from after the update. I don’t need your container images, just the compose files. I want to see how each container image was tagged before and after the update according to your compose files. Based on your compose files I plan to create a similar compose files of my own to attempt to reproduce this myself.

Best Regards,
Jeremias

Ernest · October 17, 2022, 8:39am

Hi @jeremias.tx,

Thanks for the fast response. It’s a very strange issue indeed, I’m surprised at this behaviour of prune.

I’ve attached our docker-compose files in their canonical lock form, without censoring this time as Google won’t scrape this

original: docker-compose.yml (4.9 KB)
update: docker-compose.lock.yml-2022-10-13-H12M05 (5.0 KB)

I’m going to look into some workarounds by tagging the images correctly after an update with a script, unless you see a path for a quick proper solution? If it only affects containers that didn’t receive an update, I should be able to simply tag every image with the “latest” tag with the expected tag found in the docker-compose.

Kind regards,
Ernest

Ernest · October 17, 2022, 1:03pm

I’m using the following workaround for now. It’s a simple service that runs before docker-compose.service and tags all images with the expected tag from the now updated docker-compose. The awk command works because the now untagged images will have the “latest” tag, which I believe is always true because of the pruning. If it isn’t this won’t work consistently.

[Unit]
Description=Ensures images are correctly tagged post-update
Before=docker-compose.service

[Service]
Type=oneshot
ExecStart=awk '/ image: /{split($2,im,":"); system("docker tag " im[1] " " $2)}' /var/sota/storage/docker-compose/docker-compose.yml 2> /dev/null
User=root

[Install]
RequiredBy=docker-compose.service

Please let me know if you spot any obvious issues with this. Of course, a more robust real solution needs to be developed at some point.

jeremias.tx · October 18, 2022, 12:41am

I’ve been trying to reproduce the issue on my side with no luck so far. I’m using a simple docker-compose file with just 2 containers defined. Between updates I keep 1 of the container images the same while on the other I update the tag used. In this way I try and mimic your scenario where some containers are updated and some are not. But so far no luck.

I’m taking another look at your description of the events and I have some further questions:

You described a reboot occurring. Were you also updating the OS in addition to the containers? Normally if you’re just updating the containers then there’s no need for a reboot.
Looking at the docker-compose files you provided I see there are a total of 6 containers. 4 of these are being updated, while 2 stay the same. Does this issue happen to both of the 2 containers that did not change? And does the issue always manifest in the same way? (i.e. is it reproducible).
I’m still confused where the “latest” tag is coming from in your description of the events. None of your containers in your compose files use the tag “latest”. In the Docker issue you referenced, the main issue is that prune will sometimes remove/untag the wrong image if there are multiple tags for the same image ID. But, there’s no description of a container image having it’s tag changed to latest. So then where did this “latest” tag come from in your scenario?

I did some further testing and I’m even less sure what’s going in your setup. For example say we have container image “foo/bar:tag”. Then perform an offline update but we don’t change “foo/bar:tag”. For an offline update all we do is docker save/load to transfer the container images over USB. I just did a test where if I use docker load to load a container image that already exists on the device (with the same tag), then nothing happens. No new image tag is created if it already exists.

This means there should only be 1 image tag and image ID for the containers that did not change. There should be no “old” or “new” tags since the tags never changed. Which makes me wonder again where did this “latest” tag come from?

Any further details or observations would be appreciated as I’m a bit stuck on how I should further investigate this.

Best Regards,
Jeremias

Ernest · October 18, 2022, 2:54pm

Thanks for the research. It made me realise where the latest tags come from.

There’s indeed no reboot. We’re only updating containers and I got confused.
Yes, it happens to all containers that did not change. It’s reproducible, but more on that below.
Apparently, they’re coming from the pre-provisioned containers after a TEZI install. If I manually remove all containers and images on a device and install them again via Compose updates, there are no issues and no latest tags. But if I use a TEZI image with pre-provisioned images, they will have a ‘latest’ tag even tho the included docker-compose.yml is canonicalized.

In my pipeline I first use torizoncore-builder to generate a container bundle. The input docker file at this point is not canonicalized and uses the ‘latest’ tags. The build command uses this bundle. I then use the canonicalized docker-compose.yml from the bundle in my other pipeline steps (generating a compose file update lockbox/package, an OS update lockbox and synchronized update package) as a workaround for not having a torizoncore-builder command that can canonicalize a private registry and not do anything else.

As a test, I copied our latest canonicalized docker-compose to the input docker-compose.yml file, so it would be exactly the same and use the commit hashes. After installing the TEZI image built like this, the images on the system have no tags (<none>). Now, updates seem to work consistently!

It’s unexpected to me that the preprovisioned images would behave like that. As a workaround I can canonicalize the docker-compose file first via bundle, and then use bundle again before executing the rest of the pipeline.

A torizoncore-builder canonicalize-only parameter or command that also works with private registries is definitely welcome Until then I’ll apply a personal tiny patch to cancel bundle before it stores the containers, saving some time.

Kind regards,

Ernest Van Hoecke

jeremias.tx · October 18, 2022, 7:19pm

Let me summarize to make sure I understood you correctly.

You created a TorizonCore image with an initial set of containers with bundle.
The compose file you used for this was not canonicalized and used the “latest” tags for the images. You then used this compose file in the build command to create your custom TorizonCore image.
Therefore this custom image you created had a compose file and container images tagged with “latest”.
You then canonicalized the compose file since that is required to use in a Lockbox. Which of course tags everything differently then “latest”.
Then when you try to perform an update using this Lockbox, you end up with a system that has 2 tags that reference the same container image. So when the docker system prune occurs, the known Docker bug happens and the wrong tag is pruned.

Did I understand that all correctly?

As for your workaround. Now you take the canonicalized compose file and then input that into the build command instead of your initial compose file. This way the custom TorizonCore image you produce has the correct matching tags used in offline updates. Correct?

A torizoncore-builder canonicalize-only parameter or command that also works with private registries is definitely welcome

This is in our backlog as a known limitation/issue.

Best Regards,
Jeremias

Ernest · October 19, 2022, 9:44am

Correct, I create a container bundle with the bundle command.
The compose file was indeed not canonical and used the “latest” tags. But I used the output of bundle in the build command, I did not use this non-canonical compose file. In tcbuild.yaml I specify (redacted):

output:
  easy-installer:
    bundle:
      dir: bundle/

The container images were tagged with “latest”, but the docker-compose.yml in /var/sota/storage/docker-compose/docker-compose.yml is the proper canonical form.
Indeed, I use the file bundle/docker-compose.yml generated by the bundle command as a canonical compose file.
Correct.

Indeed, I now execute bundle twice. Once to simply get the canonical docker-compose file, and then once more with this canonical file as input to get the actual bundle. The tags now seem correct.

Kind regards,
Ernest

jeremias.tx · October 19, 2022, 7:31pm

Ok given your description I think I now understand the entire situation. Well ultimately this is a Docker bug with how prune is not working as expected. But I will be submitting this entire situation to our team to see if there’s a way to perhaps improve the behavior or at least work-around this weird docker behavior.

For now please keep using your work-around. Thank you for bringing this strange behavior to our attention.

Best Regards,
Jeremias