Offline Update Fails

Technical Support
1

Software summary

Bootloader: U-Boot
Kernel version: 6.6.54-7.1.0-g3493ccd66900 #1-Torizon SMP PREEMPT Tue Dec 17 21:04:41 UTC 2024
Kernel command line: root=LABEL=otaroot rootfstype=ext4 quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3 ostree=/ostree/boot.0/torizon/b047901f316835aaf584d61dc7f9860d533091b059f656d5656501803190bbc1/0
Distro name: NAME=“Torizon OS”
Distro version: VERSION_ID=7.1.0-build.4
Distro variant: VARIANT=“Docker”
Hostname: verdin-imx8mp-15404619

Hardware info

HW model: Toradex Verdin iMX8M Plus WB on Verdin Development Board
Toradex version: 0058 V1.1B
Serial number: 15404619
Processor arch: aarch64

I have a trial Torizon Cloud account. We are looking at using Torizon for secure offline updates using lockbox mechanism. I have attempted to follow demo from Torizon Secure Offline Updates youTube video.

I have created and installed a base image with hello-mod added to kernal.
Updated versions of hello-mod and tcbuild.yaml
Uploaded OS Image to Torizon Cloud (I should note that during some earlier testing I created offline signing keys and recreated Credentials.zip with these keys).
To upload I needed to provide the Credentials.zip with the offline keys
Create lockbox in Torizon Cloud
Download lockbox using torizoncore-builder (Note: This fails if I try to use the Credentials.zip with offline keys), but works if I use my original Credentials.zip file (don’t know if this is my issue)
Copy update folder to USB drive.
Verified on device that /usr/lib/sota/conf.d/99-offline-updates.toml exists

I insert my USB drive, it starts and almost complete update, but then it reboots and reverts to a previous image.torizon@verdin-imx8mp-15404619:~$ journalctl -f -u aktualizr*
Oct 09 15:18:29 verdin-imx8mp-15404619 aktualizr-torizon[904]: Adding Secondary with ECU serial: 4a64ec90da2c6facc4573566a53b05a91e7eee90c97eab5335052e513b341595 with hardware ID: docker-compose
Oct 09 15:18:29 verdin-imx8mp-15404619 aktualizr-torizon[904]: Initializing torizon-generic Secondaries…
Oct 09 15:18:30 verdin-imx8mp-15404619 aktualizr-torizon[904]: created: /var/sota/storage/bootloader
Oct 09 15:18:30 verdin-imx8mp-15404619 aktualizr-torizon[904]: created: /var/sota/storage/bootloader/metadata
Oct 09 15:18:30 verdin-imx8mp-15404619 aktualizr-torizon[904]: Bootstrap empty SQL storage
Oct 09 15:18:30 verdin-imx8mp-15404619 aktualizr-torizon[904]: Bootstraping DB to version 26
Oct 09 15:18:30 verdin-imx8mp-15404619 aktualizr-torizon[904]: No valid Director metadata found in storage: Could not load latest root
Oct 09 15:18:30 verdin-imx8mp-15404619 aktualizr-torizon[904]: No valid Image metadata found in storage: Could not load latest root
Oct 09 15:18:30 verdin-imx8mp-15404619 aktualizr-torizon[904]: Adding Secondary with ECU serial: 758b37ea4e2f539ea9a3040ee7f3622c684550f6863b44419c6581605faa4e21 with hardware ID: verdin-imx8mp-bootloader
Oct 09 15:18:30 verdin-imx8mp-15404619 aktualizr-torizon[904]: Offline Updates are enabled
Oct 09 15:26:39 verdin-imx8mp-15404619 aktualizr-torizon[904]: fetchMetaOffUpd() called with source_path: “/media/OFFLINEUSB”
Oct 09 15:26:39 verdin-imx8mp-15404619 aktualizr-torizon[904]: Current version for ECU ID: d05fb81dda05ef9770e926054c5be5ca45de1fa9f78432e6c6684aaf3d9ad47c is unknown
Oct 09 15:26:39 verdin-imx8mp-15404619 aktualizr-torizon[904]: New updates found in Director metadata. Checking Image repo metadata…
Oct 09 15:26:39 verdin-imx8mp-15404619 aktualizr-torizon[904]: 1 new update found in both Director and Image repo metadata.
Oct 09 15:26:39 verdin-imx8mp-15404619 aktualizr-torizon[904]: Event: UpdateCheckComplete, Result - Updates available
Oct 09 15:26:39 verdin-imx8mp-15404619 aktualizr-torizon[904]: Current version for ECU ID: d05fb81dda05ef9770e926054c5be5ca45de1fa9f78432e6c6684aaf3d9ad47c is unknown
Oct 09 15:26:39 verdin-imx8mp-15404619 aktualizr-torizon[904]: Performing a local pull from file:///media/OFFLINEUSB/images/ostree
Oct 09 15:26:41 verdin-imx8mp-15404619 aktualizr-torizon[904]: ostree-pull: Scanning metadata: 931
Oct 09 15:26:41 verdin-imx8mp-15404619 aktualizr-torizon[904]: Event: DownloadProgressReport, Progress at 0%
Oct 09 15:26:41 verdin-imx8mp-15404619 aktualizr-torizon[904]: ostree-pull: 9 metadata, 5 content objects imported; 1.1 MB content written
Oct 09 15:26:41 verdin-imx8mp-15404619 aktualizr-torizon[904]: Event: DownloadTargetComplete, Result - Success
Oct 09 15:26:41 verdin-imx8mp-15404619 aktualizr-torizon[904]: Event: AllDownloadsComplete, Result - Success
Oct 09 15:26:41 verdin-imx8mp-15404619 aktualizr-torizon[904]: Current version for ECU ID: d05fb81dda05ef9770e926054c5be5ca45de1fa9f78432e6c6684aaf3d9ad47c is unknown
Oct 09 15:26:41 verdin-imx8mp-15404619 aktualizr-torizon[904]: Event: InstallStarted
Oct 09 15:26:41 verdin-imx8mp-15404619 aktualizr-torizon[904]: curl error 6 (http code 0): Couldn’t resolve host name
Oct 09 15:26:41 verdin-imx8mp-15404619 aktualizr-torizon[904]: Installing package using ostree package manager
Oct 09 15:26:41 verdin-imx8mp-15404619 aktualizr-torizon[904]: Commit metadata kargs=quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3
Oct 09 15:26:42 verdin-imx8mp-15404619 aktualizr-torizon[904]: curl error 6 (http code 0): Couldn’t resolve host name
Oct 09 15:26:43 verdin-imx8mp-15404619 aktualizr-torizon[904]: curl error 6 (http code 0): Couldn’t resolve host name
Oct 09 15:26:43 verdin-imx8mp-15404619 aktualizr-torizon[904]: Failed to post update events: 6 Couldn’t resolve host name HTTP 0
Oct 09 15:26:44 verdin-imx8mp-15404619 aktualizr-torizon[904]: Copying /etc changes: 3 modified, 2 removed, 17 added
Oct 09 15:26:45 verdin-imx8mp-15404619 aktualizr-torizon[904]: Starting syncfs() for system root
Oct 09 15:26:45 verdin-imx8mp-15404619 aktualizr-torizon[904]: Completed syncfs() for system root in 358 ms
Oct 09 15:26:45 verdin-imx8mp-15404619 aktualizr-torizon[904]: Starting freeze/thaw cycle for system root
Oct 09 15:26:45 verdin-imx8mp-15404619 aktualizr-torizon[904]: Completed freeze/thaw cycle for system root in 143 ms
Oct 09 15:26:45 verdin-imx8mp-15404619 aktualizr-torizon[904]: Transaction complete; bootconfig swap: yes; bootversion: boot.0.1, deployment count change: 1
Oct 09 15:26:46 verdin-imx8mp-15404619 aktualizr-torizon[904]: Performing sync()
[ 525.419950] watchdog: watchdog0: watchdog did not stop!
[ 527.873231] imx2-wdt 30280000.watchdog: Device shutdown.
[ 527.879880] reboot: Restarting system

2

Hi, I’m the Toradex AI Assistant.

I might make mistakes. Our support team will step in if needed. If I answered your question, please tick the “Solution” checkbox. For follow-up questions to me, please use @ToradexAI in your post.

Did you like my answer? :+1: :-1:

Hello! Thank you for the detailed description and logs of your Torizon secure offline update setup on the Verdin iMX8M Plus. Based on your information, the offline update process initiates and appears to complete, but after a reboot, the device reverts to the previous image—indicating an automatic rollback due to a failed boot or update validation.

Let’s summarize and address your situation:

Key Observations

  • Platform: Verdin iMX8M Plus WB, Torizon OS 7.1.0-build.4 (Docker variant), Kernel 6.6.54.
  • Configuration:
    • /usr/lib/sota/conf.d/99-offline-updates.toml exists.
    • Offline update appears to start (Offline Updates are enabled), finds and applies the new image, but then reboots and rolls back.
    • You used two different credentials.zip files: one original and one recreated with offline signing keys.
    • Using the credentials.zip with offline keys for lockbox downloads fails; the original works.

Analysis & Recommendations

1. Offline Trust Chain and Provisioning

The most likely root cause is a mismatch in the trust chain. If you:

  • Created new offline signing keys and re-provisioned your device (with a new credentials.zip),
  • But continued to use a lockbox (or update data) generated from another set of credentials,

the public keys and signatures will not align. The device will reject the update metadata and revert to the previous image after the reboot.

Action: Ensure that the credentials.zip used both for device offline provisioning and for generating the lockbox/update via torizoncore-builder are from the same root/trust chain.

More info: How to Use Secure Offline Updates with TorizonCore

2. Configuration File Check

Verify your /usr/lib/sota/conf.d/99-offline-updates.toml is correct, for example:

[uptane]
enable_offline_updates = true
enable_online_updates = false
offline_updates_source = "/media/OFFLINEUSB"

And be sure /media/OFFLINEUSB is the correct path when the USB is inserted.

3. Logs—Boot Failure Cause

The logs don’t show the reason for rollback. It’s vital to look at the logs from the first boot after the update—before rollback occurs. You can often find clues in journalctl or on the UART/debug console. Look for:

  • ostree errors,
  • SELinux/AppArmor denials,
  • Filesystem or boot parameter mismatches,
  • Module (hello-mod) loading errors.

4. Curl Error 6

The curl error 6 (Couldn’t resolve host name) is expected in offline updates and is not a blocker:
Known curl error in offline updates

5. Trial Account Limitation

To use secure offline updates, a commercial license is needed. On a trial/free account, certain features might not operate fully. More here: First Steps with Secure Offline Updates

Next Steps

  1. Recreate your credentials.zip and lockbox so they all use the same trust chain and keys.
  2. Reprovision your device with the matching credentials.zip (torizoncore-builder provisioning).
  3. Generate lockbox and OS update using the same credentials.
  4. Place lockbox/update on USB and apply offline update.

Carefully monitor logs on first boot after update, via serial/UART if possible.

If you still get a rollback, capture and review system logs immediately after the failed boot to see what prevented a successful boot.

Please let me know how this process goes and if you need guidance on any of these steps.

Best regards,
Toradex AI Assistant.

References:

3

Greetings @cjgannon,

It’s not clear to me what is occurring here. Let me try to understand.

You have a device that was flashed with version A, let’s say of the OS. You try to perform an offline update for the OS to version B let’s say.

You’re saying that the device reboots and it’s still on version A instead of B?

The logs you provided look like they’re from before the reboot and I don’t see anything amiss. Do you have any logs from after the reboot?

Also when the device reboots do you see anything in the boot logs that look odd or strange? Perhaps there was an issue trying to boot version B. If the system can’t boot successfully into the new OS version after 3 tries it will rollback to the previous version instead.

Best Regards,
Jeremias

4

Hi Jeremias,

It’s actually stranger than that. I had an OS Image on the device. Then I used the Easy Installer to load version A. Version A had the hello-mod as well as the offline updates file that I added in changes/ for /usr/lib/sota/conf.d/99-offline-updates.toml

After the reboot, the hello-mod is gone as is the 99-offline-updates.toml file

I’m thinking I messed things up by switching to offline signing keys and re-packing my credentials file. Because I have to use the re-packed file to push my image to the cloud but
I get errors using the re-packed file to download the lockbox, so I use my original credentials.zip file.

5

I’m thinking I should see if I can just delete my account and start over with a basic credentials.zip file

6

I’m thinking I should see if I can just delete my account and start over with a basic credentials.zip file

I’m not entirely sure if this is the cause of the issue you’re seeing here. I would expect different types of issues if your credentials were incorrect or messed up by what you did.

It’s actually stranger than that. I had an OS Image on the device. Then I used the Easy Installer to load version A. Version A had the hello-mod as well as the offline updates file that I added in changes/ for /usr/lib/sota/conf.d/99-offline-updates.toml

Okay wait, now I’m confused. Initially you said you did an offline update. But here, all you described was flashing your OS version with Easy Installer.

What caused the reboot then? An offline update or flashing with Easy Installer? These are very different actions.

After the reboot, the hello-mod is gone as is the 99-offline-updates.toml file

While I’m still confused on what your exact process here was (Easy Installer vs offline update). Based on what you said here, it sounds like maybe you have the wrong image.

You believe you created a custom OS image that has hello-mod and 99-offline-updates.toml. But, flashing/updating to this image results in an OS where these are missing. Perhaps you’re mistaken about what OS version you used. It almost sounds like you’re flashing/updating to the base image before you did any customization.

It might help to retrace and describe how exactly you produced this custom OS version that should have hello-mod and 99-offline-updates.toml.

Best Regards,
Jeremias

7

That may be the issue. I may have pushed the base image up to the cloud and packaged that into my lockbox which would have the effect of undoing my changes.

What I have done now is modified tcbuild.yaml to put my changes on a branch on ostree, then try to push that branch up to create a lockbox for offline update.

However, I now have a new problem. I am now getting an uptane error trying to push this image up to the cloud:
torizoncore-builder platform push --credentials offline-credentials.zip --package-name secure-offline-update-image-v2 --package-version v2.0 update-test-2
Pushing update-test-2 (commit checksum 8505e68aae16fcc66fea5016026d3f9c11915fe05deed49cc414ef409813bc3c) to OTA server.
Authentication with oauth2 failed
Authentication with push server failed
Error (1) running uptane command “garage-push” with arguments “[‘–credentials’, ‘/workdir/offline-credentials.zip’, ‘–repo’, ‘/storage/ostree-archive’, ‘–ref’, ‘8505e68aae16fcc66fea5016026d3f9c11915fe05deed49cc414ef409813bc3c’, ‘–loglevel’, ‘4’]”

8

However, I now have a new problem. I am now getting an uptane error trying to push this image up to the cloud:

Okay that latest error now looks like something I would expect if you were having issues related to your keys/credentials as you initially thought.

I see you are using your credentials.zip with the offline keys correct? Does it work if you use the other one?

Best Regards,
Jeremias

9

Same result

I’m thinking I must’ve done something to mess up my credentials.

10

Hey @cjgannon,

jumping in here… is there any chance you hit the “rotate credentials” button in the UI?

I ask, because the error you reported seeing above Authentication with oauth2 failed shouldn’t have anything to do with your offline metadata signing keys so that’s confusing me a bit…

You also mentioned just wanting to start over, we can do that easily enough if you want to go that route, let me know.

11

Hi Ben,

Yes, when I thought that my issue was a credentials mismatch I tried the rotate credentials. So that is probably the state I am stuck in right now.

I think the easiest route right now would be just starting over.

Thanks
Chuck