We’re running into some issues on one of our devices, that appears to be from mixing OTA and offline update lockboxes.
Essentially, we had a bug in our OS configurations that prevented offline updates from working. We produced a new build that addressed this issue and used the OTA mechanism to push this update to the affected devices.
However, these devices now refuse to accept further lockbox updates with the following error (sensitive info masked with [Product]):
Apr 30 18:24:07 [Product]-348 aktualizr-torizon[846]: fetchMetaOffUpd() called with source_path: "/var/volatile/DEV_UPDATE/update"
Apr 30 18:24:07 [Product]-348 aktualizr-torizon[846]: Director Target filename matches currently installed version, but content differs!
Apr 30 18:24:07 [Product]-348 aktualizr-torizon[846]: Inconsistency between Director metadata and available ECUs: Director Target filename [Product]_Base_OS_Image-v6.7.0-RC1 matches currently installed version, but content differs.
Apr 30 18:24:07 [Product]-348 aktualizr-torizon[846]: Event: UpdateCheckComplete, Result - Error
(Note our lockboxes typically always include the matched OS version and docker containers to keep things synchronized).
I inspected both the ostree admin status output on an affected unit, as well as our CI build logs for the lockbox we are attempting to install, and both reference the same OSTree commit in our platform package repository - so I cannot think of any reason the content would (or should) differ. The OS hash specified in the lockbox on the cloud platform is similarly correct/identical.
What can we do to troubleshoot this error further and understand what’s going wrong?
We are running CommonTorizon 6.7.0 on an intel NUC, but that shouldn’t affect the OTA/cloud platform portion as far as I know. We’ve seen this on several devices after an OTA update push, so it’s not an isolated incident.
Hi, I’m the Toradex AI Assistant. The Toradex Support team will follow up with you shortly!
Did you like my answer?
Hello!
For Torizon OS versions prior to 6.6.0-devel-202402, the online and offline update methods were mutually exclusive. However, starting with version 6.6.0-devel-202402, you can enable both methods simultaneously. To do this, set both enable_offline_updates and enable_online_updates to true in your configuration. You can find more details on this in the Torizon documentation.
If you’re troubleshooting update issues, here are some steps you can follow:
Verify Your Configuration:
Check your current Aktualizr configuration located in /etc/sota/conf.d/ to ensure your update settings are correct. More information can be found here.
Modify the Configuration File:
If needed, create or modify the configuration file with the following settings:
This one is a tricky one. Based on the logs here’s what I can infer. As the logs indicate the Lockbox has a package by the identifier [Product]_Base_OS_Image-v6.7.0-RC1. From Aktualizr’s point of view this package is already installed on the device, which normally is no issue. However, it’s determined that while the package name and version matches, something else does not match. Aktualizr blocks this action since it can be seen as a possible security issue/attack if one were to install a package that looks like (but is not exactly) another package.
Checking how Aktualizr compares package targets all the following must match for it to be the “same”:
Package name and version (i.e the “filename” as aktualizr calls it)
We know this matches since the logs tells us this
Package length
The package length for an OSTree type package should be 0, so we can assume this matches.
HWID for the targets should match, though I assume this is obvious
Target hash should match, which you say it does.
One important distinction is you said you checked ostree admin status to see if it matched or not. However, Aktualizr uses what is in it’s sql storage to determine what is currently installed on the device.
Could you share the following information:
The Lockbox metadata file which should be at <lockbox_path>/metadata/director/<name of lockbox>.json
The output of sudo aktualizr-info on the affected device.
The output of ostree admin status on the affected device.
Hopefully this gives us enough information on what Aktualizr thinks is currently installed versus what is being asked to be installed by the Lockbox.
I decided to do a quick test to see if this was easily reproducible and it is. My procedure was:
Flash 7.1.0 on a device
Perform an online update to 7.2.0
Do an offline update with a Lockbox containing the same 7.2.0 package
The offline update will fail with the familiar error:
May 01 00:44:50 verdin-imx8mp-06849059 aktualizr-torizon[872]: Director Target filename matches currently installed version, but content differs!
May 01 00:44:50 verdin-imx8mp-06849059 aktualizr-torizon[872]: Inconsistency between Director metadata and available ECUs: Director Target filename scarthgap/verdin-imx8mp/torizon/torizon-docker/release-7.2.0+build.13 matches currently installed version, but content differs.
I think I know what happened. The metadata of an offline update is not exactly the same as the metadata for an online update even with the same update package. Namely here:
An online update specifies an ECU for which the package will be applied to:
An offline update doesn’t specify ECU since it doesn’t know this info until it’s attached to a device:
"hardwareIds": [
"verdin-imx8mp"
],
The matching logic probably doesn’t know how to reconcile this difference in the metadata. This was probably overlooked when allowing for online and offline updates at the same time.
For now this means your Lockbox can’t contain the same update package, if that update package was already installed on the device by an online update.
I take it you no longer require the previously requested information anymore now that you’ve identified the issue?
Correct, since i was able to easily reproduce this on a clean Toradex standard setup, I shouldn’t need further information from your setup. Thanks for reporting this, this is one of those edge use-cases that gets missed until a user runs into it.
