Offline lockbox failing

Technical Support
, , ,
1

Hello, we are running into the following issue trying to install a lockbox via usb.

Feb 13 16:47:06 apalis-imx8-15139216 aktualizr-torizon[1325]: Initializing torizon-generic Secondaries…
Feb 13 16:47:06 apalis-imx8-15139216 aktualizr-torizon[1325]: Use existing SQL storage: “/var/sota/storage/bootloader/sql.db”
Feb 13 16:47:06 apalis-imx8-15139216 aktualizr-torizon[1325]: No valid Director metadata found in storage: Could not load latest root
Feb 13 16:47:06 apalis-imx8-15139216 aktualizr-torizon[1325]: No valid Image metadata found in storage: Could not load latest root
Feb 13 16:47:06 apalis-imx8-15139216 aktualizr-torizon[1325]: Adding Secondary with ECU serial: ae7024d84e1276af96bc1bc27d1559c7161cf98a25019f69b37ada968d3f0467 with hardware ID: apalis-imx8-bootloader
Feb 13 16:47:06 apalis-imx8-15139216 aktualizr-torizon[1325]: PROXY: initializing…
Feb 13 16:47:06 apalis-imx8-15139216 aktualizr-torizon[1325]: PROXY: using TCP port 8850.
Feb 13 16:47:06 apalis-imx8-15139216 aktualizr-torizon[1325]: Offline Updates are enabled
Feb 13 16:47:06 apalis-imx8-15139216 aktualizr-torizon[1325]: PROXY: starting thread.
Feb 13 16:47:06 apalis-imx8-15139216 aktualizr-torizon[1325]: PROXY: listening to connections…
Feb 13 16:47:20 apalis-imx8-15139216 aktualizr-torizon[1325]: fetchMetaOffUpd() called with source_path: “/media/USB/update”
Feb 13 16:47:20 apalis-imx8-15139216 aktualizr-torizon[1325]: Director metadata update failed: The root metadata was expired.
Feb 13 16:47:20 apalis-imx8-15139216 aktualizr-torizon[1325]: Event: UpdateCheckComplete, Result - Error

We are running the following setup:

  • torizoncore-builder 3.12.0 on windows using wsl2. We create an application lockbox using:
  • torizoncore-builder platform push --credentials credentials.zip --login dockerhub docker-compose.yml info
    we then create the lockbox on the torizon web app and pull it down with:
    torizoncore-builder platform lockbox --credentials credentials --platform linux/arm64(we also tried linux/arm64/v8) --login dockerhub info lockbox_name
    Attempting to download this onto the unit via usb then fails with the above error

The docker-compose we are using is simple and attached.

output of tdx-info
Software summary

Bootloader: U-Boot
Kernel version: 5.15.148-6.8.1+git.1cbf48124747 #1-TorizonCore SMP PREEMPT Fri Dec 20 08:57:54 UTC 2024
Kernel command line: root=LABEL=otaroot rootfstype=ext4 quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3 ostree=/ostree/boot.1/torizon/d9a86c146753aa0e7149e11f723f9fa23662027469de6dc61801fe915381cc20/0
Distro name: NAME=“TorizonCore”
Distro version: VERSION_ID=6.8.1-build.25
Distro variant: VARIANT=“Docker”
Hostname: apalis-imx8-15139216

Hardware info

HW model: Toradex Apalis iMX8QM V1.1 on Apalis Ixora V1.2 Carrier Board
Toradex version: 0047 V1.1D
Serial number: 15139216
Processor arch: aarch64

I have done the following debugging steps:

  1. updated from torizoncore 6.5 to 6.8 as a base image via recovery mode.
  2. updated to newest torizoncore-builder 3.12.0
  3. ensured the proper credentials are being use by redownloading credentials.zip creating shared-data.tar.gz and moving it over to the board following the documentation.
  4. changed --platform flag to linux/arm64, linux/arm64/v8, and linux/armv7
  5. tried different lockboxes both application and base image related.

Thanks, Mike
docker-compose4.yml (1.3 KB)

2

Hi, I’m the Toradex AI Assistant. The Toradex Support team will follow up with you shortly!

Did you like my answer?

:+1: :-1:

Hello Mike,

Thank you for reaching out with your issue regarding the installation of a lockbox via USB on your Toradex Apalis iMX8 device. The error message “The root metadata was expired” indicates that the metadata used for the offline update is outdated. Here are some steps you can take to resolve this:

  1. Update Metadata: Ensure that the metadata files in your lockbox are current. Metadata expiration is a security measure to prevent the use of outdated data.

  2. Regenerate the Lockbox: Try creating a new lockbox with the latest data from the Torizon Platform:

    • Re-define the lockbox in the Torizon Platform Web UI to include the latest software and metadata.
    • Use the torizoncore-builder platform lockbox command to download the updated lockbox.

  3. Provisioning Data: Verify that your device is provisioned with the latest root metadata. Ensure the provisioning data is correctly extracted into /var/sota/import/ on your device. Re-provision the device if necessary.

  4. Configuration Check: Confirm that your device is set up to accept offline updates. Check the configuration files in /etc/sota/conf.d/ to ensure offline updates are enabled and correctly configured.

  5. Credential Verification: Double-check that the credentials used for accessing the Torizon Platform are up-to-date and correctly applied. Ensure the credentials.zip file is the latest version and properly used in the lockbox creation process.

  6. Log Monitoring: Continue to monitor the logs for any additional errors or warnings that might provide further insight into the issue.

By following these steps, you should be able to resolve the expired metadata issue and successfully perform the offline update. If you have any further questions, feel free to ask.

Best regards,
Toradex AI Assistant

Please, also check if these links can help you solve your problem:

[aktualizr, offline updates] Found in archive a file with bad file type: 40960.

3

I also attempted using torizoncore 3.11.0 based off of this post to no luck:

4

looking at the 1.root and 2.root files both have expiry’s in 2024

5

updating to the latest early access torizoncore-builder resulted in a 1.root that was expired but a 2.root and root.json that were not expired(expiry 2025-04-11 stil earlier then the lockbox itself though). Still no success

6

checking the shared-data.tar.gz file that is placed onto the unit and comparing it to the 2.root file it matches the expiry date. my credentials director.root file also shows 2025-04-11

7

Greetings @MikeHA,

updating to the latest early access torizoncore-builder resulted in a 1.root that was expired but a 2.root and root.json that were not expired(expiry 2025-04-11 stil earlier then the lockbox itself though). Still no success

Wait so just to double-check you re-created the Lockbox and now you have a Lockbox where the highest numbered root.json has an expiry of “2025-04-11”. Meaning it should not be expired yet. Despite this, you are saying you still get the same exact error saying that the root metadata was expired?

That doesn’t sound quite right. On my side if I create a fresh Lockbox, I don’t see any issues.

Have you tried defining a completely new Lockbox on the Web UI, then downloading it, and trying to install that just as a sanity test?

Best Regards,
Jeremias

8

Hey Jeremias,

Yes that is correct. The highest root.json did not have an expired metadata tag and the exact same error occurred.

I created a fresh lockbox, attempted to use an old lockbox, attempted to create a lockbox with no docker-compose and only a ostree reference for the image.

No success on my end for any of those.

9

Well this is certainly an unexpected behavior. Could you please share your Lockbox files, or at the very least share all the files found in the metadata folder in your Lockbox. I want to examine the files first hand to see if I can notice any abnormalities. The metadata files should be safe to share and contain no sensitive or private information.

By the way, is this your first time using Lockboxes/offline updates? Or have you successfully used them in the past and this issue is new?

Best Regards,
Jeremias

10

Hi Michael,

Any updates here?

Best regards,

Michael Park
Toradex

11

Hey Jeremias and Michael,

Sorry for taking a bit to get back to you as I was OOO. I have attached metadata from one of the many lockboxes we have tried to build unsuccessfully.

We attempted to just clean install a new TEZI image from recovery mode and that was unsuccessful as well with downloading lockboxes. From a closer inspection it seems that it might not be finding the metadata in storage on the device? See the copied output below. This is odd since I have tried both provisioning the offline updates with known good credentials both via the TEZI build process and by hand via untarring the shared-data.tar.gz on the board and setting it up on an unprovisioned base image.

On another note, If we were to rotate credentials on torizon cloud would we still be able to update devices that were installed with the previous credentials? Or would we have to upload a new shared-data.tar.gz file to the board and restart aktualizr for the new credentials to take effect? Hoping just rotating the credentials might help something but want to know the impact before we do.

Thanks, Mike

Feb 18 20:46:41 apalis-imx8-15139216 aktualizr-torizon[1048]: Bootstrap empty SQL storage
Feb 18 20:46:41 apalis-imx8-15139216 aktualizr-torizon[1048]: Bootstraping DB to version 26
Feb 18 20:46:41 apalis-imx8-15139216 aktualizr-torizon[1048]: No valid Director metadata found in storage: Could not load latest root
Feb 18 20:46:41 apalis-imx8-15139216 aktualizr-torizon[1048]: No valid Image metadata found in storage: Could not load latest root
Feb 18 20:46:41 apalis-imx8-15139216 aktualizr-torizon[1048]: Adding Secondary with ECU serial: 3ba86e96acb8ee6e15b085defce184019aae2ccc1aa6875de8e2142bc93d6f09 with hardware ID: apalis-imx8-bootloader
metadata.zip (321.9 KB)

12

First of all just to reply to your questions.

From a closer inspection it seems that it might not be finding the metadata in storage on the device?

No this is not an error and is normal. When the device gets new metadata from an offline or online update it stores the metadata into an sql storage database on the device if the metadata is valid. If it’s the first time the device is receiving metadata, then obviously there isn’t anything in storage yet. Therefore it’s completely valid and expected to see No valid Director metadata found in storage: Could not load latest root.

On another note, If we were to rotate credentials on torizon cloud would we still be able to update devices that were installed with the previous credentials? Or would we have to upload a new shared-data.tar.gz file to the board and restart aktualizr for the new credentials to take effect?

Rotating credentials should work. This would just result in a new root metadata (i.e 3.root.json), that uses new credentials. As long as the validation of the new keys is coherent and sound then that’s not inherently an issue. This is detailed in our documentation here: Offline Signing Keys | Toradex Developer Center

The shared-data.tar.gz is only used for the initial state of the device when no valid metadata from updates have been received yet. Otherwise, once an update has brought new valid metadata then the device will use that going forward, until newer metadata comes along in another update.

Though you should test to make sure it works with whatever process/procedure you plan to do on your devices.

Hoping just rotating the credentials might help something but want to know the impact before we do.

I mean I see no reason that rotating the credentials would help. Especially since the issue is not understood. It might “fix” things as in the issue goes away, but without understanding what the issue is we can’t really guarantee it won’t happen again somehow.

With your questions out of the way, I would really appreciate it if you could address my questions below.

I gave your metadata an initial look. I didn’t see anything immediately wrong the metadata looked sane and the expiry on 2.root.json is indeed still in the future. Though I do have some questions.

  • Since you mentioned rotating the credentials. Have you already rotated your keys offline?
  • On your devices where this issue occurs, what time do the devices think it is? If your devices for some reason think it’s further in the future this could explain why it thinks the metadata is expired.
  • Looking at your metadata it’s obvious that you’ve done offline updates in the past. This means this issue you described here is new. Have you done or changed anything recently that seems to coincide with this issue?
  • How exactly do you create a new Lockbox? Do you create it using our Web UI, or do you create it using our API?

Best Regards,
Jeremias

1 Like
13

Hey Jeremias,

It is resolved now as it was due to the time being wrong. It was set correctly originally but probably was reset when we placed the device in recovery mode to install a new TEZI image and we did not notice.

Thank you for your help,
Mike

14

I see, well that would explain things then. Thank you for letting us know what the issue was.

Best Regards,
Jeremias