I created my own Torizon image with Yocto. Additionally, I activated secure-boot or chain of trust by adding ‘torizon-signed’ to local.conf file. The built image can be installed with EasyInstaller and is successfully verified by dm-verity at boot. System comes up and is running.
Now, if I bundle some docker-images into the image, it can be installed successfully but dm-verity fails verifying it:
Enabling fsverity failed (system will not boot).
sysroot.readonly configuration value: 0 (fs writable: 1)
ostree-prepare-root: Error loading signatures from repo: No commitmeta for commit 266a633544adb3e19a138aafee213079d8e2678e5b8d82150b37919cd9c74007
ERROR: There's no '/dev' on rootfs.
Pre-provisioning docker images without secure-boot also works. Is it possible that TorizonCoreBuilder doesn’t patch the bundled image correctly? Is there a possibility to do that?
Hi, I’m the Toradex AI Assistant. The Toradex Support team will follow up with you shortly!
Did you like my answer?
Hello KV,
Thank you for reaching out with your question. It sounds like you’re encountering an issue with dm-verity failing to verify your Torizon image after bundling Docker images. This is likely related to the integrity verification process, which is crucial when secure boot is enabled.
Here are some steps you can take to resolve this:
Ensure Proper Signing: After adding Docker images to your Torizon image, make sure the entire image is re-signed. The addition of Docker images can change the image’s hash, which must be updated in the signature for dm-verity to verify it correctly.
Use TorizonCoreBuilder Correctly: Ensure you are following the recommended workflow for TorizonCoreBuilder. Use the bundle command to download and prepare Docker images, and then the combine command to integrate these images into your Torizon image.
Verify Image Integrity: Before deploying, verify the image’s integrity using appropriate tools to ensure all components, including Docker images, are correctly signed and verified.
Check Configuration: Review your local.conf and other configuration files to ensure all settings related to secure boot and dm-verity are correctly configured.
By following these steps, you should be able to maintain the integrity of your Torizon image with bundled Docker images, allowing dm-verity to verify it successfully.
Best regards,
Toradex AI Assistant
Please, also check if these links can help you solve your problem:
That’s great to know that you managed to implement secure boot using meta-toradex-security layer
Right now (and as far as I know), the implementation of Full Chain of Trust (covering containers) is still ongoing. The same goes to TorizonCore Builder.
I’ve done some more tests with secure boot and read-only filesystem respectively.
I’ve installed my Torizon based image (this time without pre-provisioned containers) using EasyInstaller. System is booting, all fine. I didn’t install any containers. Afterwards I uploaded a modfied image to Torizon cloud and triggered device update. Update is downloaded and installed, device reboots and then the same error occurs:
sysroot.readonly configuration value: 0 (fs writable: 1)
ostree-prepare-root: Error loading signatures from repo: No commitmeta for commi
t e55ceb19e265deeb6d9dcb051ce97092ba6a59a3b5f6d2d6988df98c9e967ee7
ERROR: There's no '/dev' on rootfs.
System does not boot and after 3 boot attempts it recovers to old image version.
So it seems that handling images with activated secure boot is still problematic.
My built image is based on scarthgap-7.x.y branch, commit of 2024-12-30.
You haven’t answered my last post, so I’m asking again. Are Toradex developers able to reproduce the update error when secure boot is activated? Or am I possibly doing anything wrong? (look at last post regarding used software versions)
We are currently developing a new embedded device and at the moment it is planned that it will run Torizon with secure boot enabled.
Sorry about the delay in answering you. From our side, we haven’t seen such an issue so far. I will try to reproduce this tomorrow. But in the meantime, it would be helpful if you can try and answer the following questions:
What changes did you have you made to the filesystem to make it read-only?
I suppose your update image is also signed with the same keys that you used for the original image. Is that correct?
Could you send me the logs from your Aktualizr client when you apply the update?
to activate secure boot I just wrote ‘INHERIT += “torizon-signed”’ into my ‘local.conf’ file. Because we are just evaluating at the moment, I further deactivated u-boot to be signed by adding ‘TDX_IMX_HAB_ENABLE=“0”’. In your documentation it is said that keys will be generated automatically. So I haven’t done anything special here. To test fTPM and encrypted data partition I also added the following things in ‘local.conf’:
But I don’t think that activating these is causing the error.
To create an update image I just patched a file for the rootfs to be generated and re-run ‘bitbake torizon-docker’. So I think, the same keys as previously generated should have been used.
Log from ‘Aktualizr’ client:
journalctl -f -u aktualizr*
Jan 16 07:41:25 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Device Gateway URL: https://dgw.torizon.io
Jan 16 07:41:25 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Certificate subject: CN=1735d7b1-9406-448b-8daf-bc741f280e4b
Jan 16 07:41:25 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Certificate issuer: CN=ota-devices-CA
Jan 16 07:41:25 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Certificate valid from: Jan 16 07:40:15 2025 GMT until: Jan 16 07:40:15 2125 GMT
Jan 16 07:41:25 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Offline Updates are enabled
Jan 16 07:41:27 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Event: SendDeviceDataComplete
Jan 16 07:41:27 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Current versions in storage and reported by OSTree do not match
Jan 16 07:41:27 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Invalid role in root.json
Jan 16 07:41:27 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Invalid role in root.json
Jan 16 07:41:27 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Event: UpdateCheckComplete, Result - No updates available
Jan 16 07:46:27 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Current versions in storage and reported by OSTree do not match
Jan 16 07:46:27 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Invalid role in root.json
Jan 16 07:46:27 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Invalid role in root.json
Jan 16 07:46:27 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Current version for ECU ID: f64e970ad9a5fabcbe56f79effb17512f5432ed478969b6d6b1418989be64885 is unknown
Jan 16 07:46:27 verdin-imx8mp-15379635 aktualizr-torizon[1005]: New updates found in Director metadata. Checking Image repo metadata...
Jan 16 07:46:27 verdin-imx8mp-15379635 aktualizr-torizon[1005]: 1 new update found in both Director and Image repo metadata.
Jan 16 07:46:27 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Event: UpdateCheckComplete, Result - Updates available
Jan 16 07:46:27 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Invalid role in root.json
Jan 16 07:46:27 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Invalid role in root.json
Jan 16 07:46:27 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Current version for ECU ID: f64e970ad9a5fabcbe56f79effb17512f5432ed478969b6d6b1418989be64885 is unknown
Jan 16 07:46:29 verdin-imx8mp-15379635 aktualizr-torizon[1005]: ostree-pull: Scanning metadata: 888
Jan 16 07:46:29 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Event: DownloadProgressReport, Progress at 0%
Jan 16 07:46:29 verdin-imx8mp-15379635 aktualizr-torizon[1005]: libostree pull from 'aktualizr-remote' for 0 refs complete
security: GPG: disabled
security: SIGN: disabled http: CA-pinned
non-delta: meta: 8 content: 10
transfer: secs: 2 size: 58.0 kB
Jan 16 07:46:29 verdin-imx8mp-15379635 aktualizr-torizon[1005]: ostree-pull: 8 metadata, 10 content objects fetched; 56 KiB transferred in 2 seconds; 3.3 kB content written
Jan 16 07:46:29 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Event: DownloadTargetComplete, Result - Success
Jan 16 07:46:29 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Event: AllDownloadsComplete, Result - Success
Jan 16 07:46:29 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Invalid role in root.json
Jan 16 07:46:29 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Invalid role in root.json
Jan 16 07:46:29 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Current version for ECU ID: f64e970ad9a5fabcbe56f79effb17512f5432ed478969b6d6b1418989be64885 is unknown
Jan 16 07:46:29 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Event: InstallStarted
Jan 16 07:46:30 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Installing package using ostree package manager
Jan 16 07:46:30 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Commit metadata kargs=quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3 systemd.gpt_auto=0
Jan 16 07:46:37 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Copying /etc changes: 0 modified, 0 removed, 0 added
Jan 16 07:46:38 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Starting syncfs() for system root
Jan 16 07:46:38 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Completed syncfs() for system root in 0 ms
Jan 16 07:46:38 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Starting freeze/thaw cycle for system root
Jan 16 07:46:39 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Completed freeze/thaw cycle for system root in 548 ms
Jan 16 07:46:39 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Transaction complete; bootconfig swap: yes; bootversion: boot.0.1, deployment count change: 1
Broadcast message from root@localhost (Thu 2025-01-16 07:46:40 UTC):
The system will reboot now!
Jan 16 07:46:40 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Performing sync()
Jan 16 07:46:40 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Event: InstallTargetComplete, Result - Success
Jan 16 07:46:40 verdin-imx8mp-15379635 aktualizr-torizon[1005]: Event: AllInstallsComplete, Result - NEED_COMPLETION
Jan 16 07:46:40 verdin-imx8mp-15379635 aktualizr-torizon[1005]: About to reboot the system in order to apply pending updates...
Jan 16 07:46:40 verdin-imx8mp-15379635 systemd[1]: aktualizr-torizon.service: Deactivated successfully.
Jan 16 07:46:40 verdin-imx8mp-15379635 systemd[1]: aktualizr-torizon.service: Consumed 10.752s CPU time.
On debug console the following boot stops with message:
Starting systemd-udevd version 255.13^
sysroot.readonly configuration value: 0 (fs writable: 1)
ostree-prepare-root: Error loading signatures from repo: No commitmeta for commi
t e55ceb19e265deeb6d9dcb051ce97092ba6a59a3b5f6d2d6988df98c9e967ee7
ERROR: There's no '/dev' on rootfs.
I further investigated the problem and it seems that there really is no commitmeta file in the deployed directory of the new version. In the OSTree deployment directory of the original version flashed with EasyInstaller such a commitmeta file exists. So I think the origin of the problem is based on interaction of composefs checks and OSTree deployment (link in previous statement).
I was not able to reproduce the error yet as I ran into some other issues during my build. May I ask for your patience until tomorrow evening so I can give it a try again and get back to you with more information?
sorry if I’m exerting pressure on you but I’m in a hurry. My free of charge account for your Torizon cloud is going to run out soon and then I won’t be able to test remote update feature anymore. And the result of these tests determines if we can use Torizon with activated secure boot or not. I’m afraid that shipping devices without secure boot and activating it by update could be very difficult or impossible.
I did some testing here and I was able to try and reproduce what you are observing. I did a build on our Scarthgap version with the following in my local.conf:
I built 2 images like this. One image I flashed to my Verdin i.MX8M Plus using Toradex Easy Installer. The other I uploaded to our Torizon Cloud platform using TorizonCore Builder. I provisioned the device and attempted an OS update to the uploaded version. The system got the update and rebooted as expected, but the boot failed to succeed with the following:
Starting systemd-udevd version 255.13^
sysroot.readonly configuration value: 0 (fs writable: 1)
ostree-prepare-root: Error loading signatures from repo: No commitmeta for commit f28c3273fb15f8f43059cbcb7fd50118e45e2a9971226061b5005241c1d1f85b
ERROR: There's no '/dev' on rootfs.
Which more or less looks exactly like what you got. This is unexpected as our team did do work in the past to make sure an OTA update would work here. Therefore, I can only assume this is some kind of new bug or regression. I will report this to our team for further investigation and fixing. Thank you for bringing this to our attention.
Is your team optimistic to be able to fix this issue? If so, are you able to estimate when this bug will be fixed?
I’ve discussed this internally with our team and we do intend for this update-path to work properly. The team believes this shouldn’t require too much work to address and get working. As for timeline, I don’t have an exact timeline as things can always change with regards to timeline and priority. But it seems very likely that we can address this within the current quarter at the worst. Though the more likely estimation, we’re maybe talking within a few weeks before we have some kind of fix available in our testing/CI feeds.
If your current rush is due to your concern for your free trial running out on Torizon cloud. I could bring this up with our sales team and see if we could extend things or work something out.
With these changes TorizonCore Builder will not be able to upload the previously missing commitmeta objects. This should now allow you to do an update with OS images that have composefs enabled.
One small caveat. If the device already is running an image with composefs enabled, and you are doing an update to another OS image with composefs. Then you only need this early-access for TorizonCore Builder.
If your device is running an image without composefs enabled, and you are trying to do an update to an image with composefs enabled. Then, you need to also make sure the composefs enabled OS image was built using a recent version of Torizon OS. Since this transition path is more involved and required changes on the OS level.
I would like to test if update process now is working with your changes in the course of the next weeks. But as I mentioned before, my free trial of Torizon cloud has run out in the meantime. Would it be possible that you reactivate it for a few days?
