Torizon Core 7.1 OTA fails with composefs

Technical Support
, , , , ,
1

After creating a Torizon Core 7.1.0 image by declaring torizon-signed in local.conf and updating from 6.8.0 to 7.1.0 via Torizon Cloud, systemd-udevd fails to start and the application cannot start .

Error log

Starting systemd-udevd version 255.13
sysroot.readonly configuration value: 0 (fs writable: 1)
composefs+ostree: Validated commit signature using '/etc/ostree/initramfs-root-binding.key'
composefs: Verifying digest: 42482470888fd9dac06e4576ff76ebf03cd594f40792d86cc9ff026605e7e374
ostree-prepare-root: composefs: failed to mount: No such file or directory
ERROR: There's no '/dev' on rootfs.

A similar problem is the “Secure boot with bundled docker images into Torizon” issue.

This error was reproduced after uploading Torizon Core 7.1 to Torizon Cloud with torizoncore-builder:early-access.

Torizon Cloud 6.8.0 local.conf has tdx-signed declared. torizon-signed is not declared because it causes build errors in 6.8.0.

How to reproduce

  1. Install Torizon Core 6.7.0 to apalis-imx8 board.
  2. Update Torizon Core to 6.8.0 via Torizon Cloud.
  3. Update the boot loader to 7.1.0 via Torizon Cloud.
  4. Update the Torizon Core to 7.1.0 via Torizon Cloud.
  5. boot fails.
2

Hi, I’m the Toradex AI Assistant. The Toradex Support team will follow up with you shortly!

Did you like my answer?

:+1: :-1:

Hello,

I’m sorry to hear about the difficulties you’re experiencing with the update from Torizon Core 6.8.0 to 7.1.0. The error involving ‘systemd-udevd’ and the missing ‘/dev’ directory can indeed be challenging. Let’s go through some steps to help you troubleshoot and resolve this issue:

  1. Verify Configuration Files: Double-check your local.conf and other relevant configuration files to ensure they are correctly set up for Torizon Core 7.1.0. The torizon-signed declaration should be compatible with this version, so ensure there are no syntax errors or misconfigurations.

  2. Check ‘composefs’ Setup: Since the error log mentions ‘composefs’, ensure that it is properly configured. Verify that all necessary components and dependencies for ‘composefs’ are included in your build. You might want to check if there are any specific instructions or requirements for using ‘composefs’ with Torizon Core 7.1.0.

  3. Review the Update Process: Carefully review the steps you followed to update from Torizon Core 6.8.0 to 7.1.0. Ensure that each step was completed successfully and that there were no interruptions or errors during the update process.

  4. Investigate the ‘/dev’ Directory Issue: The error indicates that ‘/dev’ is missing on the root filesystem. This could be due to a misconfiguration or an incomplete update. Check if the root filesystem is correctly mounted and that all necessary directories, including ‘/dev’, are present. You might need to manually verify the filesystem structure.

  5. Compatibility of ‘torizon-signed’: Since torizon-signed caused build errors in 6.8.0, ensure that it is now compatible with 7.1.0. Look for any updates or patches that might address compatibility issues. It might also be helpful to consult the release notes for Torizon Core 7.1.0 for any changes related to signing.

If these steps do not resolve the issue, you might consider reverting to a previous stable version and reattempting the update, ensuring careful attention to the configuration and update process. This approach can help isolate the problem and ensure a successful update.

I hope this helps, and please feel free to reach out if you have further questions.

Best regards,
Toradex AI Assistant.

Please, also check if these links can help you solve your problem:

Systemd service management on Torizon.

3

When updating OTA to 7.1 without declaring torizon-signed, the systemd is successfully started, but the update is not completed in Torizon Cloud.
The reason is that aktualizr-torizon is terminating abnormally.

Log of aktualizr-torizon

Mar 05 08:39:04 apalis-imx8-07278290 systemd[1]: aktualizr-torizon.service: Failed with result 'exit-code'.
Mar 05 08:39:15 apalis-imx8-07278290 systemd[1]: aktualizr-torizon.service: Scheduled restart job, restart counter is at 55.
Mar 05 08:39:15 apalis-imx8-07278290 (-torizon)[15258]: aktualizr-torizon.service: Referenced but unset environment variable evaluates to an empty string: AKTUALIZR_CMDLINE_PARAMETERS
Mar 05 08:39:15 apalis-imx8-07278290 systemd[1]: Started Aktualizr SOTA Client.
Mar 05 08:39:15 apalis-imx8-07278290 aktualizr-torizon[15258]: Aktualizr version tdx-afedabf0 starting
Mar 05 08:39:15 apalis-imx8-07278290 aktualizr-torizon[15258]: Reading config: "/usr/lib/sota/conf.d/20-sota-device-cred.toml"
Mar 05 08:39:15 apalis-imx8-07278290 aktualizr-torizon[15258]: Reading config: "/usr/lib/sota/conf.d/30-rollback.toml"
Mar 05 08:39:15 apalis-imx8-07278290 aktualizr-torizon[15258]: Reading config: "/usr/lib/sota/conf.d/40-hardware-id.toml"
Mar 05 08:39:15 apalis-imx8-07278290 aktualizr-torizon[15258]: Reading config: "/usr/lib/sota/conf.d/50-secondaries.toml"
Mar 05 08:39:15 apalis-imx8-07278290 aktualizr-torizon[15258]: Reading config: "/usr/lib/sota/conf.d/60-polling-interval.toml"
Mar 05 08:39:15 apalis-imx8-07278290 aktualizr-torizon[15258]: Reading config: "/usr/lib/sota/conf.d/70-reboot.toml"
Mar 05 08:39:15 apalis-imx8-07278290 aktualizr-torizon[15258]: Use existing SQL storage: "/var/sota/sql.db"
Mar 05 08:39:15 apalis-imx8-07278290 aktualizr-torizon[15258]: Couldn`t import data: empty path received
Mar 05 08:39:15 apalis-imx8-07278290 aktualizr-torizon[15258]: could not load sysroot at : loading sysroot: Opening sysroot repo: opening repo: composefs required, but libostree compiled without support
Mar 05 08:39:15 apalis-imx8-07278290 systemd[1]: aktualizr-torizon.service: Main process exited, code=exited, status=1/FAILURE
Mar 05 08:39:15 apalis-imx8-07278290 systemd[1]: aktualizr-torizon.service: Failed with result 'exit-code'.

Since there is no composefs, loading sysroot fails, resulting in an error.

4

Greetings @access,

Before I begin investigating I have some questions to make sure I understood exactly what you did.

  • First you started with Torizon 6.7.0, with no security features enabled.
  • Then you built a Torizon 6.8.0 with tdx-signed and pushed that to our Cloud with TorizonCore Builder
    • Are there any more specifics to how you built this image or was it just inheriting tdx-signed?
  • You then updated from 6.7.0 to 6.8.0 and this succeeded without issue.
  • Next, you built a Torizon 7.1.0 with torizon-signed and pushed that to our Cloud.
    • Again, any more specifics on how you did the build or was it just adding torizon-signed?
  • Finally you updated the bootloader and OS from the 6.8.0 with tdx-signed to the 7.1.0 with torizon-signed, and this is where you got the failure on boot.

Did I understand that all correctly?

When updating OTA to 7.1 without declaring torizon-signed, the systemd is successfully started, but the update is not completed in Torizon Cloud.

To clarify this statement of yours. You created a Torizon 7.1. with no security features enabled (i.e no tdx-signed and no torizon-signed)?

Also, what OS were you updating from to this 7.1?

Best Regards,
Jeremias

5

Hello @jeremias.tx

Thank you for your quick reply.

Q.1. First you started with Torizon 6.7.0, with no security features enabled.

A.1. No. Torizon 6.7.0 with security feature enabled.

There were the following configurations in local.conf of 6.7.0

INHERIT += "tdx-signed torizon-signed"
DISTROOVERRIDES:append = ":cfs-signed"

Q.2. Are there any more specifics to how you built this image or was it just inheriting tdx-signed?

A.2. There were the following configurations in local.conf of 6.8.0

INHERIT += "tdx-signed"
DISTROOVERRIDES:append = ":cfs-signed"

Q.3. You then updated from 6.7.0 to 6.8.0 and this succeeded without issue.

A.3. Yes.

Q.4. Again, any more specifics on how you did the build or was it just adding torizon-signed?

A.4. There were the following configurations in local.conf of 7.1.0 with torizon-signed.

INHERIT += "torizon-signed"

CFS_GENERATE_KEYS="1"
CFS_SIGN_KEYDIR="/work/cfs-key"
CFS_SIGN_KEYNAME="default-fsverity-key"

Q.5. Finally you updated the bootloader and OS from the 6.8.0 with tdx-signed to the 7.1.0 with torizon-signed, and this is where you got the failure on boot.

A.5. Yes.

Q.6. To clarify this statement of yours. You created a Torizon 7.1. with no security features enabled

A.6. There were the following configurations in local.conf of 7.1.0 without torizon-signed

INHERIT += "tdx-signed"

CFS_GENERATE_KEYS="1"
CFS_SIGN_KEYDIR="/work/cfs-key"
CFS_SIGN_KEYNAME="default-fsverity-key"

I apologize for the many ambiguities. I hope I have answered all your questions.

6

I discussed this internally with our team.

The outcome was that going from an image that has tdx-signed to an image that has torizon-signed via an OTA update requires additional steps than just doing an update. These steps still need to be documented and tested by our team. If possible, I would ask that you wait until we have this documentation available.

Another factor is that in your 6.7.0 and 6.8.0 builds, I see that you were using things like torizon-signed and cfs-signed. The rootfs/composefs protection was not complete yet in 6.X and was only officially complete and supported in 7.X images.

In any case, doing an OTA Update to an OS that has composefs enabled from an OS that did not have it enabled originally will require additional steps that are yet to be documented.

Is this item urgent for you and your project?

Best Regards,
Jeremias

7

Hello @jeremias.tx
Apologies for the delayed response. Based on your answer, we discussed it within our team.
Our project requires secure boot and fs-verity, and we believe it is helpful these features can be enabled via OTA.
We hope that addressing this issue will be one of the improvements included in the upcoming quarterly release.

8

Understood, I will try to inform you when the additional documentation and improvements are available. Also keep an eye out for any changes from us regarding this topic.

Just to clarify, and make sure I understand your use-case.

Do you want to perform an OTA update from a Torizon 6.X image with tdx-signed to a Torizon 7.X image with torizon-signed? If not, what is your exact use-case?

If I can understand your use-case I can try to ensure our upcoming documentation will cover this.

Best Regards,
Jeremias

9

Thank you for your reply. Your understanding is right.