Secure Boot with ECoT: fsverity is enabled again after every offline update

Technical Support
, , ,
1

Hi,

I’m building Torizon with Yocto using BSP 7.7 with enabled secure boot (ECoT). I’m also using the Torizon Cloud to generate a lockbox for offline updates.

When installing such lockbox on a device, that as already enabled secure boot with ECoT, I’m getting a delayed reboot, because fsverity is enabled again on the ostree repo.

After this article this should happen only when running the transitional image for the first time. But I get this log after every offline update:

U-Boot 2024.07-7.6.0-devel+git.3f772959501c (Jan 01 1970 - 00:00:00 +0000)

CPU:   Freescale i.MX8MMQ rev1.0 1600 MHz (running at 1200 MHz)
CPU:   Industrial temperature grade (-40C to 105C) at 48C
Reset cause: POR
DRAM:  2 GiB
Core:  150 devices, 26 uclasses, devicetree: separate
WDT:   Started watchdog@30280000 with servicing every 1000ms (60s timeout)
MMC:   FSL_SDHC: 0, FSL_SDHC: 1, FSL_SDHC: 2
Loading Environment from MMC... Reading from MMC(0)... OK
get_tdx_eeprom: cannot find EEPROM by node
MISSING TORADEX CARRIER CONFIG BLOCKS
get_tdx_eeprom: cannot find EEPROM by node
In:    serial@30860000
Out:   serial@30860000
Err:   serial@30860000
Model: Toradex 0055 Verdin iMX8M Mini Quad 2GB WB IT V1.1F
Serial#: 15506120
SEC0:  RNG instantiated
Net:   eth0: ethernet@30be0000 [PRIME]
Saving Environment to MMC... Writing to MMC(0)... OK
Hit any key to stop autoboot:  0
MMC: no card present
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
970 bytes read in 1 ms (947.3 KiB/s)
## Executing script at 50280000
14093 bytes read in 2 ms (6.7 MiB/s)
Unknown command 'tdx_secboot_get' - try 'help'
## WARNING: Fusing feature not supported by the bootloader.
54 bytes read in 2 ms (26.4 KiB/s)
Applying Overlay: verdin-imx8mm_displaylc_mi1010aqt-1.dtbo
25586213 bytes read in 153 ms (159.5 MiB/s)
## Loading kernel from FIT Image at 50300000 ...
   Using 'conf-freescale_imx8mm-verdin-wifi-LP-11828-04.dtb' configuration
   Trying 'kernel-1' kernel subimage
     Description:  Linux kernel
     Type:         Kernel Image
     Compression:  gzip compressed
     Data Start:   0x503000ec
     Data Size:    11570544 Bytes = 11 MiB
     Architecture: AArch64
     OS:           Linux
     Load Address: 0x48200000
     Entry Point:  0x48200000
     Hash algo:    sha256
     Hash value:   6dbb3e38ffc281949c5a0fd16deb1223494cbf91178e5d0b280a0d248e180f60
   Verifying Hash Integrity ... sha256+ OK
## Loading ramdisk from FIT Image at 50300000 ...
   Using 'conf-freescale_imx8mm-verdin-wifi-LP-11828-04.dtb' configuration
   Trying 'ramdisk-1' ramdisk subimage
     Description:  initramfs-ostree-torizon-image
     Type:         RAMDisk Image
     Compression:  uncompressed
     Data Start:   0x50e23874
     Data Size:    13883439 Bytes = 13.2 MiB
     Architecture: AArch64
     OS:           Linux
     Load Address: 0x52300000
     Entry Point:  unavailable
     Hash algo:    sha256
     Hash value:   c396b02ac56ba44dadb5e784b431b53f4436235c47fc2586f533989d72a88090
   Verifying Hash Integrity ... sha256+ OK
   Loading ramdisk from 0x50e23874 to 0x52300000
## Loading fdt from FIT Image at 50300000 ...
   Using 'conf-freescale_imx8mm-verdin-wifi-LP-11828-04.dtb' configuration
   Trying 'fdt-freescale_imx8mm-verdin-wifi-LP-11828-04.dtb' fdt subimage
     Description:  Flattened Device Tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x50e08f8c
     Data Size:    67736 Bytes = 66.1 KiB
     Architecture: AArch64
     Load Address: 0x50200000
     Hash algo:    sha256
     Hash value:   4483fe5b6e9a9fc0805ad578e1666c9ed1c9012c1fa1e71c1e8ed42412ef726f
   Verifying Hash Integrity ... sha256+ OK
   Loading fdt from 0x50e08f8c to 0x50200000
## Loading fdt from FIT Image at 50300000 ...
   Using 'conf-verdin-imx8mm-secboot-kargs_overlay.dtbo' configuration
   Trying 'fdt-verdin-imx8mm-secboot-kargs_overlay.dtbo' fdt subimage
     Description:  Flattened Device Tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x50e20c68
     Data Size:    558 Bytes = 558 Bytes
     Architecture: AArch64
     Load Address: 0x50240000
     Hash algo:    sha256
     Hash value:   b256969e497d91b778f10065cdc1afa49597f6cb6ff45f9739712bc4b775c8f9
   Verifying Hash Integrity ... sha256+ OK
## Loading fdt from FIT Image at 50300000 ...
   Using 'conf-verdin-imx8mm_displaylc_mi1010aqt-1.dtbo' configuration
   Trying 'fdt-verdin-imx8mm_displaylc_mi1010aqt-1.dtbo' fdt subimage
     Description:  Flattened Device Tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x50e1a39c
     Data Size:    2653 Bytes = 2.6 KiB
     Architecture: AArch64
     Load Address: 0x50240000
     Hash algo:    sha256
     Hash value:   f3aeb973ea106e50597a8fcc5d77363b4d27b10699d23a2e9107327a19a4bd51
   Verifying Hash Integrity ... sha256+ OK
   Booting using the fdt blob at 0x50200000
Working FDT set to 50200000
   Uncompressing Kernel Image to 48200000
   Loading Device Tree to 00000000bcedb000, end 00000000bceeed8d ... OK
Working FDT set to bcedb000

Starting kernel ...

Starting systemd-udevd version 255.21^

Enabling fsverity on the ostree repository - this may take a few minutes.

Progress: [==================================================] (15983/15983)

Enabling fsverity took 288 seconds.
sysroot.readonly configuration va[  291.322093] overlayfs: cannot append lower layer
lue: 0 (fs writable: 1)
composefs+ostree: Validated commit signature using '/etc/ostree/initramfs-root-binding.key'
composefs: Verifying digest: 65d004f246b824d6aa6d280c1579d08386f0c2ecf288cd3b27ebf0228b92e1ab
composefs: mounted successfully

What are your thoughts about this?

Best regards,
Markus

2

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.

Based on the documentation and your description, the behavior you’re experiencing appears to be unexpected. According to the in-field upgrade documentation, the fs-verity enablement process should only occur during the transitional phase when upgrading to ECoT(1).

Expected Behavior

When upgrading from BCoT to ECoT, the process should work as follows:

  1. Transitional Image Boot: The first boot with a transitional torizon-signed image enables fs-verity on the OSTree repository files. This process can take 2-5 minutes and will resume if interrupted(1).

  2. Standard ECoT Image: Subsequent boots with the standard (non-transitional) torizon-signed image should not re-enable fs-verity, as it’s already enabled. The system should simply mount the composefs image successfully(1).

Your Situation

Since you’re experiencing fs-verity enablement after every offline update on a device that already has ECoT enabled, this suggests a potential issue with how the offline update packages are being generated or applied.

Troubleshooting Steps

Before I can provide more specific guidance, could you clarify:

  1. Which exact version of Torizon OS are you building with BSP 7.7? (e.g., 7.7.0, 7.7.x)
  2. Are you building your offline update packages with the transitional configuration (PREP_ROOT_CFS_ENABLED = "maybe" and CFS_UPGRADE_ENABLE = "1") or the standard ECoT configuration?(1)
  3. Are all your offline updates being generated from the same build configuration, or are you switching between different configurations?

The fact that fs-verity is being re-enabled suggests that either:

  • The offline update packages might be built with transitional settings when they shouldn’t be
  • There might be a mismatch in the OSTree deployment metadata

Understanding your build configuration will help identify the root cause.

Best regards,
Toradex AI Assistant.

Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

3

@ToradexAI

  1. I’m using BSP 7.7.0
  2. I’m building the offline update packages with the standard ECoT configuration
  3. The offline updates are generated from the same build configuration

I noticed that in the file “meta-toradex/recipes-core/initramfs-framework/files/composefs,” the function “composefs_ensure_fsverity()” checks whether the currently running deployment already has the “ .ostree.cfs” file secured with fsverity. Since every new OSTree deployment creates a new .ostree.cfs file without this fsverity flag, this triggers fsverity to be re-enabled. Is this a possible explanation?

Best regards,
Markus

4

Thank you for providing those details. Your observation about the composefs_ensure_fsverity() function is insightful and appears to identify a key aspect of the behavior you’re experiencing.

Analysis of Your Observation

Based on the root filesystem protection documentation, when composefs is integrated with OSTree, each deployment is represented by a composefs image file (.ostree.cfs) that holds the directory tree structure and metadata(1). Your observation that each new OSTree deployment creates a new .ostree.cfs file is correct, as this is part of how composefs works with OSTree deployments(1).

Expected vs. Observed Behavior

According to the documentation:

  • During the first boot after upgrading to ECoT with a transitional image, fs-verity should be enabled on the OSTree repository files, which can take 2-5 minutes(2).
  • On subsequent boots with standard ECoT images, the system should simply mount the composefs image successfully without re-enabling fs-verity(2).

However, the documentation also notes that the composefs image contains “expected fsverity hashes of the referenced files”(1), which suggests that fs-verity metadata is integral to each deployment’s composefs image.

Your Specific Case

Since you’re:

  • Building with BSP 7.7.0
  • Using standard ECoT configuration (not transitional)
  • Generating offline updates from the same build configuration
  • Observing fs-verity re-enablement after every offline update

This behavior warrants investigation by the Toradex support team, as it doesn’t align with the documented expectation that fs-verity enablement should only occur during the transitional phase(2).

Your hypothesis about the composefs_ensure_fsverity() function checking the .ostree.cfs file and triggering re-enablement for new deployments is a valuable technical insight that should be reviewed by the development team.

I recommend opening a support ticket with Toradex directly, providing:

  • Your build configuration details
  • The specific BSP version (7.7.0)
  • Your observation about the composefs_ensure_fsverity() function
  • Logs from the offline update process

You may find relevant discussions on the Toradex Community by searching for “composefs fsverity offline update”.

Best regards,
Toradex AI Assistant.

Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

5

Greetings @Mowlwurf,

This might be a side-effect to how offline updates handle OSTree compared to an online update.

As a sanity-check can you repeat your procedure but do an online update instead of an offline update with the same update package. If you do this, do you observe the same redundant re-enabling of fsverity?

This would help narrow down if the issue is with the OS update package itself, or specifically with offline updates.

Best Regards,
Jeremias

6

Hi @jeremias.tx,

thank you for your answer.

I’m going to test the process using online updates. However, this will take some time, since we’ve never used online updates before.

In the meantime, would it be possible for you to test offline updates with Secure Boot + ECoT enabled?

Best regards,
Markus

7

Hi Markus,

Since you have a custom image, the best way to go is to try Jeremias’ suggestion to isolate the issue. If this behaviour is also present with online updates, the issue may be in the image.
However, if this works fine, the issue may be in our offline update implementation. If this is the case, we can focus the investigation here.

Let us know how the tests go.
Best regards

Maximiliano Zapata
Toradex

8

Hi @jeremias.tx,

I was quicker than expected to put the online updates into operation and perform a test update.

It turns out that fsverity is not reinitialized when the system restarts after an online update. This is in contrast to a reboot after an offline update, where the device remains idle for about 5 minutes after the reboot because the initialization of fsverity takes place.

Best regards,
Markus

9

Greetings @Mowlwurf,

Thank you for doing that test. Based on that it seems the issue is somehow unique to offline updates. However, I attempted to do a test on my own setup to confirm and try to reproduce. I got a different results than you.

Here was my setup:

First I did a few Yocto builds using the latest Torizon OS Yocto manifest at the time of writing (versioned 7.8).

  • 1st build was a BCoT image, with the following in my local.conf
INHERIT += "tdx-signed"
TDX_IMX_HAB_CST_DIR = "/workdir/torizon/layers/cst"
  • 2nd build was the ECoT transition image with the following config:
INHERIT += "torizon-signed"
TDX_IMX_HAB_CST_DIR = "/workdir/torizon/layers/cst"
PREP_ROOT_CFS_ENABLED = "maybe"
CFS_UPGRADE_ENABLE = "1"
  • Last build was the final standard ECoT image (I did two builds with this configuration so I had two ECoT images to update between):
INHERIT += "torizon-signed"
TDX_IMX_HAB_CST_DIR = "/workdir/torizon/layers/cst"
#PREP_ROOT_CFS_ENABLED = "maybe"
#CFS_UPGRADE_ENABLE = "1"

All these builds I pushed to Torizon Cloud and made individual offline update Lockboxes for each.

I then did the following on my device:

  • I flashed the BCoT image as my initial image.
  • I provisioned and configured the device for offline updates.
  • First offline update I targeted the ECoT transition image.
  • On reboot from this update I saw the following (expected) logs:
Enabling "verity" feature on the filesystem succeeded.

Enabling fsverity on the ostree repository - this may take a few minutes.

Progress: [==================================================] (10731/10731)

Enabling fsverity took 122 seconds.
  • I then did an offline update to the 1st standard ECoT image.
  • On reboot I did not see the any logs that show fsverity was being “re-enabled”
  • I then did another offline update to the 2nd standard ECoT image.
  • On reboot still no logs that show fsverity being re-enabled.

So based on my test I did not see the “re-enabling” that you saw after every offline update. I only ever saw fsverity being enabled upon update to the transition image where it was expected.

Now the question is what is different about your setup where you saw this.

Can you note any discrepancies or major differences in our procedures?

Did you maybe customize or modify the OS image in any way? For reference my Yocto builds were not customized at all outside of setting the required configurations for BCoT and ECoT.

As a suggestion, maybe try to see if this can be reproduced from a freshly flashed device to see if it’s consistent in your setup. This will also allow you to note down your entire procedure to see if there’s a key difference between your setup and mine.

Best Regards,
Jeremias

10

Hi @jeremias.tx,

thank you for the test. The main difference I see is that I’m using the latest release 7.7.0 and you are using the current working state 7.8.0.

Furthermore I have the following changes in my build setup:

  • In the local.conf I’ve added the following lines:
# path where an offline update is located on the device
TORIZON_SOTA_OFFLINE_PATH = "/tmp/update"

# test config for online updates (default is offline)
GEMAC_ENABLE_ONLINE_UPDATES?= "false"
TORIZON_SOTA_PROV_MODE = "${@'online' if d.getVar('GEMAC_ENABLE_ONLINE_UPDATES') == 'true' else 'offline'}"

# automatic upload to torizon cloud
TORIZON_SOTA_PROV_CREDENTIALS=<path_to_credentials>
SOTA_PACKED_CREDENTIALS = "${TORIZON_SOTA_PROV_CREDENTIALS}
SOTA_HARDWARE_ID = "${MACHINE}"
GARAGE_TARGET_NAME = "cbt3.image"
  • I’ve added the following ostree_%.bbappend to prevent automatic reboots:
# Disable automatic reboot
SYSTEMD_AUTO_ENABLE = "disable"
  • I’ve added the following aktualizr-torizon_%.bbappend for provisioning the device for offline updates:
FILESEXTRAPATHS:prepend := "${THISDIR}/files:"

# configure device for offline updates
SRC_URI += "file://99-offline-updates.toml"
# first-boot one-shot service
SRC_URI += "file://aktualizr-once.service"

# default: offline updates (Lockbox)
GEMAC_ENABLE_ONLINE_UPDATES ?= "false"

do_configure:append() {
	sed -i 's\TORIZON_SOTA_OFFLINE_PATH\${TORIZON_SOTA_OFFLINE_PATH}\g' ${WORKDIR}/99-offline-updates.toml
	sed -i 's\@@GEMAC_ENABLE_ONLINE_UPDATES@@\${GEMAC_ENABLE_ONLINE_UPDATES}\g' ${WORKDIR}/99-offline-updates.toml
}

do_install:append() {
    # configure device for offline updates
    install -m 0755 -d ${D}${sysconfdir}/sota/conf.d 
    install -m 0644 ${WORKDIR}/99-offline-updates.toml ${D}/${sysconfdir}/sota/conf.d/99-offline-updates.toml

    # install first-boot one-shot service
    install -m 0755 -d ${D}${systemd_unitdir}/system/
    install -m 0644 ${WORKDIR}/aktualizr-once.service ${D}${systemd_unitdir}/system/
    # Enable by creating the WantedBy symlink directly in /usr/lib/systemd/system/.
    # This is more reliable than SYSTEMD_AUTO_ENABLE for a sub-package and works
    # under ECoT (torizon-signed) because the symlink lives in the OSTree commit,
    # not in the volatile /etc overlay.
    install -m 0755 -d ${D}${systemd_unitdir}/system/multi-user.target.wants/
    ln -sf ../aktualizr-once.service \
        ${D}${systemd_unitdir}/system/multi-user.target.wants/aktualizr-once.service
}

FILES:${PN} += " \
    ${systemd_unitdir}/system/aktualizr-once.service \
    ${systemd_unitdir}/system/multi-user.target.wants/aktualizr-once.service \
"

SYSTEMD_AUTO_ENABLE:${PN} = "${@'enable' if d.getVar('GEMAC_ENABLE_ONLINE_UPDATES') == 'true' else 'disable'}"
  • with 99-offline-updates.toml:
[uptane]
enable_offline_updates =  true
enable_online_updates = @@GEMAC_ENABLE_ONLINE_UPDATES@@
offline_updates_source = "TORIZON_SOTA_OFFLINE_PATH"
  • and aktualizr-once.service:
[Unit]
Description=Run aktualizr-torizon once (first-boot initialization)
# Only run if aktualizr has not been initialized yet (no state database).
# This covers both fresh installations and OTA updates on existing devices:
# - Fresh install without prior aktualizr state: sql.db absent -> service runs
# - OTA update on existing device: sql.db present -> service skipped
ConditionPathExists=!/var/sota/sql.db
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/aktualizr-torizon once
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

After flashing an image with ECoT I see the expected initial enabling of fsverity. No changes are done on the device afterwards. After the installing a lockbox of the same image but different build the enabling of fsverity happens again.

Best regards,
Markus

11

Hi @jeremias.tx.

I’ve found my error. The ostree_%.bbappend with

# Disable automatic reboot
SYSTEMD_AUTO_ENABLE = "disable"

goes way too far and disables ostree-repo-config.service, too. But this service is important for setting fsverity of the repo from “maybe” to “yes”.

So I changed the bbappend to the following

# Only disable the auto-reboot-after-update mechanism; ostree-repo-config.service
# (and any other unit the vendor recipe adds) must keep the default (enabled).
# SYSTEMD_AUTO_ENABLE only works per package, not per unit, and the postinst
# calls "systemctl enable" directly - a preset file cannot override that.
# Pulling these two units out of SYSTEMD_SERVICE:${PN} is what actually exempts
# them; they stay installed/packaged but are never auto-enabled.
SYSTEMD_SERVICE:${PN}:remove = "ostree-pending-reboot.path ostree-pending-reboot.service"
FILES:${PN} += "${systemd_system_unitdir}/ostree-pending-reboot.path ${systemd_system_unitdir}/ostree-pending-reboot.service"

Now only the auto-reboot service is disabled and the initialization of fsverity only happens once. After installing an offline update the device starts normally.

Thanks for your help.
Best regards,
Markus

12

Glad you were able to narrow it down and find the root cause in your setup. Glad we were able to assist.

Best Regards,
Jeremias