Rare OTA boot failure on Torizon OS 7.2.0, /etc/fstab: Input/output error` after deployment switch

access · June 10, 2026, 4:29am

Hi Toradex

We have encountered a critical issue with low reproducibility that urgently needs to be resolved. If there are any reports of similar issues or workarounds, please let us know.
We appreciate your cooperation.

Rare OTA boot failure on Torizon OS 7.2.0 (Apalis iMX8QM, secure boot + composefs): `/etc/fstab: Input/output error` after deployment switch

Summary

We are following the Toradex “Secure Boot in Field Upgrades” procedure
(https://developer.toradex.com/torizon/security/secure-boot-in-field-upgrades)
to migrate already-deployed Apalis iMX8QM devices with secure boot enabled and composefs rootfs from a previous OS to Torizon OS 7.2.0. The procedure is two-step:

OTA from the old OS to a transitional 7.2.0 image (signed so that both the old and new SRK accept it).
OTA from the transitional 7.2.0 image to the non-transitional (final) 7.2.0 image.

Step 1 succeeded on every device. In step 2 (transitional → non-transitional), the failure has so far occurred on 2 devices out of many. On those devices, systemd’s fstab and ostree-system generators failed to read /etc/fstab with EIO during early boot, leaving the system unable to start. Greenboot/aktualizr correctly auto-rolled back to the previous (transitional) deployment and the affected devices are now healthy.

The same OTA combination (same transitional source image, same non-transitional target image) was applied to many sibling devices and only these 2 failed. We have been unable to reproduce the failure on the affected devices after rollback, and we are looking for guidance from the community / Toradex on how to root-cause this.

Environment

Hardware: Apalis iMX8QM
OS: Torizon OS 7.2.0
Kernel: 6.6.84
Bootloader: U-Boot 2022.04-6.7.0-devel (not updated by this OTA)
Rootfs: OSTree + composefs (EROFS-backed, verified)
Secure boot: enabled (HABv4, signed FIT image)
OTA agent: aktualizr-torizon
Health-check / rollback: greenboot
Storage: eMMC (single-storage layout, /dev/mmcblk0p1 ext4 = /sysroot)
/etc layout: transient overlay (tmpfs upper) — per Toradex secure boot guide (“when composefs is used, changes are stored on an overlay and are lost upon reboot”)

Failure symptoms (failed boot)

Key kernel/userspace messages from the failed boot:

[    2.992720] overlayfs: cannot append lower layer
[    3.xxxxx] composefs: mounted successfully
[    3.756428] systemd-fstab-generator[456]: Failed to open /etc/fstab: Input/output error
[    3.7xxxx] systemd[1]: Failed to populate /etc with files from /usr/etc: Input/output error
[    3.7xxxx] ostree-system-generator[455]: Failed to open /etc/fstab: Input/output error
...
(boot does not progress; greenboot marks the deployment as failed)

After greenboot triggers rollback:

The last update failed and you are in a rollback state.

Boot is then healthy again on the previous deployment.

What we already ruled out

We pinned the failed deployment with ostree admin pin 0 && ostree admin set-default 0 && reboot and were able to reproduce-and-rollback once for live investigation. After rollback, with both deployments still on disk, we collected the following:

1. eMMC is healthy

mmc info / mmc part are normal
All files in both deployments are readable
dmesg of the healthy boot has no MMC / ext4 errors

2. The `.ostree.cfs` files are NOT corrupted

For the failing deployment vs. the healthy deployment:

	Failing (pending)	Healthy (rolled-back)
OSTree commit	`d9da872b…`	`59a43cf0…`
Version	`7.2.0-devel-20260526180657`	`7.2.0-devel-20260526171504`
`.ostree.cfs` size	3,489,792 B	3,489,792 B
`.ostree.cfs` SHA-256	`fffd2d71…f45489`	`8fb38f69…357eb2`

Both .ostree.cfs files mount cleanly via mount -t erofs and the file trees look complete.

3. `/etc/fstab` content is identical

After rollback, cat-ing /etc/fstab from both deployments under /sysroot/ostree/deploy/torizon/deploy/<hash>/etc/fstab returns byte-identical content. So this is not a content-level corruption.

4. `overlayfs: cannot append lower layer` is a red herring

This message appears in the dmesg of the successful boot too (at [2.326925]), so it is not the cause of the failure — it is a benign message during multi-layer overlay construction.

5. `/etc` is a transient overlay by design

This is the documented Toradex secure-boot behavior, confirmed on the running device:

overlay on /         type overlay (ro,relatime,
                     lowerdir=/run/ostree/.private/cfsroot-lower::/sysroot/ostree/repo/objects,
                     redirect_dir=on,metacopy=on)
overlay on /etc      type overlay (rw,relatime,
                     lowerdir=/sysroot.tmp/usr/etc,
                     upperdir=/run/ostree/transient-etc/upper,
                     workdir=/run/ostree/transient-etc/work,
                     uuid=on)
/dev/mmcblk0p1 on /sysroot type ext4 (rw,relatime)

Kernel cmdline of the healthy deployment:

ostree=/ostree/boot.1/torizon/199364e6b4ce8312673886b6518527b0f1ebe0b47c87754f15f6524347b23d65/0

Our current hypothesis

Because:

the storage and the .ostree.cfs are intact,
the on-disk /etc/fstab is byte-identical to a working deployment,
the failure is not reproducible on the same device after rollback,
many sibling devices boot the same OTA payload without issue,

we suspect a rare race condition in the initramfs stage, somewhere in the construction of /sysroot.tmp/usr/etc (the overlay lowerdir for /etc) or the transient /etc overlay itself, that leaves /etc/fstab momentarily unreadable to systemd-fstab-generator and ostree-system-generator. Once the system rolls back, the condition disappears.

We have not been able to confirm this — it is just where the evidence points.

What we are asking

Has anyone seen systemd-fstab-generator: Failed to open /etc/fstab: Input/output error on Torizon 7.x with composefs + secure boot, especially right after an OTA?
Are there known race conditions in the Torizon initramfs around /sysroot.tmp/usr/etc setup or the /etc transient overlay?
Is there a recommended way to enable a persistent kernel log / initramfs trace on a secure-boot Torizon device so that the next time this happens we can capture the early-boot path without losing logs to the rollback? (We cannot use journalctl -b -1 because there is no persistent journal — /etc/fstab itself was never readable.)
Any suggestions for additional diagnostics we should run on the still-pending failed deployment before we discard it?

Thank you in advance for any pointers.

ToradexAI · June 10, 2026, 4:30am

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.

I understand you’re experiencing a rare but critical boot failure during secure boot ECoT transitions on Apalis iMX8QM with Torizon OS 7.2.0. This is indeed a complex issue involving the intersection of secure boot, composefs, and OTA updates.

Validation of Your Approach

Your procedure follows the documented In-field Upgrades to Secure Boot on Torizon OS correctly (1). The two-step process you’re using (transitional → standard torizon-signed image) is the recommended approach for BCoT to ECoT upgrades (1).

Addressing Your Questions

1. Known Issues with `/etc/fstab` I/O Errors

While the documentation doesn’t specifically mention this exact error pattern, your analysis pointing to a race condition during the initramfs stage is plausible given the complexity of the composefs + secure boot + transient overlay setup.

2. Race Conditions in Initramfs

The documentation confirms that during ECoT transitions, the initial ramdisk performs critical operations including enabling fs-verity on OSTree repository files, which can take 2-5 minutes (1). The process involves:

Enabling “verity” feature on the filesystem
Enabling fsverity on individual OSTree repository files
Creating and mounting the composefs image

This complex initialization sequence could potentially create timing windows where /etc/fstab becomes temporarily inaccessible.

3. Persistent Logging for Secure Boot Devices

For enhanced logging during updates, you can modify the Aktualizr systemd service to enable additional debugging (2). Create or modify /etc/sota/conf.d/99-debug-logging.toml:

[logger]
loglevel = 0

(2)

Additionally, you can enable fuse-specific logging by modifying the aktualizr.service systemd service:

[Service]
...
Environment="FUSE_LOG_ENABLED=1"
...

(2)

This creates an additional log file at /var/lib/rollback-manager/fuse-update.log (2).

4. Additional Diagnostics

Before discarding the failed deployment, consider:

Examining the OSTree commit metadata for any anomalies
Checking if the fs-verity enablement completed successfully on all repository files
Verifying the composefs image integrity using the digest mentioned in boot logs

Rollback Mechanism Validation

Your experience with Greenboot correctly triggering rollback after 3 boot attempts aligns with the documented behavior. The system properly rolled back to the previous version when boot-complete.target failed (2).

Recommendations

Enable Enhanced Logging: Implement the logging configurations mentioned above before attempting the next upgrade
Staged Rollout: Consider upgrading devices in smaller batches to limit exposure
Pre-validation: Ensure the transitional image’s fs-verity enablement completed successfully before proceeding to the standard image

The rarity of this issue (2 out of many devices) suggests it may be related to timing-sensitive operations during the composefs mount or overlay construction phase. The fact that it’s not reproducible after rollback supports the race condition hypothesis.

Best regards, Toradex AI Assistant.

Did you like my answer?

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

jeremias.tx · June 11, 2026, 12:12am

Greetings @access,

Has anyone seen systemd-fstab-generator: Failed to open /etc/fstab: Input/output error on Torizon 7.x with composefs + secure boot, especially right after an OTA?

Internally we’ve never seen this error, nor have we heard reports from other customers about similar errors. Though given that you said this is a rare not reproducible issue it could just be a matter of luck that no one else has seen this. Especially considering a majority of users are not implementing composefs + secure-boot in their products.

Just to clarify, your update procedure you said the procedure was:

Old OS → 7.2.0 transitional → 7.2.0 final

Was this “Old OS” already implemented with composefs + secure-boot?

Was the transitional image needed because you were doing BCoT to ECoT as per our documentation? Which would mean the “Old OS” had secure-boot, but not composefs. Is that correct?

Are there known race conditions in the Torizon initramfs around /sysroot.tmp/usr/etc setup or the /etc transient overlay?

Not that we are aware of. But again, if this is a rare race condition as you suspect it would be difficult for us to know for certain. So it may be possible, and we just don’t know yet.

s there a recommended way to enable a persistent kernel log / initramfs trace on a secure-boot Torizon device so that the next time this happens we can capture the early-boot path without losing logs to the rollback?

Hmm I’m not sure if there is a way to get persistent logs that survive the rollback. Especially during the early boot when no logging is being recorded to disk. Best I can think of at the moment is to set ignore_loglevel earlycon in the kernel command-line. This would enable early kernel messages to be printed to the console. But, this is still only seen on the console and wouldn’t be written anywhere persistent.

Any suggestions for additional diagnostics we should run on the still-pending failed deployment before we discard it?

At the moment I’m discussing with the team. Though, it’s hard to say what else to try given that we have never seen this kind of error before, and we can’t reproduce this on our own devices.

If you could find some way to prove your theory about this race condition that would be helpful. Not sure what could be done as proof though.

Best Regards,
Jeremias

access · June 11, 2026, 1:53am

Hi Jeremias, thanks for the quick reply and for confirming that this error pattern hasn’t been seen internally. Let me answer your two clarifying questions first, and then share what we found after running additional read-back diagnostics on one of the affected devices — the picture has shifted toward a transient eMMC I/O error rather than an OTA / composefs software bug.

Answers to your questions

Q1. Was the “Old OS” already composefs + secure-boot enabled?
No. The pre-OTA OS on these devices had secure boot disabled and composefs disabled. It was a pre-ECoT, pre-secure-boot Torizon image.

Q2. Was the transitional image required for the BCoT → ECoT migration (i.e. old OS had secure-boot but not composefs)?
The transitional 7.2.0 image was used because we are migrating these fielded devices to both secure boot and composefs at the same time, following the Toradex “Secure Boot in Field Upgrades” procedure. So in our case the transition is not “secure-boot-only → secure-boot+composefs”, it’s “no-secure-boot, no-composefs → secure-boot + composefs”, performed via the documented two-step (transitional → non-transitional) flow. Step 1 (old → transitional) succeeded everywhere; the failure shows up in step 2 (transitional → non-transitional) on the 2 devices.

New diagnostic results from one of the affected devices (after rollback)

Both deployments are still on disk, so we ran cold-cache read-back of the failed deployment’s .ostree.cfs and collected eMMC health info.

1. The failed .ostree.cfs is fully readable now (cold cache).

$ sync && echo 3 > /proc/sys/vm/drop_caches
$ dd if=/.../d9da872b.../.ostree.cfs of=/dev/null bs=4k status=progress
3489792 bytes (3.5 MB) copied, 0.063 s, 55.2 MB/s
$ # mount + walk every file inside the image
$ find /mnt/cfs-failed -type f -exec cat {} + > /dev/null 2>read-errors.log
$ wc -l read-errors.log
0 read-errors.log

→ Zero I/O errors. The same .ostree.cfs that produced Failed to open /etc/fstab: Input/output error at boot is now read through end-to-end without a single read error.

2. The .ostree.cfs image was never corrupted on disk.
The fs-verity Merkle digest of the failed image, the digest stored in OSTree commit metadata under ostree.composefs.digest.v0, and the digest printed in the failed boot’s dmesg (composefs: Verifying digest: ebc954…) all match exactly:

ostree.composefs.digest.v0  : ebc9545602e77c65aecf680f56433a2bcca2701b4bfe0a52b6590e2f376820da
failed-boot dmesg           : ebc9545602e77c65aecf680f56433a2bcca2701b4bfe0a52b6590e2f376820da
fsverity measure (now)      : ebc9545602e77c65aecf680f56433a2bcca2701b4bfe0a52b6590e2f376820da

So the image bytes on disk are identical to what was originally written, and identical to what fs-verity hashed at the failing boot. There is no on-disk corruption.

3. The /etc/fstab file itself does not have fs-verity enabled — and that is the same on the healthy deployment.

$ lsattr  .../d9da872b.../etc/fstab     # failed deployment
--------------e-------  .../etc/fstab
$ lsattr  .../59a43cf0.../etc/fstab     # healthy deployment
--------------e-------  .../etc/fstab
$ fsverity measure .../d9da872b.../etc/fstab
ERROR: FS_IOC_MEASURE_VERITY failed: No data available
$ fsverity measure .../59a43cf0.../etc/fstab
ERROR: FS_IOC_MEASURE_VERITY failed: No data available

So whatever the boot-time EIO was, it was not an “fs-verity is half-enabled on /etc/fstab on one deployment but not the other” situation. Both deployments look the same from this angle. The actual read path is read("/etc/fstab") → overlayfs (/etc, lowerdir = /sysroot.tmp/usr/etc) → composefs → erofs on .ostree.cfs → fs-verity on .ostree.cfs → ext4 read of .ostree.cfs. Where things broke must have been somewhere along that chain at the failed boot, but the chain works perfectly now.

4. eMMC info on the affected device.

name        : IX2932            (Apalis iMX8QM eMMC)
manfid      : 0x000070          (Micron)
oemid       : 0x0100
date        : 03/2024
fwrev       : 0x8a00000000000000
serial      : 0x39637bd4
life_time   : 0x01 0x01         (0–10 %, normal)
pre_eol_info: 0x01              (Normal)

ext4 features on /dev/mmcblk0p1 include verity, state clean, no errors. Current-boot dmesg has no mmc / ext4 / fs-verity errors at all.

Why this points at a transient eMMC I/O error

We have a separate, previously reported case of a transient eMMC I/O failure on the same Apalis iMX8QM platform with the same eMMC part (Micron IX2932, manfid 0x000070). That case was reported to Toradex as a private forum post here:

→ https://community.toradex.com/t/emmc-transient-failure-report/30214

In that earlier case the failure surfaced very differently — mmc0: Card stuck being busy! → cache-flush timeout (-110 ETIMEDOUT) → HW reset failed → ext4 journal abort → read-only remount — but the shape is the same: a single device, a transient I/O error, and full recovery after a power cycle with health indicators still in normal range.

Putting it together:

The failed .ostree.cfs is bit-perfect and fully readable now.
The same image triggered an EIO during one specific boot, on one specific device, which then never reproduced on that device again after rollback.
The platform has a prior, separately reported transient eMMC failure on the same eMMC part with normal SMART-equivalent health values.
Linux 6.6 + composefs + fs-verity is much stricter than plain ext4: even a one-off bad read of an .ostree.cfs block can fail Merkle verification and surface as EIO, which then propagates up through erofs / overlayfs and reaches systemd-fstab-generator as Failed to open /etc/fstab: Input/output error. In the older non-composefs case the same kind of one-off read error showed up at the ext4 layer (read-only remount); in the composefs case it shows up at the verified-overlay layer.

So our current best-supported hypothesis is not an OTA or initramfs-stage software bug, but a rare transient eMMC I/O error that hits the .ostree.cfs read path during the first boot of the new deployment, and is therefore amplified by fs-verity into a hard EIO that systemd cannot recover from, instead of being silently retried as a plain disk hiccup would have been on a non-verity filesystem.

Updated questions for Toradex

Given the above, the questions that would help us most are:

Are you aware of any transient I/O / busy-hang reports on Apalis iMX8QM modules with the Micron IX2932 eMMC part (manfid 0x000070), particularly modules manufactured around early 2024? Internal data points or service-bulletin-level information would be very valuable, even if it’s only “we’ve seen this rarely”. The previously filed private report referenced above is from the same module family.
Inside the composefs / erofs / fs-verity path on Torizon 7.2 (Linux 6.6), is there any retry on a transient block-read error before EIO is propagated to userspace? If a single bad read of an .ostree.cfs block is enough to brick a boot until rollback, we’d like to understand whether that’s by design or whether enabling some kernel knob would soften it.
For our next OTA wave we plan to enable persistent journaling and add ignore_loglevel earlycon to the kernel command line as you suggested. Are there any other early-boot logging knobs (e.g. for the initramfs ostree-prepare-root / composefs-mount stage specifically) that are worth turning on so that, if this happens again, we can capture the exact failing read?
If the eMMC angle pans out, do you have a recommendation for a “post-OTA fs-verity sanity sweep” we could run on the device right after the new deployment is staged but before reboot — something that would proactively read every block of .ostree.cfs (and ideally every fs-verity-protected file in the new deployment) so that any transient read error gets observed (and retried by the eMMC layer’s normal mechanisms) before greenboot has to make a pass/fail decision?

Thanks again for engaging on this — even confirming “we haven’t seen this internally” is useful, and the pointer to ignore_loglevel earlycon is appreciated.

jeremias.tx · June 11, 2026, 10:38pm

Are you aware of any transient I/O / busy-hang reports on Apalis iMX8QM modules with the Micron IX2932 eMMC part (manfid 0x000070), particularly modules manufactured around early 2024?

Do you mean the Kingston IX29? Looking at our product history IX29* is typically associated with Kingston not Micron: Apalis iMX8 | Toradex Developer Center

In any case, I couldn’t find any past tickets or support cases revolving around this eMMC part. Our newer hardware revisions stopped using this specific part. But, this was not due to any technical issues, rather supply-chain issues.

Inside the composefs / erofs / fs-verity path on Torizon 7.2 (Linux 6.6), is there any retry on a transient block-read error before EIO is propagated to userspace?

Something higher level like composefs wouldn’t really concern itself with low level reading errors like this. Typically retires on this level would be the responsibility of the emmc/block device driver.

For our next OTA wave we plan to enable persistent journaling and add ignore_loglevel earlycon to the kernel command line as you suggested. Are there any other early-boot logging knobs (e.g. for the initramfs ostree-prepare-root / composefs-mount stage specifically) that are worth turning on so that, if this happens again, we can capture the exact failing read?

As per the commit here: initramfs-framework: improve debugging of composefs enabling · torizon/meta-toradex-torizon@7e83c71 · GitHub

You can also add verbose to the kernel command-line. To add additional log output to the initramfs module responsible for composefs bring-up.

If the eMMC angle pans out, do you have a recommendation for a “post-OTA fs-verity sanity sweep” we could run on the device right after the new deployment is staged but before reboot — something that would proactively read every block of .ostree.cfs (and ideally every fs-verity-protected file in the new deployment) so that any transient read error gets observed (and retried by the eMMC layer’s normal mechanisms) before greenboot has to make a pass/fail decision?

There is the general ostree fsck command: ostree-fsck(1) — ostree — Debian testing — Debian Manpages

I’m not 100% sure how well it works with a composefs protected OSTree deployment, but it may provide some assurance at the least.

Best Regards,
Jeremias

access · June 19, 2026, 1:40pm

The eMMC hypothesis turned out to be wrong. We reproduced the failure on demand by stripping fs-verity from a single ostree object (/etc/fstab’s backing .file) and rebooting: signed-mode composefs returns EIO on that file, exactly matching the field symptom.

The real cause is that an OTA can leave isolated ostree objects without fs-verity (content-addressed reuse via skip-if-exists does not re-run fsverity enable), and none of the narrow canaries in init.d/94-composefs (lockfile / .ostree.cfs missing verity / CFS_UPGRADE_ENABLE per-deployment os-release scan) catches such an object. Setting CFS_UPGRADE_ENABLE=1 as recommended by the “Secure Boot in Field Upgrades” page is therefore not sufficient for our case: composefs_enable_fsverity_files() runs only when one of those canaries fires, and the per-deployment os-release check does not see an isolated verity-less file elsewhere in the repo.

The unconditional canary CFS_ALWAYS_ENABLE=1 does cover this case, but it is not exposed in the same way: in recipes-core/initramfs-framework/initramfs-framework_1.0.bbappend only CFS_UPGRADE_ENABLE is declared as a recipe variable and sed-substituted into the script, while CFS_ALWAYS_ENABLE is hardcoded to 0 in files/composefs with no substitution hook. Setting CFS_ALWAYS_ENABLE = "1" in local.conf therefore has no effect on the FIT-embedded script. We have worked around this with a project-local bbappend patch.

Questions

Is there a reason CFS_ALWAYS_ENABLE is intentionally not user-configurable (the script comment says “mostly for debugging purposes”)? If so, what is the recommended way for an integrator to guarantee fs-verity coverage on every object after an OTA, given that CFS_UPGRADE_ENABLE=1 alone leaves isolated verity-less objects unrepaired?
Would Toradex consider exposing CFS_ALWAYS_ENABLE the same way CFS_UPGRADE_ENABLE is (declare it in the bbappend and sed-substitute it into the script), so projects do not need a downstream patch?

Thanks.

rogerio.tx · June 22, 2026, 5:15pm

Hi @access

The real cause is that an OTA can leave isolated ostree objects without fs-verity (content-addressed reuse via skip-if-exists does not re-run fsverity enable), and none of the narrow canaries in init.d/94-composefs (lockfile / .ostree.cfs missing verity / CFS_UPGRADE_ENABLE per-deployment os-release scan) catches such an object.

Can you explain how you concluded that OTA can leave isolated ostree objects without fs-verity?

The way things are supposed to work is:

With the transitional image , we switch to a version of ostree that has support for composefs/fs-verity; when booting that image, the script in the initramfs is responsible for enabling fs-verity in the whole ostree repository (and at the end on the .ostree.cfs file).
After the transition when running a normal (i.e. non-transitional image) , the code to enable fs-verity is no longer needed because ostree has composefs support enabled and when it downloads new objects it automatically enables fs-verity on them.

The above item explains why CFS_ALWAYS_ENABLED is not exposed to users: it is not supposed to be needed.

Since fs-verity enabling only happens during the transition and upon downloads, if you drop fs-verity data from files, no process would run to fix that situation. Therefore, your test produces the expected failure result. But it isn’t supposed to ever happen in practice.

If ostree is leaving files without fs-verity data during OTA after the transition, then this is a bug that we’d need to address.

If you don’t make this manual removal of fs-verity from /etc/fstab does the problem happen during an update to the final non-transitional image?

I think a way to confirm your hypothesis would be to do an OTA update where the /etc/fstab file (or any other file under /etc) is different from the previous deployment (i.e. compared to the transitional image). If the failure happens, the existence of a bug would be confirmed.

We may try to confirm this on our side, but it may take some time. So, if you could confirm it, it would be great.

Thanks and best regards,

Rogerio

access · June 23, 2026, 8:17am

Hi Rogerio, thanks for the explicit design intent — that clarifies what is supposed to happen on a non-transitional image. We accept that CFS_ALWAYS_ENABLE is not meant to be exposed and will not push on that further.

How we concluded that the failing device had verity-less objects (Method A was confirmation, not discovery)

The synthetic strip (“Method A”) is not how we arrived at the hypothesis — it was used afterwards to close the causal loop. The hypothesis came from a direct measurement on one of the two field-affected devices, taken before any manual modification.

One caveat first: the failure is rare. We have run the same transitional → non-transitional OTA combination many times across several devices and have observed this /etc/fstab EIO on only 2 occasions in total — the two times in the original report. On every other device a post-OTA fsverity measure sweep shows every .file object verity-enabled, matching the design intent you described. We are not claiming ostree drops fs-verity on every post-transition OTA, only that it does so on a rare subset we cannot yet trigger deterministically.

On the affected device, with both deployments still on disk after rollback, we walked /sysroot/ostree/repo/objects/**/*.file and ran fsverity measure on each:

~20 % of the sampled .file objects returned FS_IOC_MEASURE_VERITY: No data available (ENODATA) — i.e. the object exists in the repo but has no fs-verity digest attached.
The .file backing /etc/fstab (object 2bcd285f…) was in that verity-less subset.
The .ostree.cfs of the failing deployment did have fs-verity, and its digest matched ostree.composefs.digest.v0 exactly.

So the device was in an inconsistent state: the composefs image’s own fs-verity was fine but a non-trivial fraction of the objects it points at lacked fs-verity. That is the failure mode signed-mode composefs is allergic to — overlayfs reading a lower-layer object without fs-verity returns EIO, which propagates up to systemd-fstab-generator.

After observing that 20 % gap on the field device, we used the synthetic strip on a separate clean device to confirm that this single condition (one verity-less /etc/fstab object, everything else intact) is sufficient to reproduce the exact boot symptom. The chain:

Field observation: real device, real failed deployment, ~20 % of .file objects (incl. /etc/fstab’s backing object) lack fs-verity.
Synthetic confirmation (Method A): inject only that one condition on a clean device; boot fails identically.

That’s why we believe the OTA path on a non-transitional image is, under some conditions, importing .file objects into the local repo without running fsverity enable on them. We don’t yet know which ostree code path is responsible.

The experiment you suggested

The /etc/fstab-modified OTA test sounds like a good way to probe the hypothesis, and we will consider how to fit it into the OTA cycles we are already running. But given the failure has only surfaced twice across many OTAs, a small number of runs is unlikely to reproduce the verity-less state and a negative result wouldn’t rule the bug out — so please treat this as “we’ll consider it” rather than a commitment.

What would help most from Toradex

Is the non-transitional ostree+aktualizr path on Torizon 7.2 supposed to always fsverity enable every imported object — including objects content-addressed-reused from a prior deployment that did not itself have verity attached?
Are you aware of any past upstream ostree change around fsverity enable on skip-if-exists reuse, or on ostree-finalize-staged.service’s hardlink path, that could relate to this? Even a pointer to a relevant commit/PR would help us narrow the search.

Thanks again for the careful response.

rogerio.tx · June 23, 2026, 9:35pm

Hi again @access

Let me start by answering your questions:

Is the non-transitional ostree+aktualizr path on Torizon 7.2 supposed to always fsverity enable every imported object — including objects content-addressed-reused from a prior deployment that did not itself have verity attached?

My understanding is that during an (OTA) update, ostree will enable verity on all objects it imports into its local repo when pulling the new commit. That is, it would only enable verity on objects it needed to fetch (as a dependency of the commit), not those reused because those are already present in the local repo. That is the reason we enable verity on all files with the transitional image.

I’m personally not aware of upstream changes or bugs that could explain the observed problem. Searching the web now, I couldn’t find anything particularly related to that.

Now, let me also offer an idea if you want to further investigate the issue on your side:

If the state of the failed devices was kept, you could traverse the whole ostree repository determining the following pieces of information for each object:

A: Is the object used by the previous deployed commit?
B: Is the object used by the newly deployed commit?
C: Does the object have verity information?

These would be boolean variables giving 8 possible combinations. You could then count how many times each combination happens and make a table out of that. That information could help in identifying if the problem happened during the transition or after it.

For example, if all cases where C==No have also A==No and B==Yes, then for sure the problem happened during the ostree pull of the new commit, which would rule out problems with the initramfs code responsible for the transition. With that confirmation we could more confidently focus on finding some path in libostree that could leave behind fetched objects without verity data…

Best regards,
Rogerio