Kernel fails to boot after remounting root filesystem as R/W on Verdin iMX8M Plus (Torizon OS 6)

Hello Toradex Community,

I’m developing a product using the Verdin iMX8M Plus Quad 4GB Wi-Fi/BT SOM running Torizon OS 6.
Under normal operation, the system runs with a read-only root filesystem, and specific directories are overlaid (overlayfs) to support Docker.

Recently, I needed to modify some files directly in the root filesystem.
So I performed the following steps:

  1. Remounted the root filesystem from read-only to read/write

  2. Overwrote several files

  3. Executed sync

  4. Rebooted the device

After this, the kernel no longer boots.
The device resets during early kernel boot, and I consistently see the messages shown below.

Boot Log (excerpt)

[... U-Boot logs ...]
Starting kernel ...

...
unable to select a mode
device_remove: Device 'mmc@30b60000.blk' failed to remove, but children are gone
"Error" handler, esr 0xbf000002
elr: ffffffff81b0b100 lr : 000000004020263c (reloc)
...
Resetting CPU ...
resetting ...

Questions

  • What could cause the kernel to fail immediately after switching the root filesystem to R/W and modifying files?

  • Is it possible that switching the rootfs mount mode (or modifying files directly) damaged the integrity of the root filesystem?

  • Are there known constraints or recommended procedures for temporarily enabling R/W on Torizon OS 6’s rootfs?

If any additional information would help (modified files, exact remount commands, OS version string, etc.), I’m happy to provide it.

Thank you in advance for your assistance.

Best regards,
p-uchi

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.


Root Filesystem Protection and Integrity Issues

The kernel boot failure you’re experiencing after modifying files in the root filesystem is likely related to root filesystem protection mechanisms in Torizon OS. Let me explain what’s happening and why.

Understanding Torizon OS Root Filesystem Protection

Torizon OS uses OSTree for filesystem management, which mounts the rootfs as read-only to improve device security and reduce corruption probability (1). Additionally, when Secure Boot is enabled (which may be the case with your image), Torizon OS implements root filesystem protection using composefs technology integrated with OSTree (2).

Why Your Modifications Caused Boot Failure

When root filesystem protection is active, the system uses integrity verification mechanisms (2). The boot failure occurs because:

  1. Integrity Verification: The system verifies the integrity of filesystem components during boot using cryptographic hashes (2)
  2. Modified Files Break Verification: When you overwrote files directly, you changed their content, which breaks the integrity verification process
  3. Boot Process Halts: The kernel fails to boot because the filesystem integrity checks fail

Key Technical Details

The root filesystem protection in Torizon OS provides integrity and authenticity guarantees (2). When composefs is integrated with OSTree:

  • The system maintains cryptographic hashes of files for verification
  • Any unauthorized changes to the root filesystem are detected during boot
  • The protection prevents persistent modifications that could compromise system security

Recommended Recovery and Future Approach

For Recovery:

  1. You’ll likely need to reflash the device with a clean Torizon OS image using Toradex Easy Installer
  2. The filesystem integrity has been compromised and cannot be easily repaired

For Future Modifications:
Instead of directly modifying the root filesystem, use the proper Torizon OS customization methods:

  1. Use TorizonCore Builder for OS-level customizations (1)
  2. Use Container-based Applications for application deployment (1)
  3. Use Device Tree Overlays for hardware configuration changes
  4. Use OSTree’s proper update mechanisms for system changes

Important Note About /etc Directory

If you need to modify configuration files, be aware that with root filesystem protection enabled, the /etc directory becomes volatile - changes made at runtime are lost on reboot (2). This is by design for security purposes.

The system provides mechanisms to make specific files persistent when needed, but this requires proper configuration during image building (2).

Conclusion

Your approach of remounting the root filesystem as read-write and directly modifying files bypassed the integrity protection mechanisms, causing the boot failure. This is actually the security system working as intended - preventing unauthorized filesystem modifications from persisting.

For future development, I recommend using the official Torizon OS customization tools and workflows rather than direct filesystem modifications.

Best regards,
Toradex AI Assistant.


Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

Greetings @p-uchi,

Let me try to address your questions one by one.

What could cause the kernel to fail immediately after switching the root filesystem to R/W and modifying files?

This is really hard to determine. It would help to know what files you modified exactly, also how exactly you switched the filesystem to R/W. Though even with this knowledge it might still be difficult to determine how this caused the issue you saw.

Is it possible that switching the rootfs mount mode (or modifying files directly) damaged the integrity of the root filesystem?

It certainly could be possible, again it’s hard to say without more specific details.

Are there known constraints or recommended procedures for temporarily enabling R/W on Torizon OS 6’s rootfs?

So first of all, I have a question back to you regarding this. Why exactly did you decide to remount/enable R/W on the filesystem to do your changes?

In general we don’t really recommend trying to change the filesystem to R/W at all to do changes. We either recommend using TorizonCore Builder to apply filesystem changes as documented here: Capturing Changes in the Configuration of a Board on Torizon OS | Toradex Developer Center

Or modifying and rebuilding the OS from source via Yocto: Build Torizon OS from Source With Yocto Project/OpenEmbedded | Toradex Developer Center

Either of these would be far less error prone.

Best Regards,
Jeremias