CAAM - best way to make writable aread for key blob

BDSKPEF · August 11, 2025, 6:22am

Hi,
when enabling caam keys in combination with read-only-rootfs i get the following

 localhost tdx-enc.sh[408]: caam: Key blob not found. Creating it...
localhost tdx-enc.sh[429]: mkdir: cannot create directory '/var/local/private': Read-only file system
localhost tdx-enc.sh[408]: /usr/sbin/tdx-enc.sh: line 205: can't create /var/local/private/.keys/tdx-enc-key.blob: nonexistent directory

Should there be overlay filesystem on /var or create /var/local/private as link to writable /data filesystem or patch tdx-enc.sh to store in /data ?

Any suggestions
Best Regards - Paul

ToradexAI · August 11, 2025, 6:23am

Hi, I’m the Toradex AI Assistant.

I might make mistakes. Our support team will step in if needed. If I answered your question, please tick the “Solution” checkbox. For follow-up questions to me, please use @ToradexAI in your post.

Did you like my answer?

Hello Paul,

This is a common issue when using CAAM keys with a read-only root filesystem. By default, the encryption key blob is stored in /var/local/private/.keys/, which is not writable on a read-only rootfs.

There are a couple of ways to address this. Here are the options:

1. Partition-Based Key Storage (Recommended for Read-only Rootfs)

Instead of storing the key in the root filesystem, you can configure the system to store the encryption key blob in a reserved block of your data partition (outside the encrypted volume). To do this, set the following variables in your configuration:

TDX_ENC_KEY_LOCATION="partition"
TDX_ENC_STORAGE_RESERVE="1"

TDX_ENC_KEY_LOCATION="partition" tells the system to use partition-based key storage (not the root filesystem).
TDX_ENC_STORAGE_RESERVE="1" reserves a block for the key blob.

Ensure your TDX_ENC_STORAGE_LOCATION points to the correct partition intended for data-at-rest encryption.

This approach is specifically designed for read-only rootfs systems.

2. Change the Key Directory to a Writable Partition

If you prefer to continue storing the key blob in the filesystem, redirect the storage location to a writable part of your filesystem, such as /data/.keys:

TDX_ENC_KEY_DIR="/data/.keys"

You would then ensure /data is a writable partition and exists before running the encryption setup. This avoids modifying the root filesystem while keeping the key storage flexible.

Overlay or Symlink

While adding an overlay, bind-mount, or symlink from /var/local/private to /data/.keys could work as a workaround, it is typically better to use the configuration options above. This keeps your system maintainable and leverages the built-in flexibility.

Please see the Toradex documentation for more details and examples:

Let me know if you need help with modifying your configuration files or have other questions.

Best regards,
Toradex AI Assistant.

BDSKPEF · August 11, 2025, 6:49am

@ToradexAI
that solves the key generation but next step gives error

 tdx-enc.sh[438]: device-mapper: reload ioctl on encdata (254:0) failed: Device or resource busy

Best Regards - Paul

ToradexAI · August 11, 2025, 6:49am

Thanks for your response.

Please remember to tick the “Solution” checkbox below if my response solves your question. If you have follow-up questions, just continue to tag me using @ToradexAI in your new posts.

Did you like my answer?

Hello Paul,

Thank you for your follow-up and glad the key storage issue is resolved!

The error:

device-mapper: reload ioctl on encdata (254:0) failed: Device or resource busy

indicates that the encdata dm-crypt device is already in use, mounted, or there is a timing conflict during service startup.

Troubleshooting Steps:

Check if encdata is already active:
```
dmsetup ls
lsblk
```
Look for a device named encdata in the output. If it’s there, list the mount points:
```
lsblk -f
```
Unmount and Remove Device if Needed:
If /run/encdata or a similar mount point is in use, unmount it:
```
umount /run/encdata
```
Then, remove the device mapper target:
```
dmsetup remove encdata
```
Restart the Encryption Service (after cleanup):
```
systemctl restart tdx-enc-handler.service
```
Check for Service Conflicts/Timing:
It’s possible that automatic mounting or another service is trying to use the device at the same time as tdx-enc-handler. Double-check that:
- The partition for encryption isn’t being auto-mounted by systemd or /etc/fstab.
- No other service is accessing the device during boot.
There have been recent improvements to address timing issues; ensure you’re using the latest Torizon image and meta-toradex-security layer (see discussion: Device or resource busy error: community thread).
Persistent Issue:
If the error persists after a reboot and following the steps above, please provide:
- Output of lsblk and dmsetup ls
- The content of your relevant encryption configuration (e.g., TDX_ENC_* variables)
- Journal output: journalctl -u tdx-enc-handler.service

Helpful links:

Device or resource busy (community thread)

Let me know the outcome or share more details if the issue continues!

Best regards,
Toradex AI Assistant.

BDSKPEF · August 11, 2025, 6:53am

@ToradexAI
the device to encrypt is /dev/mmcblk0p3 which is already mounted on /data.
Is it a problem, that the .keys directory is on the same partition ?

ToradexAI · August 11, 2025, 6:53am

Thanks for your response.

Please remember to tick the “Solution” checkbox below if my response solves your question. If you have follow-up questions, just continue to tag me using @ToradexAI in your new posts.

Did you like my answer?

Hello Paul,

Yes, this is exactly the core issue. You cannot encrypt a partition that is already mounted and in use—especially when the encryption key is stored on the same partition. This creates a circular dependency and prevents proper setup of dm-crypt.

Here’s why:

/dev/mmcblk0p3 is mounted on /data
The encryption key (.keys) is stored on /data/.keys (same partition)
You’re trying to set up dm-crypt on /dev/mmcblk0p3, but you can’t access the key once the partition is locked, and you can’t set up encryption while it’s mounted

To solve this, you have two reliable options:

1. Use Partition-Based Key Storage (Recommended)
This method stores the key blob in a reserved block outside the encrypted filesystem area, avoiding the circular dependency:

TDX_ENC_KEY_LOCATION="partition"
TDX_ENC_STORAGE_RESERVE="1"
TDX_ENC_STORAGE_LOCATION="/dev/mmcblk0p3"

This approach is designed for exactly this scenario. For more details, see:
https://www.toradex.com/blog/yocto-meta-toradex-security-layer-chuang-jian-jia-mi-shu-ju-fen-qu

2. Store Keys on a Different Writable Partition
If you still want to use filesystem-based storage, set the key directory to a location NOT on /dev/mmcblk0p3. For example:

TDX_ENC_KEY_DIR="/var/persistent/.keys"  # or another partition, not /data

Extra:
If you must perform encryption setup, ensure /dev/mmcblk0p3 is not mounted. You can unmount it and restart the encryption handler:

umount /data
systemctl restart tdx-enc-handler.service

But underlying issue remains: with keys and data on the same partition, future system reboots will break unless using partition-based key storage.

Summary:

Never store keys on the partition you’re encrypting.
Partition-based key storage is the reliable, long-term solution.

If you follow these changes, the device busy error should be resolved!

Best regards,
Toradex AI Assistant.

Reference: How meta-toradex-security addresses encrypted partitions

BDSKPEF · August 11, 2025, 7:17am

@ToradexAI

I can’t read that language - and with auto translate it does not make total sense

ToradexAI · August 11, 2025, 7:18am

Thanks for your response.

Please remember to tick the “Solution” checkbox below if my response solves your question. If you have follow-up questions, just continue to tag me using @ToradexAI in your new posts.

Did you like my answer?

Hello Paul,

Thank you for your feedback! You are correct—the page is in Chinese, and machine translations may not be clear. Let me summarize the key technical points in English for clarity:

How to Solve Key Storage for Encrypted Partitions (/dev/mmcblk0p3 Mounted as /data):

When you want to encrypt a partition that is already used for data (like /dev/mmcblk0p3 on /data), do not store your encryption key (.keys directory) on the same partition. This leads to a circular dependency and the “Device or resource busy” error.

The recommended solution is to use Partition-Based Key Storage.
Here’s how you configure it:

In your Yocto/local.conf or relevant config, set these variables:
```
TDX_ENC_KEY_LOCATION = "partition"
TDX_ENC_STORAGE_RESERVE = "1"
TDX_ENC_STORAGE_LOCATION = "/dev/mmcblk0p3"
```
- TDX_ENC_KEY_LOCATION = "partition" tells the encryption handler to store the key in a reserved block outside the dm-crypt filesystem (not on /data).
- TDX_ENC_STORAGE_RESERVE = "1" reserves a small block (typically 512 bytes) at the end of the data partition for the key.
- TDX_ENC_STORAGE_LOCATION points to the partition you are encrypting.
Before running the encryption setup:
- Make sure /dev/mmcblk0p3 (i.e., /data) is unmounted:
```
umount /data
systemctl restart tdx-enc-handler.service
```
After Encryption:
- The encrypted partition will be available through /dev/mapper/encdata and typically mounted at /run/encdata by default.

This approach avoids storing the key blob on a filesystem that will be locked/encrypted, fully resolving the circular dependency and device busy errors.

Relevant documentation (in English) is also available:

If you need further step-by-step examples for your use case, let me know!

Best regards,
Toradex AI Assistant

bruno.tx · August 11, 2025, 3:05pm

Hello @BDSKPEF,

To summarize, my understanding is that the current scenario is the following:

You are creating an encrypted data partition with meta-toradex-security
You are using a read-only rootfs

And this is still causing issues, correct?
I will try to reproduce this here.

Best Regards,
Bruno

BDSKPEF · August 12, 2025, 6:07am

Hi Bruno,
it might just be that my expectations were wrong.

Because i made the rootfs read-only and needed a writable partition as well i used INHERIT += “tdx-tezi-data-partition” to get the /data partition where i also created the overlay-fs for writing to /etc.

I then expected that when adding the INHERIT += “tdx-encrypted” that i would end up with a /data partition that was encrypted.

But as the default is to store keys in /var/local/private that didn’t work with the read-only rootfs.

So as rootfs (mmcblk0p2) is read-only and /data (mmcblk0p3) is being encrypted, the keys cant be on a filesystem.

TDX_ENC_KEY_LOCATION="partition"
TDX_ENC_STORAGE_RESERVE="1"

did not work - i end up with the kernel in emergency mode

(If /data is on mmcblk0p3 and i manually add mmcblk0p4 in image.json and then chose p4 for encryption i of course get both /data (uncrypted) and /run/encdata partitions.)

Best Regards - Paul

BDSKPEF · August 19, 2025, 6:50am

@ToradexAI
how do i prevent the system also is trying to mount the encrypted partition as normal filesystem during boot?

ToradexAI · August 19, 2025, 6:51am

Thanks for your response.

Please remember to tick the “Solution” checkbox below if my response solves your question. If you have follow-up questions, just continue to tag me using @ToradexAI in your new posts.

Did you like my answer?

Hello Paul,

This is exactly the issue you’re encountering. When using tdx-tezi-data-partition with tdx-encrypted, the system is trying to mount the same partition both as a regular filesystem and as an encrypted device.

The Solution: Configure Automount Settings

The key is to prevent the automatic mounting of the data partition when it’s going to be encrypted. The tdx-tezi-data-partition class has built-in logic for this scenario (1)(2).

Set the following configuration:

INHERIT += "tdx-tezi-data-partition"
INHERIT += "tdx-encrypted"
TDX_ENC_STORAGE_LOCATION = "/dev/mmcblk0p3"
TDX_TEZI_DATA_PARTITION_AUTOMOUNT = "-1"
TDX_ENC_KEY_LOCATION = "partition"
TDX_ENC_STORAGE_RESERVE = "1"

Key Points:

TDX_TEZI_DATA_PARTITION_AUTOMOUNT = "-1" - This prevents the partition from being added to /etc/fstab entirely, so systemd won’t try to mount it during boot (1)(2)
TDX_ENC_KEY_LOCATION = "partition" - Stores the encryption key in a reserved block outside the encrypted filesystem (1)(2)
TDX_ENC_STORAGE_RESERVE = "1" - Reserves one block (512 bytes) for the key blob (1)(2)

Why This Happens

By default, when tdx-encrypted is used with tdx-tezi-data-partition, the system automatically sets TDX_TEZI_DATA_PARTITION_AUTOMOUNT = "-1" to avoid exactly this conflict (1)(2). However, if you’ve explicitly set it to a different value, you need to ensure it’s set to -1 to prevent the boot-time mounting conflict.

After Configuration

Once properly configured:

The raw partition /dev/mmcblk0p3 won’t be auto-mounted
The tdx-enc-handler service will set up dm-crypt on /dev/mmcblk0p3
The encrypted filesystem will be available at /dev/mapper/encdata
It will be mounted at /run/encdata (or your configured TDX_ENC_STORAGE_MOUNTPOINT)

This should resolve both the emergency mode issue and prevent the dual mounting attempts.

Best regards,
Toradex AI Assistant.

BDSKPEF · August 19, 2025, 6:56am

@ToradexAI
no, i now made a separate mmcblk0p4 partition that gets encrypted which is not the TDX_TEZI_DATA_PARTITION. Is there a variable to set mount options for that ?

INHERIT += "tdx-tezi-data-partition tdx-encrypted"
TDX_ENC_STORAGE_LOCATION = "/dev/mmcblk0p4"
TDX_ENC_KEY_BACKEND = "caam"
TDX_ENC_KEY_LOCATION="partition"
TDX_ENC_STORAGE_RESERVE="1"

TDX_TEZI_DATA_PARTITION_AUTOMOUNT = "1"
EXTRA_IMAGE_FEATURES += "read-only-rootfs"
TEZI_DATA_FSOPTS = "-O verity -E nodiscard"

ToradexAI · August 19, 2025, 6:57am

Thanks for your response.

Please remember to tick the “Solution” checkbox below if my response solves your question. If you have follow-up questions, just continue to tag me using @ToradexAI in your new posts.

Did you like my answer?

Let me check the available documentation for configuration variables related to mount options for encrypted partitions.

Based on the available documentation, there are specific variables for configuring mount options for encrypted partitions:

Mount Options for Encrypted Partitions

For your encrypted partition (/dev/mmcblk0p4), you can use the following variables:

TDX_ENC_STORAGE_MOUNT_ARGS = "your_mount_options_here"
TDX_ENC_STORAGE_MKFS_ARGS = "your_mkfs_options_here"

These variables allow you to:

TDX_ENC_STORAGE_MOUNT_ARGS - Set extra arguments passed to mount when mounting the encrypted filesystem
TDX_ENC_STORAGE_MKFS_ARGS - Set extra arguments passed to mkfs.ext4 when creating the filesystem on the encrypted storage

Your Configuration

Your current setup looks correct for having separate partitions:

/dev/mmcblk0p3 - Regular data partition (from tdx-tezi-data-partition)
/dev/mmcblk0p4 - Encrypted partition

The TEZI_DATA_FSOPTS = "-O verity -E nodiscard" you’re using applies to the regular data partition created by tdx-tezi-data-partition, not the encrypted one.

Example Configuration

INHERIT += "tdx-tezi-data-partition tdx-encrypted"
TDX_ENC_STORAGE_LOCATION = "/dev/mmcblk0p4"
TDX_ENC_KEY_BACKEND = "caam"
TDX_ENC_KEY_LOCATION = "partition"
TDX_ENC_STORAGE_RESERVE = "1"

# Mount options for the encrypted partition
TDX_ENC_STORAGE_MOUNT_ARGS = "noatime,nodiscard"
TDX_ENC_STORAGE_MKFS_ARGS = "-O verity -E nodiscard"

TDX_TEZI_DATA_PARTITION_AUTOMOUNT = "1"
EXTRA_IMAGE_FEATURES += "read-only-rootfs"
# This applies to the regular data partition, not encrypted
TEZI_DATA_FSOPTS = "-O verity -E nodiscard"

This way you can set specific mount and filesystem creation options for your encrypted partition separately from the regular data partition.

Best regards,
Toradex AI Assistant.

BDSKPEF · August 19, 2025, 7:12am

@ToradexAI
that results in the encrypted partition not getting mounted on /run/encdata - i only wanted to avoid the attempt to mount mmcblk0p4 directly and the system trying to determine the filesystem type.

ToradexAI · August 19, 2025, 7:13am

Thanks for your response.

Please remember to tick the “Solution” checkbox below if my response solves your question. If you have follow-up questions, just continue to tag me using @ToradexAI in your new posts.

Did you like my answer?

Thanks for your response.

Please remember to tick the "Solution" checkbox below if my response solves your question. If you have follow-up questions, just continue to tag me using @ToradexAI in your new posts.

Did you like my answer?

Hello Paul,

I understand now - you want to prevent the system from trying to mount /dev/mmcblk0p4 directly as a regular filesystem (which causes filesystem detection issues), but you still want the encrypted partition to be properly mounted at /run/encdata.

The Issue

When you create a separate partition (/dev/mmcblk0p4) for encryption, the system may try to auto-mount it during boot before the encryption handler can set up dm-crypt. This causes conflicts and filesystem type detection errors.

Solution: Prevent Auto-mounting of the Raw Partition

To prevent the system from trying to mount /dev/mmcblk0p4 directly while still allowing the encrypted version to mount at /run/encdata, you need to ensure that:

The raw partition is not in /etc/fstab - Since you're creating mmcblk0p4 manually (not via tdx-tezi-data-partition), make sure it's not being added to fstab by any other mechanism.

Use the correct configuration:

INHERIT += "tdx-tezi-data-partition tdx-encrypted" TDX_ENC_STORAGE_LOCATION = "/dev/mmcblk0p4" TDX_ENC_KEY_BACKEND = "caam" TDX_ENC_KEY_LOCATION = "partition" TDX_ENC_STORAGE_RESERVE = "1" Keep the regular data partition auto-mounted TDX_TEZI_DATA_PARTITION_AUTOMOUNT = “1” EXTRA_IMAGE_FEATURES += “read-only-rootfs” TEZI_DATA_FSOPTS = “-O verity -E nodiscard” Don’t add TDX_ENC_STORAGE_MOUNT_ARGS unless you need specific mount options

Ensure proper partition creation in image.json - When you manually add mmcblk0p4 to your image.json, make sure it doesn't have an fstab entry or auto-mount configuration.

Verification

After boot, you should see:

/dev/mmcblk0p3 mounted as /data (regular data partition)
/dev/mmcblk0p4 NOT directly mounted
/dev/mapper/encdata mounted at /run/encdata (encrypted partition)

Check with:

lsblk
mount | grep encdata
systemctl status tdx-enc-handler.service

The key is ensuring that /dev/mmcblk0p4 is never referenced in /etc/fstab or any systemd mount units, so only the encryption handler accesses it.

Best regards,
Toradex AI Assistant.