Guidance on meta-toradex-security: encryption

Technical Support
, , , , ,
1

Hello,

I am trying to enable encryption for our product which is based on a verdin-imx8mp module. I got secure boot working with the meta-toradex-security layer with the inheritance of tdxref-signed which provides me an extended chain of trust (ECoT).

The next step we want to take is to encrypt the signed (read-only) root filesystem, and later on we want to encrypt a separate data partition. In the meta-toradex-security layer I have read the page: https://github.com/toradex/meta-toradex-security/blob/kirkstone-6.x.y/docs/README-encryption.md
However some things are not yet clear to me or needs some clarification in combination with our current implementation. We are working with the A/B partitioning which means we have duplicate boot partitions (containing the signed fitimage with the tdx-reference-ramdisk-image), two root filesystem (containing the verity images), and some other data partitions. This is to provide fail-safe updating of the system at a later stage. I added the INHERIT += “tdx-encrypted” variable together with other required variables to enable the encryption, for now I will first start with the TDX_ENC_KEY_BACKEND=“cleartext” to get a correct understanding. The same applies for the TDX_STORAGE_LOCATION, I currently set it to the A root filesystem partition but I think I will need some extension here later to also support A/B partitioning.

When building the image I see that the tdx-enc.sh script and the services are being installed to the our image/root filesystem, caused by the IMAGE_INSTALL in the tdx-encrypted.bbclass. From my point of view, I would expect this to be appended to the tdx-reference-ramdisk-image since this decrypts the root filesystem, verifies the signature and will then mount it.

  1. Is this correct and should I adapt this?

Furthermore I am questioning if there is any order for the dm tools, is dm-crypt performed before dm-verity or otherwise (e.g. first decrypt or first verify).

  1. What is the correct order here? Or do you maybe have a boot log of a system where this already is working, it would be very useful as a reference.

Apart from this implementation questions I have a question regarding the keys. I understand that a symmetric key solution is chosen and this is stored in a secure element such as the CAAM module using the trusted keys feature.

  1. Do I need to provide a key to the system at installation point?
  2. Or does it generate and load a key itself and will the key be never known to the outside world? (as I saw in tdx-enc.sh tdx_enc_key_gen_caam/cleartext/tpm).

If this last one is true then I guess it could make sense to generate this key at installation and then encrypt partitions A and B at the same time with the common key, since I want to be able to update the B partition also at a later stage.

  1. Do you know if for example swupdate provides a feature to use the CAAM and dm-crypt?

I am using kirkstone-6.x.y branch from meta-toradex-security with hash: 8389b4b60725b2804afd790cb194affd73f9b84b

Thanks in advance!
Kind regards,
Randy

2

Hi, I’m the Toradex AI Assistant. I’m in early beta, and I would appreciate your feedback.
The Toradex Support team will follow up with you shortly!

Did you like my answer?

:+1: :-1:

Hello,

Enabling encryption on your Verdin iMX8MP module with the meta-toradex-security layer is a significant step towards enhancing your product’s security. Let’s address your queries one by one:

  1. Regarding the placement of the tdx-enc.sh script and services: Your observation is insightful. Typically, the decryption mechanisms should be part of the initial boot process to ensure the root filesystem is decrypted before mounting. If your current setup involves appending these to the root filesystem instead of the ramdisk image used for booting, you might need to adjust this. The scripts and services responsible for decryption should ideally be included in the tdx-reference-ramdisk-image or an early boot partition to ensure they are executed before the root filesystem mount process.

  2. Order of dm-crypt and dm-verity: The correct order is to first decrypt the filesystem with dm-crypt and then verify its integrity with dm-verity. This sequence ensures that the data being verified is in its decrypted state, matching the hashes generated at build time for the verity metadata.

  3. Regarding the keys: For systems utilizing a secure element like the CAAM module, the key handling process is designed to enhance security. Typically, the key is generated and loaded into the secure element without being exposed externally. This approach ensures the key remains confidential and is managed within the secure boundaries of the hardware.

  4. Key generation and loading: The system usually generates and loads the key internally, especially when using features like CAAM. The key is not known outside the secure element, aligning with the secure handling practices you’ve observed in the tdx-enc.sh script.

  5. Encrypting partitions A and B with a common key: Generating the key at installation and using it to encrypt both partitions is a practical approach, especially for systems with A/B partitioning. This method facilitates seamless updates by ensuring both partitions are accessible with the same key. Regarding swupdate’s capability to use CAAM and dm-crypt, swupdate is designed to be flexible and can integrate with various encryption and secure boot mechanisms. However, specific support for CAAM and dm-crypt would depend on the implementation details and how swupdate is configured in your environment. It’s advisable to review swupdate documentation and sources for features related to secure key storage and encryption mechanisms.

For your setup, focusing on the integration of encryption with A/B partitioning and ensuring the secure handling of encryption keys through mechanisms like CAAM are crucial steps. As you proceed, consider the system’s

Please, also check if these links can help you solve your problem:

Build and Run the Amazon SageMaker Edge Manager Demo on Torizon | Toradex Developer Center.

3

Hi,

One question I would have is are you building the Torizon image or a custom one based on the reference images?
I ask as the security layer is developed primarily for Torizon.

For some good info on the encryption side of things you could look at issue 20 and some of the recent closed ones.
I have been trying to help in the area of the tdx-enc-handler.
One big thing to note is that currently you should not enable the handler if you intend to blow the secure boot fuse (last one mentioned in the fuse.txt file in the image).
Before that is blown a test key is used, after blowing the fuse a real one is used rendering any previous encrypted partition useless.

Also note that in the initial setup of the CAAM key there is a blob file written to disk which is needed to loads the trusted key for use by dm-crypt.
In the case of tdxref-signed class, the rootfs will be read-only and this file will fail to be written.
I am unsure if it is possible to write this blob into the ramdisk image at install time (which is where it would need to be to encrypt your roots).
However, this is also something I will be digging into deeper soon for my use case.

Not sure if any of that helps.

Regards,
MMG

4

Hi,

Thank you for your reply.

I am not directly using Torizon OS but use a custom solution which is strongly inspired on Tezi, thats also how I got secure boot working with some minor adjustments.

I have looked at issue #20 but I guess that is more regarding secure boot and u-boot hardening. I am looking for more information the encryption side.

It makes sense what you tell about the blob file which gets written to the read-only root file system, but does that then mean that tdx-encrypted and tdxref-signed are incompatible with each other?

5

Greetings @rb92nl,

Let me help clarify what I can.

When building the image I see that the tdx-enc.sh script and the services are being installed to the our image/root filesystem, caused by the IMAGE_INSTALL in the tdx-encrypted.bbclass. From my point of view, I would expect this to be appended to the tdx-reference-ramdisk-image since this decrypts the root filesystem, verifies the signature and will then mount it.

Is this correct and should I adapt this?

The way we have the encryption feature implemented it’s a sort of standalone with regards to the rest of the chain of trust. Which is why it doesn’t really care about the ramdisk in this case, or any other part of the chain.

It simply takes a partition/filesystem location encrypts it and mounts it elsewhere on the system. If you want it to serve a different/extended purpose then you would need to modify it, I think given what you’ve described. That said meta-toradex-security is open-source and takes contributions. It would be appreciated if you could contribute back to the meta-layer if possible.

Furthermore I am questioning if there is any order for the dm tools, is dm-crypt performed before dm-verity or otherwise (e.g. first decrypt or first verify).

What is the correct order here? Or do you maybe have a boot log of a system where this already is working, it would be very useful as a reference.

As I said above we designed the encryption feature as a sort of standalone functionality. As it stands out we use dm-verity to do signature checking of the root file system. Afterwards via the systemd service the encryption script runs.

Apart from this implementation questions I have a question regarding the keys. I understand that a symmetric key solution is chosen and this is stored in a secure element such as the CAAM module using the trusted keys feature.

Do I need to provide a key to the system at installation point?
Or does it generate and load a key itself and will the key be never known to the outside world? (as I saw in tdx-enc.sh tdx_enc_key_gen_caam/cleartext/tpm).

You do not. The key is generated based on the backend you use as seen in the script: meta-toradex-security/recipes-core/tdx-enc-handler/tdx-enc-handler/tdx-enc.sh at kirkstone-6.x.y · toradex/meta-toradex-security · GitHub

CAAM for example generates a test key until the device is closed (via secure boot). In which case it will then use a OPTMK key which is unique and fused into each SoC at manufacturing time as described here: meta-toradex-security/docs/README-encryption.md at kirkstone-6.x.y · toradex/meta-toradex-security · GitHub

Do you know if for example swupdate provides a feature to use the CAAM and dm-crypt?

Is swupdate your current update solution?

I’m not very familiar on swupdate, but I did a quick search regarding swupdate and CAAM. I did not find much concrete details. There’s this thread from NXP but seems kind of a dead-end: https://community.nxp.com/t5/i-MX-Processors/IMX6Q-encryption-swupdate/m-p/760105

Best Regards,
Jeremias

6

That depends. In my case I also use the tezi Data partition class and change the variable with the encryption blob path to point to the /data directory.
This way the blob can be written and I believe (have not actually tested it yet as only triggered a new image build today) that should allow both to be possible.

For some extra research the original encryption stuff I helped with was in PR#26 [#25 Preserving original data on encrypted partition. by MMG-DG · Pull Request #26 · toradex/meta-toradex-security · GitHub]

However, I am not sure if that will be of any use from the ramdisk image. But I expect you could have that image based off a custom reference image with just the encryption bits.

7

Hi Jeremias,

Thank you for your reply. Your reply actually clarifies a lot and with having the encryption as a standalone implementation in mind it all makes more sense. However I still want to combine the chain-of-trust with encryption of the root filesystem and I think this is possible with some modifications.

As you explained the key is generated on the device itself and stays there. I however have some questions about the “key blob” which is saved to /var/local/private/.keys/tdx-enc-key.blob when generating the key in CAAM. As I understand this is an encrypted blob based on the key in the CAAM, which is then imported in the kernel keyring and then used to create an encrypted partition and is later used to mount the encrypted partition.
Regarding the blob:

  1. Is the “blob” comparable to a public key and is the “private key” itself in the CAAM?
  2. Would it be possible to retrieve the “blob” each time at boot from the CAAM and then load it in the keyring?
  3. Or does it really require always the same “blob” (and thus needs to be stored at a non-volatile location)?

You do not. The key is generated based on the backend you use as seen in the script: meta-toradex-security/recipes-core/tdx-enc-handler/tdx-enc-handler/tdx-enc.sh at kirkstone-6.x.y · toradex/meta-toradex-security · GitHub
CAAM for example generates a test key until the device is closed (via secure boot). In which case it will then use a OPTMK key which is unique and fused into each SoC at manufacturing time as described here: meta-toradex-security/docs/README-encryption.md at kirkstone-6.x.y · toradex/meta-toradex-security · GitHub

Regarding the closure of the device with the fuses, I understand that when the device is closed the OPTMK will become active and the keys which were used earlier are invalid for use.

I’m not very familiar on swupdate , but I did a quick search regarding swupdate and CAAM. I did not find much concrete details. There’s this thread from NXP but seems kind of a dead-end: https://community.nxp.com/t5/i-MX-Processors/IMX6Q-encryption-swupdate/m-p/760105

Regarding the update mechanism, I think we will have to look into some pre-install script where we first decrypt the partition to write to (partition B when A is active and otherwise) before flashing the other partition but I will look into that later.

Thank you for your effort on explaining/clarifying this, I really appreciate it.

Kind regards,
Randy

8

Before we go further with this, maybe you can help clarify something for me. For what specific reason do you want to encrypt your entire rootfs for?

Most of the times, it doesn’t make much sense to encrypt the complete rootfs. Most software there is open-source anyways, so there is no need to protect its confidentiality. And doing so can cause issues like performance hits and Tivoization (if GPLv3 is being used).

Furthermore if there’s a specific IP to protect (usually one’s application code/binary), then it would make more sense to move this sensitive information to it’s own partition and encrypt this, rather than encrypt the entire rootfs.

Of course there are some merits and reasons to encrypt the entire rootfs. This is why I’m asking you though so I can understand better where you are coming from.

Best Regards,
Jeremias

9

Hi Jeremias,

The end-usage of this device is a medical device class III. This requires strict regulations in terms of risk management and cyber security aspects. That is why we would go for the full disk encryption to mitigate risks.

I could imagine that full disk encryption might be overhead and could have some performance hits. Thats already why we have decided to only encrypt the root file system itself instead of also the boot partitions with the fitImage. Maybe it is possible to have some more lightweight solutions which also fits in the regulations and covers our risks.

If we would have another partition which is encrypted it feels like a long chain of booting, e.g. first from ROM, then from hardware boot partitions (with SPL and u-boot) then user area boot partition (with fitimage), root file system, and then encrypted partitions. That does also feels a bit complex.

The root file system indeed contains our IP in terms of applications and other binaries which we want to protect from reading. We already covered the writing/changing part with dm-verity so that should be good. But when moving the IP to a separate partition it also means to verify that partition. It also still means we have to store the CAAM key blob somewhere, while the root file system is read only.

I am open for suggestions on how to continue this but would also still like to know how it works with the blob from CAAM to make a good considered choice here.

Kind regards,
Randy

10

Thank you for sharing more information about your use-case it certainly helps with my understanding of your situation. I can’t comment on whether this is the best path for you or not since I’m not quite familiar with the specifics surrounding the security regulations involved with such a medical device. Perhaps this is something Toradex can discuss with you in more detail over a call if you’re interested.

Meanwhile, let me go back and try to answer those questions you had prior.

Is the “blob” comparable to a public key and is the “private key” itself in the CAAM?

The answer here is technically “neither”. For encryption with CAAM we’re utilizing what is called the “Trusted and Encrypted Keys” framework in the kernel. Just to give a summary of what this means.

As the name suggests there are two types of keys at work here “trusted” keys and “encrypted” keys. The “trusted” key comes from a trusted root of trust. In the case of CAAM, this would be the OTPMK key that is unique for every SoC.

A new separate key is created that will be the “encrypted” key. This key is then encrypted by the trusted key. Meaning it can’t be accessed without your root of trust hardware. This encrypted key is what is used to encrypt the filesystem partition.

So back to your question the key “blob” in this case would be the “encrypted” key. If you’re still curious about this framework there is a more detailed write-up about it found in the kernel documentation here: Trusted and Encrypted Keys — The Linux Kernel documentation

Would it be possible to retrieve the “blob” each time at boot from the CAAM and then load it in the keyring?

The key blob is not stored in CAAM it’s stored in an encrypted state on the filesystem. When needed the key is securely loaded into the kernel keyring using the CAAM’s OTPMK key which was used to encrypt the key in the first place.

If you closely examine our tdx-enc.sh script: https://github.com/toradex/meta-toradex-security/blob/kirkstone-6.x.y/recipes-core/tdx-enc-handler/tdx-enc-handler/tdx-enc.sh

It can be seen that the script loads the key blob in order to mount the encrypted partition.

Or does it really require always the same “blob”

I’m a little confused now what you mean by this. The encrypted key blob is what is used to encrypt/decrypt your encrypted partition. If it’s not the same blob then this partition can no longer be accessed anymore.

If you remove the key blob from it’s location, then the tdx-enc.sh script will generate a new key blob. The partition will then be formatted and encrypted with this new key blob. Which would mean any data that was there prior would be lost. Since the only way to access this partition prior would be the original key blob that was used to encrypt it in the first place.

Best Regards,
Jeremias

11

Thank you for you explanation in last post. Today I have experimented a bit with the generation of keys and blobs and using it to create an encrypted storage as described in the IMX_LINUX_USERS_GUID.pdf (IMXLUG_6.6.23_2.0.0) and tried two concepts that are described there.

Chapter 10.5.3.1 DM-Crypt with Trusted keys backed by CAAM

Using this chapter, the description in AN12714, and with the tdx-enc.sh I did not manage yet to generate keys and blobs using the keyctl tool. It properly loads all the kernel modules as specified in the script and documentation, but when invoking: “keyctl add “trusted” “mykey” “new 32” @s”. It tells me: “add_key: No such device”. It is like it could not find any trusted hardware driver, however I see that there is a /dev/caam-keygen available. I have dived a bit into the source code but cannot yet find why this is failing and requires more investigation. Maybe you have any clue here what could be wrong?

Chapter 10.5.3.2 DM-Crypt with CAAMs tagged key

Using this chapter I managed to generate keys and blobs as described in the document with the caam-keygen application. I have then successfully added this key to the key retention service using keyctl and after that managed to create a new device-mapper device and mount data, write to the device, unmount it etc.

Regarding the question of using always the same blob. I think it would work for me when installing and partitioning the device to setup the partitioning as well as generating the key/blob at that moment so that I can directly do the encryption of the A/B root filesystems. At that moment I will need to store the encrypted key and blob in some read-write partition but I will manage to think of something for that.

Regarding the “trusted” keys and the “tagged” keys described in the IMX_LINUX_USERS_GUIDE, is there are reason why the Toradex encryption solution uses the “trusted” solution?

Thank you for your time.

12

Maybe you have any clue here what could be wrong?

Strange, I just tried the same command myself and it worked on my end with no error. How are you building your image in Yocto here? For reference here’s what I have in my test image:

INHERIT += "tdx-signed"
TDX_IMX_HAB_CST_DIR = "/workdir/torizon/layers/cst"
TDX_IMX_HAB_CST_KEY_SIZE = "4096"
INHERIT += "tdx-encrypted"
TDX_ENC_STORAGE_LOCATION = "/dev/sdb1"

I am building on top of our Torizon OS image, though I don’t think that matters here.

Regarding the “trusted” keys and the “tagged” keys described in the IMX_LINUX_USERS_GUIDE, is there are reason why the Toradex encryption solution uses the “trusted” solution?

I believe our goal here was to use the common “Trusted and Encrypted Keys” framework that I linked you earlier. It served our needs since it supports multiple trusted backends including CAAM. This way we can use a similar script/process for multiple trusted backends, in case a user would rather use a TPM instead of CAAM for example.

Would you rather prefer something along the lines of this “tagged” keys?

Best Regards,
Jeremias

13

Hi @rb92nl !

Were you able to check last @jeremias.tx’s message?

Do you still need any help?

It would be great if you could share any feedback :slight_smile:

Best regards,

14

Hi @henrique.tx ,

We have made some progress on this but it is not yet finished. Currently I am enjoying my holiday but I will get back to you with some information after next week.

Kind regards,
Randy