HSM build signing process and single point of failure

Technical Support
, , ,
1

We are in the process of standing up CI with secure boot signing and best practice dictates using an HSM to secure/store the signing keys over filesystem-based since the physical tokens would be in an access-controlled room.

I’ve been working through the guide provided by Toradex (Secure Boot Image Signing for NXP iMX With HSM-Backed Keys | Toradex Developer Center)

but there are several major caveats to following this process given that all keys are generated with never extractable and so cannot be transferred to additional HSMs.

This has two implications:

  1. The Yubikey is a single point of failure - if it becomes damaged or quits working, you can no longer produce a build for your fused device. This risk is never mentioned in the article and really should be highlighted - the other guides make significantly more mention of the risk of lost keys or key compromise.
  2. You cannot provision more than one CI worker to handle your builds if you have a fleet of them, which quickly leads to backlogs given yocto builds can take an appreciable amount of time to complete under some circumstances.

I have a few questions after walking through the guide workflow once as a test:

  1. During the HAB generation process I chose ECC instead of RSA. It appeared to complete okay (I see matching ECC objects in the yubikey on inspection) but I saw errors in the output that I’m not entirely certain I should ignore:40A76644A07F0000:error:0300007F:digital envelope routines:evp_pkey_get0_RSA_int:expecting an rsa key:crypto/evp/p_legacy.c:43: Does this mean only RSA key types are supported via the HSM pathway? the documentation makes no mention of such a restriction.

  2. What are the recommended options for dealing with the issue of making multiple (including a backup) token?

  • Is there a way to make the keys generated into the first token transferable but apply the never extractable flag on duplication so that the HSMs in the build agents cannot have the keys extracted? (i.e. a “primary” token which is never used in any process except to create subordinate restricted tokens for actual CI usage)
  • Or should I follow the “filesystem” key generation process and then load those on-disk keys to each HSM manually?
  • Something other process I’m overlooking?
  1. It looks like the script patches still implement some hard-coded default management keys:
    --login --login-type so --so-pin 010203040506070801020304050607080102030405060708 \
    Clearly this works if the HSM’s credentials are still the default but seems like it would cause a fault if you have already changed them before running the script. Is this an oversight or is it intentional?

Regards,
~BW

2

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.

Your concerns about single points of failure and key backup strategies are valid and important for production environments. Let me address your questions based on the Toradex documentation.

ECC Key Support

The documentation specifies RSA key generation throughout the HSM signing workflow. For HAB-based SoMs, the commands explicitly use RSA 2048-bit keys(1):

$ pkcs11-tool --module $PKCS11_MODULE_PATH --login --login-type so \
 --so-pin $PIV_MGMT_KEY_HEX --keypairgen --key-type rsa:2048 --id $BOOT_KEY_ID

(1)

The CST PKI tree generation examples also specify RSA parameters (-use-ecc n -kl 2048)(1). The error message you encountered suggests the HSM signing pathway expects RSA keys. While the CST scripts support ECC (-use-ecc parameter), the documentation does not demonstrate ECC key generation for the HSM workflow specifically.

Key Backup and Multi-Token Strategy

The documentation does not address backup strategies or multi-token provisioning for production CI environments. However, it does acknowledge the PIN security limitation(1):

The guide warns that “Passing the PIN through the environment reduces its exposure compared to storing it in local.conf, but it does not eliminate all risks. For production environments, consider injecting the PIN through a dedicated secret management mechanism such as a CI secret store or a vault service, with appropriate access control and auditing.”(1)

Available Approaches

Based on the documented workflows, you have two primary options:

Option 1: Filesystem-based key generation followed by HSM import
Generate keys using the standard CST PKI tree scripts (filesystem-based), then manually import them to multiple HSMs. This would allow you to provision backup tokens and multiple CI workers.

Option 2: SoftHSM2 for CI environments
The documentation presents SoftHSM2 as an alternative specifically suited for “development, testing, and CI environments”(1). SoftHSM2 tokens are file-based and can be backed up or replicated across multiple CI workers by copying the token directory specified in SOFTHSM2_TOKENS_DIR(1).

The documentation does not describe a process for creating a “primary” token with transferable keys that become non-extractable on duplication, nor does it address YubiKey backup procedures for production signing scenarios.

For your production CI setup with multiple workers, you may want to reach out to Toradex support directly to discuss enterprise key management strategies that aren’t covered in the current documentation.

**

Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

3

Greetings @bw908,

Thank you for bringing these concerns to our attention. I’ll forward this to our security team to look at, since it’s a bit out of my own expertise.

Best Regards,
Jeremias

4

Hi BW,

Thanks for reporting these issues. Your concerns are valid, and the documentation should make these limitations clearer. Let me address them inline.

The Yubikey is a single point of failure.

Yes. If the required signing keys exist only on one YubiKey, losing that token can prevent you from signing new images accepted by devices already provisioned with those keys. Existing signed images remain usable, but future updates require a surviving trusted signing path.

The article demonstrates how to use the layer’s PKCS#11 integration. Production deployments also need a key-management strategy covering backup, recovery, access control, and availability. I will update the article to explain this scope and explicitly highlight the risk of generating the only copy of your signing keys on a single token.

You cannot provision more than one CI worker to handle your builds.

That is a limitation of the illustrated setup with a locally attached YubiKey. The layer’s integration is intended to work with other PKCS#11 providers, although each provider needs to be integrated and validated with the signing tools. See the HSM integration documentation.

For multiple workers, possible approaches include provisioning multiple tokens with the same signing keys or introducing a dedicated signing service. With a shared signing service, compilation can run across workers while signing requests are handled centrally. This requires additional workflow integration; it is not a ready-made feature provided by the article. A central service also needs its own backup and recovery strategy.

Does this mean only RSA key types are supported via the HSM pathway?

I would not say that error alone does not establish an RSA-only restriction. The CST HSM provisioning script includes an ECC path, and YubiKey’s PKCS#11 implementation supports ECDSA. However, I have not thoroughly validated the complete ECC flow with this setup, so I cannot yet confirm that it works end to end.

I took some notes here to look into it, but if you feel this is an issue with the layer implementation, would you please open an issue in meta-toradex-security. Please make sure to report your complete setup, tool versions (CST, OpenSSL, libp11, and yubico-piv-tool), and the surrounding output, with credentials removed.

Is there a way to make the keys generated into the first token transferable, but prevent extraction from the CI tokens?

I think YubiKey PIV does not support exporting private keys, including keys generated on the device. Yubico documents external generation followed by import as an option when backups are needed.

Other HSMs support protected backup or transfer through encrypted key wrapping. For example, YubiHSM 2 is a different product from YubiKey and supports encrypted backup and restore. Whether you can restrict subsequent export on the destination depends on the HSM’s capabilities and policies; it is not something we can assume for every PKCS#11 device.

Or should I follow the filesystem key-generation process and load those keys into each HSM manually?

Yes, that is a valid approach if your security requirements permit key generation outside the HSM. You can generate the HAB PKI using the standard hab4_pki_tree.sh on a controlled offline machine, then import the required signing keys and matching certificates into each token. Preserve an encrypted offline backup under appropriate access controls, and securely remove temporary working copies after verifying the imports and recovery procedure.

The script patches still implement hard-coded default management keys. Is this an oversight?

It is, indeed. The HAB patch still uses the factory management key in two certificate-import commands, even though other commands use PIV_MGMT_KEY_HEX. I will fix this so the configured management key is used consistently.

Best regards,

Sergio Prado

5

Thanks @sergio.tx.

but if you feel this is an issue with the layer implementation,

I don’t think so, these errors manifest when using the NXP HSM script so if anything it’s a bug or lack of documentation on their part, rather than an issue with meta-toradex-security itself.

I’ll experiment more today, it does seem like the pkcs11-tool command has a flag indicating whether an item is exportable so it could be possible that a small modification to the generation script is all that’s needed here - but I can’t tell for sure without just trying it since the usual answers of “no” don’t seem to take this into account.

FWIW I selected ECC rather than the default RSA because it’s supposed to perform better with smaller key sizes which seems like a logical choice for an embedded platform where computing resources are more constrained than a conventional processor.

~BW

6

Hi @bw908 ,

Sounds good. If you need any further assistance, please let me know.

And if you run into any issue with the layer itself, please report it directly in the repository so I can make sure it gets prioritized.

Sergio Prado

7

A further question (likely for @jeremias.tx) on this - I just realized the tcbuild.yml for TorizonCore builder does not offer any notes or parameters on configuration for an HSM. Does that mean TCB does not yet support signing via HSM yet? Is this on the roadmap to compliment the yocto build support?

@sergio.tx Per your request, I filed HSM signing doesn't seem to be working - silently produces unsigned build. · Issue #197 · toradex/meta-toradex-security · GitHub since I was unable to get the example configuration to work in my builds - it just silently fails and produces an unsigned output.

~BW

8

Does that mean TCB does not yet support signing via HSM yet?

Correct, at the time of writing TCB can only be used to do basic re-signing of an image via keys provided on the filesystem. There is no support yet for keys/secrets backed by an HSM.

Is this on the roadmap to compliment the yocto build support?

It’s on the roadmap in the sense that this is a known limitation. Though it has not been planned yet to the point I can even give you a rough timeline of when it will be done other than “eventually”. Which I realize is not the most helpful answer, but it’s the most accurate “timeline” i can share right now.

Per your request, I filed

On behalf of the team thank you for creating that issue, they will get to it when able.

Best Regards,
Jeremias

9

Thanks for the update/info.

It turns out the issue was a configuration error on my part but it did expose a rough edge in the configuration handling that could be improved with warning messaging.

11

Is this a region where you’d be willing to consider an external PR either for inclusion or as a starting point?

I realized just now that the FIT key isn’t even password-protected on disk which is quite a departure from current best practices - The TCB modification doesn’t really seem too invasive since the underlying tools already support it - it’s just a matter of argument plumbing and the required libraries (e.g. libykcs11.so)

If yes I can attempt to structure my work accordingly and offer it up, otherwise I’ll get something temporary in place that works for now and await the official implementation when it arrives.

~BW908

12

We’re always willing to consider external contributions. The only caveat here is that other than the usual technical review of the contribution we may also have an internal discussion about other points. Mostly product related discussions, UX, and stuff that aren’t strictly related to the code itself.

That said, please if you are able do make a contribution then do so. Just be aware we might need to discuss some points internally. Especially considering we haven’t even planned this feature yet, so we have no vision on how we want to go about it in the first place.

Best Regards,
Jeremias

13

Yes, of course - that makes sense and I don’t expect it to be accepted instantly or without feedback/rework - Mostly I just wanted to know whether it was a non-starter; it stands to reason you have your own internal processes and guidelines that need to be met when implementing any new feature, and this can be a sensitive area.

14

With that said, please just contribute what you can. Then whatever discussions need to happen on our side, we will take care of as needed. If anything this will push us to have the discussion about HSM usage on TCB sooner rather than later.

Best Regards,
Jeremias