I am building a yocto image using the kirkstone branch of toradex manifests.
My image is based on toradex-minimal-image.
The image is for an iMX8MP verdin SoM.
Due to security requirements we have implemented secure boot (currently not including signing of rootfs).
We have also been requested to blow the fuse to disable SDP and JTAG. Also due to the device being closed the UART console is also unavailable.
Our initial image is flashed using TEZI and the tezi image built by the yocto recipe.
Is there any other process we can use to flash updated images to the SoM eMMC to enable major updates? Or is this now a case of flash a new SoM only and bin the existing one?
Note, at present we cannot use TorizonOS (a decision I hope to change after phase 2 of this project is complete).
Hi, I’m the Toradex AI Assistant. The Toradex Support team will follow up with you shortly!
Did you like my answer?
Hello,
Thank you for reaching out with your question. Updating the eMMC on your iMX8MP Verdin SoM with secure boot enabled and SDP disabled does present some challenges, as you’ve noted. Here are some potential methods you might consider for performing major updates:
Bootloader with Update Capabilities: If your bootloader supports it, you could implement an update mechanism that allows flashing new images from an external storage medium, such as a USB drive or SD card. This would require the bootloader to securely authenticate and apply updates.
Network-Based Updates: If your device is network-enabled, you could set up a secure update mechanism over the network. This involves downloading the update package to the device and applying it locally, ensuring the process is secure and authenticated.
External Programming Tools: Using external hardware tools to directly program the eMMC is another option. This typically involves using a JTAG or similar interface, but it requires specialized equipment and expertise.
Replace the SoM: If the above methods are not feasible, replacing the SoM might be necessary for major updates. While more drastic, this ensures you can deploy the latest software securely.
Each of these methods has its own set of challenges and may require significant development effort to implement securely. It’s important to ensure that any update mechanism maintains the integrity and authenticity of the new software to comply with secure boot requirements.
I hope this helps you in planning your update strategy.
Best regards,
Toradex AI Assistant.
Please, also check if these links can help you solve your problem:
For option 1, we have a signed imx-boot for the bootloader (if I’m not mistaken). So not sure if that means we could do something there?
Option 3 I don’t believe is possible due to no SDP, UART or JTAG availability.
Option 4 I hope is the “if all else fails” option (and the current plan unless I can find a better solution).
Option 2 may be a possibility depending on what that requires. Is this just hosting all the image packages and doing apt update? If so does this actually include all the u-boot and fs changes needed to replace everything from a tezi image? My understanding is it’s just the kernel and rootfs changes (would that not break the Hab boot cert verification?).
Just to make sure I understand. You’re basically asking for a way to update the device after it has been closed via secure boot mechanisms, correct? Are their specific software components you want to update, or just everything? Is this something you plan to do often or for what reasons would it be needed to update your closed systems?
Well given that the device is closed and you are also planning to disable SDP this does limit your options quite a bit. In my mind the most straightforward would be to have a software update mechanism already on your device prior to closing it. Similar to what we have on Torizon OS.
Could in theory even update your signed bootloader to a different bootloader (also signed with the same keys). Though of course this can be a risky action since one mistake and that device won’t be booting anymore.
That is correct. Sadly the original team decided not to use Torizon as we could not remove enough stuff from it.
Currently we are building the kirkstone branch, but the plan is to move to scarthgap as soon as we can.
The problem being with a closed device and no SDP or JTAG or UART, how could we deploy any updated mix-boot, u-boot and the rootfs?
Sadly I don’t have enough time to implement much to what is on the image already (should have been delivered 2 weeks ago).
I was looking at UUU but couldn’t get anything I tried to work and parked it for later. Now I’m getting to the point where this will be the #1 priority and hoping I can get some ideas before starting to attack the issue again.
Ideally, I will be suggesting we switch to Torizon for the first major upgrade. But will need a way to deploy the tezi image manually to the device.
Perhaps no big regrets about disabling JTAG, but disabling SDP is questionable. SDP on closed device is as secure as secure boot is. Even if someone tries to reuse image of your signed U-Boot fir eMMC, chances are it won’t be usable to boot over SDP. At least it is the case for non-SPL/TPL U-Boot for iMX7. U-Boot for SDP has to be signed differently than U-Boot for native boot. It is so because of DCD, which booting over SDP has to be signed to addresses in SRAM. Not knowing too much about IMX8MP, you may try recovering over SDP with your signed U-Boot, just replace it in recovery files. I doubt you will be able to boot with it over SDP. If indeed you are able, then it is still possible to make your signed production image not SDP friendly.
So you have no access to U-Boot prompt, no SDP, no JTAG. What about Linux root login or su from non-root? You could try for example changing U-Boot bootcmd with fw_setenv and boot from USB/SD/NFS, then replace all you wish on offline FS along with new U-Boot.
I agree with the view that there is little benefit in disabling SDP via the fuse if the device is closed, because you need a signed boot image anyway.
For IMX8MP it is likely to be possible for the production boot image to be the same as the boot image for SDP (DCD isn’t used for imx8m devices). As Edward says, the production image can have SDP disabled.
I think that Toradex boot image builds normally have SDP enabled - from what I’ve seen.
To use the recovery method over SDP (assuming the fuse isn’t already blown), then you would also need to sign the itb since, by default, U-Boot requires the FIT image to be signed.
With root or su access you can change the boot image and the kernel image and manually modify the rootfs.
You can make quite complex scripts (eg. using plink in a Windows batch file) to do this, but only suitable for prototypes as it would probably expose your password and definitely a last resort.
Minimal Yocto images do not normally have a package manager such as apt unless you have specifically included one.
As Edward says, booting the OS from some other source would allow you to change everything.
If you have fw_setenv, this may be your easiest option in the short term. If you don’t then you can probably add it manually - it doesn’t have many dependencies.
Again, you would probably need root or su access to run fw_setenv
Ultimately though, you need to have a robust update mechanism. Either use an existing solution or develop your own.
This will be a hard requirement in Europe for many classes of product when the CRA comes into force.
As others have said, disabling SDP provides questionably little value. But, if you are already settled on this, then UUU won’t be of much use to you. It really does sound like you need an integrated update solution in your system if you want to handle this properly and of course securely. I realize your timeline doesn’t quite allow this for the current iteration, but there’s not really many options with how you’ve chosen to design your system.
Thanks for all the info.
Sadly it’s as bad as I feared. Lol.
I will certainly be sending these details “up the chain of command”.
I think my plan will have to be to deploy what we have and try to incorporate the cost of new SoMs into our support contracts so we can actually get it right.
It is interesting to hear about SDP in relation to security and I will dig deeper into the report we had done about this as it was the main driver for going down this dead end road.
As part of my horizon plans I will suggest a fair few corrective actions.
Regards
I’ll echo what others have said here, and point out that the combination of choices made have really left you in a difficult spot with regard to security incident mitigation. The fact that the rootfs is left without any integrity protection, in particular, seems like a very unfortunate situation: if there were to be an exploitable flaw (either in the base OS or in your application software) that allowed an attacker access and privilege escalation, they would be able to get in, patch the hole behind them, and take up residence permanently, essentially using all your defenses against you. If I can infer that there’s no intentional way for you to get root access to the device (because otherwise you would use that as your update mechanism), I can also imagine that it might be somewhere between very difficult and impossible to detect that a device had been compromised in such a way.
I don’t believe you’ve mentioned what industry you’re in or what your typical threat model is, but I am guessing there are high security needs (at least superficially), given the extent of the early boot protection measures you’ve taken. That’s precisely what the Uptane framework is designed for: it allows you to have an update system that is highly resilient to compromise even under extremely adverse conditions, and it provides a mechanism to safely rotate keys. Many (or perhaps most) of the real-world applications and deployments aren’t publicly disclosable, of course, but just as an example I’ll refer you to this presentation from last year’s escar where Cameron Mott from SWRI reported on a red-team exercise they did with the US DoD attempting to compromise an Uptane software update system, with privileges for the attackers that started with a full Dolev-Yao adversary between vehicle and server, and escalated in attacker privilege from there.
Of course we don’t lock down Torizon OS nearly as heavily as their custom seL4-based microkernel OS did in that exercise, but the possibility to address such a strong threat model is why we built our update system around Uptane. Perhaps it’s counter-intuitive, but the more locked-down and security sensitive your system is, the more you need a remote update system–unless you’re prepared to do a full-on recall for every potentially exploitable vulnerability that emerges.
@jon.tx
Indeed my project (and possibly some new ones coming in the pipeline) are not far from what you have mentioned here.
I did take a brief look at uptane (and I have included the repo manually in my build system), however have not had much of a chance to see how to implement this yet.
It is just a shame that I was not on this project when it started as there have been a fair few assumptions and mistaken interpretations of the reports we had done that I think lead them down a path that causes more problems than it solves.
I have submitted a request for a full re-design after our initial delivery. Unfortunately that will mean we need to ensure plans for equipment rotation so we can apply this via a SoM swap once we have the right things in place.
I guess we do have one “special” account with the ability to escalate to root privileges. But I don’t think we expose the hardware to connect any form of external storage (I may need to check that though, as there may be a caveat where we can use a dev board to achieve the goal).
Before, this though I am hoping to be able to cleanup some work I did on the tdx-security-layer for ensuring encrypted partitions only get encrypted at startup if the SEC_BOOT fuse is blown. Once I have managed to propose and hopefully get that merged into the TDX repo I can start on updating everything and working to figure out how to get to where we need to be. But I fear that may be a long road to plan.
