Custom bootloader via OTA

Hi,

We recently moved our yocto build system for torizon-core-docker to a cloud vm for integration into an Azure pipeline. However, I had not understood that there is a second set of keys in keys/fit above and beyond keys/cst. Though this is sorted out now, we still have a few units that refuse to boot new kernels. I was hoping we could simply update the bootloader through OTA but I can’t seem to find any information on pushing custom bootloaders? The impression I got is that it’s something you wish to prevent?
Is there a way around this? Could I bypass aktualizr and call bl_actions.sh manually to install a new bootloader?

Looking forward to your response.

Cheers,
Lloyd

Hi, I’m the Toradex AI Assistant. The Toradex Support team will follow up with you shortly!

Did you like my answer?

:+1: :-1:


Hello there!

Thank you for your question about custom bootloader updates via OTA on Torizon. While specific guidance on OTA bootloader updates or bypassing aktualizr isn’t directly available, I can provide some general information on bootloader customization options that might be helpful.

Torizon supports bootloader customization through U-Boot, which is the default bootloader used. You can configure U-Boot to suit your needs by modifying its environment variables and scripts. This can include setting up custom splash screens, adjusting boot sequences, and more. For detailed instructions on configuring U-Boot, you can refer to the U-Boot configuration guide.

If you’re looking to perform OTA updates, Torizon provides a robust framework for managing software updates, including the use of aktualizr for secure and reliable updates. While aktualizr primarily handles application and system updates, integrating bootloader updates would require careful planning to ensure system stability and security.

I hope this gives you a good starting point for exploring bootloader customization on Torizon. If you have more specific questions or need further assistance, feel free to ask!

Best regards,
Toradex AI Assistant

Greetings @lblackbeard,

Just to understand the situation first it sounds like you have a secure Torizon OS image, correct?

How is this related to the devices that are refusing to boot, and how did you come to the conclusion that the issue would be solved with a bootloader update?

I’m trying to understand the relation between these various points that you mentioned. I want to make sure I’m not suggesting something incorrect for your setup here without understanding the full story.

Best Regards,
Jeremias

Hi Jeremias,

Yes, we’re using the Tezi artefact from a torizon-core-docker (6.8.1) build that inherits tdx-signed as the input to torizoncorebuilder.

The public key of keys/fit is written into u-boot’s device tree which is concatenated into the bootloader bin, right?
Since each build system had different fit keys, we ended up with different bootloaders. Devices flashed with key1 only ota upgrade to fit images signed with key1 and devices and vice versa.
Since the devices in question are closed, we cannot simply reflash to a bootloader with key1, so I would like to swap out the bootloader some other way.

Cheers,
Lloyd

Since each build system had different fit keys, we ended up with different bootloaders. Devices flashed with key1 only ota upgrade to fit images signed with key1 and devices and vice versa.

Wait let me try to understand this point since I’m still a little confused. So you had one build with one set of keys, then when you did your build on the cloud VM for azure a second set of keys was used instead, causing this confusion.

If that is the case, would it not just make more sense to use the same set of keys? You can just take the keys from your first build system and share them with the second. This would be a lot better for maintenance than having 2 sets of keys floating around and trying to manage which device is using which set of keys. I mean for your current devices I guess this doesn’t help since you can’t re-flash them cause they’re closed, but I mean going forward.

Also, how did your devices get into this state to begin with where they have a mismatching bootloader? Were the devices originally flashed with the OS image from build system 1, and then you did an OTA update for just the OS from build system 2?

In any case to do a bootloader update with your custom bootloader you just need to upload your own bootloader binary. However, this needs to be done in a certain way. First of all you can upload your bootloader binary using TorizonCore Builider like so:

torizoncore-builder platform push --credentials credentials.zip  --hardwareid verdin-imx8mp-bootloader --custom-meta <json-blob> imx-boot

Important things to note here, the --hardwareid must be of the form “(name of module)-bootloader”. So in this example if the device is a Verdin i.MX8MP then the hardwareid is “verdin-imx8mp-bootloader”. Also the name of the bootloader binary might be different depending on the SOM, for the Verdin i.MX8MP the binary is called “imx-boot”.

Finally, the most important detail is the json information passed to the --custom-meta flag. This json information should be of the form:

"bootloader": {
            "ddOptions": "seek=0",
            "dtVersion": "2022.04-6.2.0-devel+git.0e1f11392251",
            "env": {
              "type": "embedded",
              "resetOnUpdate": true,
              "embeddedOffset": 1005975,
              "embeddedSize": 4130,
              "keepVars": null,
              "setVars": null
            }
          }

Of course pass it to the tool as a one-liner json. i just presented it as multi-line for readability. Now the values in this json will be specific to your bootloader. In your Yocto build, in the output directory where all your build output is, there should be a file called u-boot-ota.json. The file contents should look something like this:

{
  "ubootversion": "2022.04",
  "ubootrelease": "2022.04-6.4.0-devel+git.dc27426aa417",
  "envoffset": 1025146,
  "envsize": 4128
}

This contains the values that you should use for your json that you pass to --custom-meta:

  • ubootreleasedtVersion
  • envoffsetembeddedOffset
  • envsizeembeddedSize

Every other field in the json can be the same as what I showed in my example json above. It’s important that these values correspond to the bootloader binary that you will upload. If these do not match the update will probably fail or not work correctly.

Best Regards,
Jeremias

Wait let me try to understand this point since I’m still a little confused. So you had one build with one set of keys, then when you did your build on the cloud VM for azure a second set of keys was used instead, causing this confusion.

Yes, that’s what happened.

If that is the case, would it not just make more sense to use the same set of keys? You can just take the keys from your first build system and share them with the second. This would be a lot better for maintenance than having 2 sets of keys floating around and trying to manage which device is using which set of keys. I mean for your current devices I guess this doesn’t help since you can’t re-flash them cause they’re closed, but I mean going forward.

This is what we’ve done, yes. The problem is indeed restricted to the small number of closed devices that were flashed with the keys generated on the VM.

Also, how did your devices get into this state to begin with where they have a mismatching bootloader? Were the devices originally flashed with the OS image from build system 1, and then you did an OTA update for just the OS from build system 2?

That’s right.

In any case to do a bootloader update with your custom bootloader you just need to upload your own bootloader binary. However, this needs to be done in a certain way. First of all you can upload your bootloader binary using TorizonCore Builider like so.

Perfect, thank you! I will ask one of our team to have a go at this at let you know.

Best regards,
Lloyd

Do let us know how this works out.

Best Regards,
Jeremias

Hi Jeremias,

We successfully uploaded the bootloader to Torizon Cloud. There were two devices that needed the bootloader update — I’ll refer to them as Device-1 and Device-2.

Device-1:

  • We updated the bootloader using Torizon Cloud.
  • The resetOnUpdate flag was set to false.
  • Despite this, the device rebooted itself after the bootloader update, which we did not expect.
  • Since the base OS hadn’t been updated yet, the new bootloader failed to verify the existing OS and the device entered a permanent bootloop.

The plan was to first update the bootloader, then push the new base OS, and only after both were installed should the device reboot — this way, the bootloader and base OS would be in sync (key-wise). However, the device rebooted prematurely.

Device-2:

  • This device did not receive the bootloader update.
  • We manually set the rollback to 0.
  • After rebooting the device, it also ended up in a permanent bootloop.
  • We expected that, after a few failed boots, the rollback mechanism would trigger and restore the previous working state — but that did not happen.

Questions:

  1. Is it expected behavior for the device to automatically reboot after a bootloader update even when resetOnUpdate is set to false?
  2. What is the proper way to stage a bootloader and base OS update so the reboot happens only after both are in place?
  3. Regarding rollback: under what conditions does the rollback system actually trigger and revert to a previous working state? Does setting the rollback alone initiate rollback, or is more required?
  4. Is there any way to recover these devices?

Is it expected behavior for the device to automatically reboot after a bootloader update even when resetOnUpdate is set to false

No that field has nothing to do with the automatic reboot. If you notice in the json structure that field is underneath env:

...
"env": {
              "type": "embedded",
              "resetOnUpdate": true,
...

That field controls whether the U-Boot environment should be reset during the update. If you want to disable automatic reboots for all updates that require a reboot then what you want is here: Aktualizr - Modifying the Settings of Torizon Update Client | Toradex Developer Center

What is the proper way to stage a bootloader and base OS update so the reboot happens only after both are in place?

I thought you’d only be performing a bootloader update, so this complicates things a bit.

I imagine you would issue a single update containing both the Bootloader and OS update packages you want. This should cause the update client on the device to download and install the OS package and then the Bootloader package. Only then the device would reboot. In theory after the reboot the device should boot into the new bootloader and the new OS.

This is because the update client first tries to install all the packages as directed by the update. Then if any updates requires reboot to complete, it will do the reboot at the end. So I imagine this should work for what you want, but this may need testing in a use-case where everything is signed just to be sure.

Regarding rollback: under what conditions does the rollback system actually trigger and revert to a previous working state? Does setting the rollback alone initiate rollback, or is more required?

There is no rollback available for bootloader updates. This is a known limitation as documented here: Bootloader Updates in Torizon OS | Toradex Developer Center

Is there any way to recover these devices?

Well if both devices are in a permanent boot loop, I understand this to mean they can’t get into Linux which means further updates are not possible. As said prior these devices are closed so they can’t be easily re-flashed.

I’m afraid then there’s not a good way to recover these devices if that is the case. This is one of the dangers of working with closed devices as they can be easy to get locked out of.

Best Regards,
Jeremias