Secure Boot by update: burning fuses failes

Technical Support
, , ,
1

Hello,

I want to update devices in the field to secure boot with ECoT ( https://developer.toradex.com/torizon/security/secure-boot-in-field-upgrades/#upgrading-from-non-secure-boot-to-ecot ).

I got stuck at the last step (perform a fuse update). I successfully create a lockbox in the Torizon Cloud containing the fuse update (with close = false). The installation is without errors and the device reboots.

On the reboot, I got the following output of uboot:

Authenticate image from DDR location 0x44000000...
NOTICE:  Do not release JR0 to NS as it can be used by HAB
NOTICE:  BL31: v2.10.0  (release):lf-6.6.52-2.2.1-dirty
NOTICE:  BL31: Built : 00:00:00, Jan  1 1970


U-Boot 2024.07-7.6.0-devel+git.3f772959501c (Jan 01 1970 - 00:00:00 +0000)

CPU:   Freescale i.MX8MMQ rev1.0 1600 MHz (running at 1200 MHz)
CPU:   Industrial temperature grade (-40C to 105C) at 45C
Reset cause: POR
DRAM:  2 GiB
Core:  146 devices, 27 uclasses, devicetree: separate
WDT:   Started watchdog@30280000 with servicing every 1000ms (60s timeout)
MMC:   FSL_SDHC: 0, FSL_SDHC: 1, FSL_SDHC: 2
Loading Environment from MMC... Reading from MMC(0)... OK
get_tdx_eeprom: cannot find EEPROM by node
MISSING TORADEX CARRIER CONFIG BLOCKS
get_tdx_eeprom: cannot find EEPROM by node
In:    serial@30860000
Out:   serial@30860000
Err:   serial@30860000
Model: Toradex 0055 Verdin iMX8M Mini Quad 2GB WB IT V1.1F
Serial#: 15506165
SEC0:  RNG instantiated
Net:   eth0: ethernet@30be0000 [PRIME]
## U-Boot CLI access is enabled
Hit any key to stop autoboot:  0
MMC: no card present
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
973 bytes read in 2 ms (474.6 KiB/s)
## Executing script at 50280000
13583 bytes read in 3 ms (4.3 MiB/s)
## WARNING: Command execution WOULD BE DENIED in closed state (blocked by category) for `fuse prog -y 6...`.
Programming bank 6 word 0x00000000 to 0x<word0>...

Word 0x00000000:
Value 0x<word0>:0x00000000
failed
## NOTE: Bootloader seems to support secure boot.
Saving Environment to MMC... Writing to MMC(0)... OK
54 bytes read in 3 ms (17.6 KiB/s)
Applying Overlay: verdin-imx8mm_displaylc_mi1010aqt-1.dtbo
25358377 bytes read in 152 ms (159.1 MiB/s)
## Loading kernel from FIT Image at 50300000 ...
   Using 'conf-freescale_imx8mm-verdin-wifi-LP-11828-04.dtb' configuration
   Verifying Hash Integrity ... sha256,rsa2048:dev+ OK
   Trying 'kernel-1' kernel subimage
     Description:  Linux kernel
     Type:         Kernel Image
     Compression:  gzip compressed
     Data Start:   0x503000ec
     Data Size:    11384497 Bytes = 10.9 MiB
     Architecture: AArch64
     OS:           Linux
     Load Address: 0x48200000
     Entry Point:  0x48200000
     Hash algo:    sha256
     Hash value:   c9dcfe904daec8b7fe90f210428111fb491cfd0414f6afd1d98fdbe584d2bd9e
   Verifying Hash Integrity ... sha256+ OK
## Loading ramdisk from FIT Image at 50300000 ...
   Using 'conf-freescale_imx8mm-verdin-wifi-LP-11828-04.dtb' configuration
   Verifying Hash Integrity ... sha256,rsa2048:dev+ OK
   Trying 'ramdisk-1' ramdisk subimage
     Description:  initramfs-ostree-torizon-image
     Type:         RAMDisk Image
     Compression:  uncompressed
     Data Start:   0x50df60ac
     Data Size:    13841916 Bytes = 13.2 MiB
     Architecture: AArch64
     OS:           Linux
     Load Address: 0x52300000
     Entry Point:  unavailable
     Hash algo:    sha256
     Hash value:   7b0a5e670c1dcd899a7ef74d6ef3945c4b559ebfd9789a640f84f1b92a82e44a
   Verifying Hash Integrity ... sha256+ OK
   Loading ramdisk from 0x50df60ac to 0x52300000
## Loading fdt from FIT Image at 50300000 ...
   Using 'conf-freescale_imx8mm-verdin-wifi-LP-11828-04.dtb' configuration
   Verifying Hash Integrity ... sha256,rsa2048:dev+ OK
   Trying 'fdt-freescale_imx8mm-verdin-wifi-LP-11828-04.dtb' fdt subimage
     Description:  Flattened Device Tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x50ddb8d0
     Data Size:    67468 Bytes = 65.9 KiB
     Architecture: AArch64
     Load Address: 0x50200000
     Hash algo:    sha256
     Hash value:   096215305ac6e401c2fc390148e6d362c3b726e587d52122fca34153913cb669
   Verifying Hash Integrity ... sha256+ OK
   Loading fdt from 0x50ddb8d0 to 0x50200000
## Loading fdt from FIT Image at 50300000 ...
   Using 'conf-verdin-imx8mm-secboot-kargs_overlay.dtbo' configuration
   Verifying Hash Integrity ... sha256,rsa2048:dev+ OK
   Trying 'fdt-verdin-imx8mm-secboot-kargs_overlay.dtbo' fdt subimage
     Description:  Flattened Device Tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x50df34a0
     Data Size:    558 Bytes = 558 Bytes
     Architecture: AArch64
     Load Address: 0x50240000
     Hash algo:    sha256
     Hash value:   b256969e497d91b778f10065cdc1afa49597f6cb6ff45f9739712bc4b775c8f9
   Verifying Hash Integrity ... sha256+ OK
## Loading fdt from FIT Image at 50300000 ...
   Using 'conf-verdin-imx8mm_displaylc_mi1010aqt-1.dtbo' configuration
   Verifying Hash Integrity ... sha256,rsa2048:dev+ OK
   Trying 'fdt-verdin-imx8mm_displaylc_mi1010aqt-1.dtbo' fdt subimage
     Description:  Flattened Device Tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x50decbd4
     Data Size:    2653 Bytes = 2.6 KiB
     Architecture: AArch64
     Load Address: 0x50240000
     Hash algo:    sha256
     Hash value:   f3aeb973ea106e50597a8fcc5d77363b4d27b10699d23a2e9107327a19a4bd51
   Verifying Hash Integrity ... sha256+ OK
   Booting using the fdt blob at 0x50200000
Working FDT set to 50200000
   Uncompressing Kernel Image to 48200000
   Loading Device Tree to 00000000bdeef000, end 00000000bdf02c81 ... OK
Working FDT set to bdeef000
## Validation of bootargs succeeded.

Starting kernel ...

[    1.048221] nvmem imx-ocotp0: cell mac-address raw len 6 unaligned to nvmem word size 4
[    1.108783] rtc-ds1307 0-0032: hctosys: unable to read the hardware clock
Starting systemd-udevd version 255.21^
sysroot.readonly configuration value: 0 (fs writable: 1)
composefs+ostree: Validated commit signature using '/etc/o[    2.887665] overlayfs: cannot append lower layer
stree/initramfs-root-binding.key'
composefs: Verifying digest: be87215fd328a6ca1d2f7c188b50c5c99ae58b989f80ab29d403b70dab1ac2d9
composefs: mounted successfully
[    8.487277] usbmisc_imx 32e40200.usbmisc: vbus is error
[    8.492540] usbmisc_imx 32e40200.usbmisc: Error occurs during detection: -22

Torizon OS 7.6.0-devel-20260610125315+build.0 (scarthgap) verdin-imx8mm-15506165 ttymxc0
Verdin-iMX8MM_CBT3-image

verdin-imx8mm-15506165 login:

The devices reboots successfully, but the programming of the first fuse failed. The output of fw_printenv after this first reboot is as follows:

fuse_dry_run=0
fuse_prog_close=0
fuse_prog_list=0x<word0> 0x<word1> 0x<word2> 0x<word3> 0x<word4> 0x<word5> 0x<word6> 0x<word7> 
fuse_status=failed
fuse_val_1=0x0
fuse_val_2=0x0
fuse_val_3=0x0
fuse_val_4=0x0
fuse_val_5=0x0
fuse_val_6=0x0
fuse_val_7=0x0
fuse_val_8=0x0
fuse_val_close=0

When I reboot again, uboot donā€™t repeat the fuse burning. But the output of fw_printenv is now this:

fuse_dry_run=0
fuse_prog_close=0
fuse_prog_list=0x<word0> 0x<word1> 0x<word2> 0x<word3> 0x<word4> 0x<word5> 0x<word6> 0x<word7>
fuse_status=failed
fuse_val_1=0x<word0>
fuse_val_2=0x0
fuse_val_3=0x0
fuse_val_4=0x0
fuse_val_5=0x0
fuse_val_6=0x0
fuse_val_7=0x0
fuse_val_8=0x0
fuse_val_close=0

So the programming of the first fuse worked in the first place but the comparing failed.

When I repeat the fuse update I get the following on the next boot:

[...]
Found U-Boot script /boot.scr
973 bytes read in 2 ms (474.6 KiB/s)
## Executing script at 50280000
13583 bytes read in 3 ms (4.3 MiB/s)
## WARNING: Command execution WOULD BE DENIED in closed state (blocked by category) for `fuse prog -y 6...`.
Programming bank 6 word 0x00000000 to 0x<word0>...
Comparing bank 6:

Word 0x00000000:
Value 0x<word0>:0x<word0>
passed
## WARNING: Command execution WOULD BE DENIED in closed state (blocked by category) for `fuse prog -y 6...`.
Programming bank 6 word 0x00000001 to 0x<word1>...
Comparing bank 6:

Word 0x00000001:
Value 0x<word1>:0x00000000
failed
[...]

And after a another reboot, fw_printenvshows the following:

fuse_dry_run=0
fuse_prog_close=0
fuse_prog_list=0x<word0> 0x<word1> 0x<word2> 0x<word3> 0x<word4> 0x<word5> 0x<word6> 0x<word7>
fuse_status=failed
fuse_val_1=0x<word0>
fuse_val_2=0x<word1>
fuse_val_3=0x0
fuse_val_4=0x0
fuse_val_5=0x0
fuse_val_6=0x0
fuse_val_7=0x0
fuse_val_8=0x0
fuse_val_close=0

My /usr/lib/ostree-boot/uEnv.txt is appended.

uEnv.txt (12.0 KB)

Claude Sonnet 4.6 suggest to add a short delay between the programming of the fuse (line 132) and the comparing (line 141):

env set temp fuse prog -y \$fuse_bank_word_$index ${fuse_val};
run temp;
...
mdelay 50;               ā† here
env set temp fuse cmp \$fuse_bank_word_$index ${fuse_val};

What do you think about this?

Best regards,

Markus

2

Hi, Iā€™m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.

To troubleshoot this fuse programming issue properly, could you let me know which hardware platform and Torizon OS version youā€™re running? You can use the tdx-info command to get this information: tdx-info

However, I can provide some initial guidance based on your description. The issue youā€™re experiencing with fuse programming failures during secure boot updates is a timing-related problem that can occur during the verification process (1).

Troubleshooting Steps

1. Check Fuse Status

You can check the result of the secure boot fusing operation using the fuse_status U-boot environment variable (1):

sudo fw_printenv fuse_status

A fuse_status value of ā€œsuccessā€ means everything went correctly, while ā€œfailedā€ indicates something went wrong (1).

2. Enable Additional Logging

For in-field programming, you can configure additional logs during the update process by modifying the systemd service that launches the update client (aktualizr.service) (1):

[Service]
...
Environment="FUSE_LOG_ENABLED=1"
...

This will create an additional log file at /var/lib/rollback-manager/fuse-update.log with more detailed information about the fuse update process (1).

3. Verify Fuse Values

You can manually check the actual fuse values using the fuse read command in U-Boot to confirm what values were actually programmed versus what was expected (1).

Important Safeguards

The system includes several safeguards during fuse programming (1):

  • If there was an error detected while trying to set the fuses, the logic will avoid closing the device to prevent bricking
  • If the hab/ahab_status commands return an event (indicating the device shouldnā€™t be closed), the fusing logic will also not close the device

Power Considerations

Make sure the device remains powered during the entirety of the fusing operation until it has fully booted (1). Power interruption during this process could cause issues.

Regarding your suggestion about adding a delay - while timing issues can occur, Iā€™d recommend first gathering more information about your specific hardware platform and OS version to provide the most accurate guidance for your setup.

Best regards, Toradex AI Assistant.

Did you like my answer? :+1: :-1:

If I answered your question, please tick the ā€œSolutionā€ checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

3

Greetings @Mowlwurf,

This is a known issue as seen here: Verdin IMX8MP Torizon OS 7.6.0 Secure Boot: Programming fuses fails

The core issue is due to the unintuitive way the i.MX chip-set interacts with the fuse subsystem in U-Boot. This causes compared values to not be updated till after a reboot of the system. Our team already has a ticket for this issue and will address it when possible.

Best Regards,
Jeremias

5

Hi @jeremias.tx,

thank you very much for your prompt reply. Can you tell me when a solution can be expected?

I think Iā€™ll remove the comparison step from the script as a workaround until then, so I can test the update procedure thoroughly.

Best regards,
Markus

6

Can you tell me when a solution can be expected?

Currently I canā€™t provide any timelines. Our team is going to be doing our planning and prioritization for the upcoming quarter. So many aspects regarding timeline may change drastically in the coming weeks.

But, Iā€™ll try to notify you when our team gets around to this specific issue.

Best Regards,
Jeremias

1 Like
7

Hello @Mowlwurf,

This issue should be solved by the following commit: u-boot-distro-boot: Reorder fuse operations Ā· torizon/meta-toradex-torizon@ea8768c Ā· GitHub

This commit should be present from Torizon 7.7.0, but keep in mind that updating the bootloader that is on the device would be needed to take advantage of this change.

Best Regards,
Bruno