OTA update of M7 firmware

[vix]

@drew.tx

I see the error but there does not appear to be any info as to why it failed to load. Does the file exist? Is it correct? Do you have filesystem corruption?

Here is the outpiut from the command line

Verdin iMX8MP # ext4ls mmc 2:1 /ostree/deploy/torizon/var/
<DIR>       4096 .
<DIR>       4096 ..
<DIR>       4096 rootdirs
<DIR>       4096 tmp
               0 .ostree-selabeled
<DIR>       4096 lib
<SYM>          6 run
<SYM>         17 log
<SYM>         11 lock
<DIR>       4096 sota
               0 firmware_imxcm7.bin
<DIR>       4096 volatile
<DIR>       4096 roothome
<DIR>       4096 usrlocal
<DIR>       4096 cache
<DIR>       4096 spool
             190 .updated
         3145728 firmware_imxcm7.bin.rollback

So, the firmware_imxcm7.bin.rollback is there (with right size), but firmware_imxcm7.bin has 0 byte size.
I suspect something went wrong in the command cp in the script

     install)
        # Perform normal processing for this action.
        [ -f /var/firmware_imxcm7.bin ] && mv /var/firmware_imxcm7.bin /var/firmware_imxcm7.bin.rollback
        cp "${SECONDARY_FIRMWARE_PATH}" /var/firmware_imxcm7.bin
        echo '{"status": "ok", "message": "New firmware installed"}'
        exit 0 
        ;;

Probably another command to verify that the after the cp the file is there with the expected size is needed.
Because to load a .bin file, uboot must be set with the .bin size (3145728 in my case).
And if the size of the file is different from this value, I don’t know what is loaded.
Maybe this is the reason for the kernel oops.

@jeremias.tx

Could you share what commands you have running in U-Boot to boot/start the M-core on your device?

u-boot-initial-env-sd is modified with the following additions:

bootcmd=run m4boot; run bootcmd_mmc2
...
loadaddr=0x48200000
...
load_cmd=ext4load mmc 2:1
m4image=/ostree/deploy/torizon/var/firmware_imxcm7.bin
m4image_size=3145728
m4boot=${load_cmd} ${loadaddr} ${m4image}; cp.b ${loadaddr} 0x80000000 ${m4image_size}; dcache flush; bootaux 0x80000000

@jeremias.tx

Also the fix for the 1.1B was to set reinitalize a region of memory in U-Boot like so mw.w 0x550ff000 0 64. We already added this command to our documentation, so if you copied the commands there you should have this already.

I customized uboot some months ago, but now I see that the doc here has changed.
And now mw.w 0x550ff000 0 64; has been added to cm_boot before bootaux

I think that I must do the same to my uboot customization.
@jeremias.tx can you confirm this?

@jeremias.tx

If you trying to load your m7-binary without the update mechanism does the system boot fine then? If yes this could indicate something went wrong during the update procedure.

I’m sure that something went wrong during the update.
It worked before the attempt to update. And I’ve been able to successfully update three or four times, before the failing attempt.

[jeremias.tx]

So, the firmware_imxcm7.bin.rollback is there (with right size), but firmware_imxcm7.bin has 0 byte size.

Okay well this is the root cause then to why the system is failing. Keep in mind since this type of update you created requires a reboot to be fully complete you need to be aware of the atomicity of your commands in the action handler.

For one you’re using the cp command to write the final firmware_imxcm7.bin file. The cp command is not guaranteed to be atomic in Linux. The mv command is, or you could add an explicit sync command before the reboot to ensure all file writes are completed.

Whatever happened here it looks like the file write was not completed before your system rebooted causing an empty 0 byte file instead of the full proper binary. This obviously then causes issues during the boot since the M-core was expecting to load a proper binary file.

Best Regards,
Jeremias

[vix]

@jeremias.tx

For one you’re using the cp command to write the final firmware_imxcm7.bin file. The cp command is not guaranteed to be atomic in Linux. The mv command is, or you could add an explicit sync command before the reboot to ensure all file writes are completed.

Thank you.
I added sync command to my action handler.

Even if the update needs a reboot (and in a previous post you suggested to use need-completion method + complete-install method) I haven’t done it yet, because I don’t understand if this breaks the synchronous updates or it doesn’t.
Can you clarify this, please? Or point to a KB that explains this?

[jeremias.tx]

I haven’t done it yet, because I don’t understand if this breaks the synchronous updates or it doesn’t.
Can you clarify this, please? Or point to a KB that explains this?

Synchronous updates in the context of our update system are only defined between an OS update and an application/docker-compose update. Any other combination of updates is not a synchronous update by our definition.

All a synchronous update does is guarantee that the failure/success states of the OS and docker-compose updates are the same. For example in a synchronous update if the OS update succeeds but the docker-compose update fails, the mechanism will force a roll-back on the OS update to ensure both components have the same end state (i.e. failure).

If you perform any other combination of updates then the updates will still happen at the same time, but their failure/success states are independent from one another. For example, if you do a docker-compose and then a subsystem update, each one can either fail or succeed and they will not care if the other component failed or succeeded.

A more lengthy explanation of synchronous updates can be found here: Software Packages Management | Toradex Developer Center

Best Regards,
Jeremias

[drew.tx]

Hi @vix

Our BSP team has investigated the issue of the kernel rebooting when the M7 firmware is incorrect. It seems that caused by something in NXP code that we do not have access to. Our recommendation here is to store the firmware binary as a file in our OS, and then do an OSTree update when you need to update it. That will give you reliable rollback in case of issues with the M7. Since the rollback logic for OSTree is handled in the bootloader, having the kernel crash due to the incorrect firmware will be protected against by the OSTree rollback processing.

Drew

[vix]

@drew.tx

ur recommendation here is to store the firmware binary as a file in our OS, and then do an OSTree update when you need to update it. That will give you reliable rollback in case of issues with the M7. Since the rollback logic for OSTree is handled in the bootloader, having the kernel crash due to the incorrect firmware will be protected against by the OSTree rollback processing.

Thanks @drew.tx
Not ideal , because every update of M7 firmware would require an OSTree update (and not a subsystem).
But it’s a compromise that works towards reliability.