OTA update of M7 firmware

[vix]

The topic has been discussed several times on this community, but with Torizon OS 6 on Verdin iMX8M-Plus it’s only the bootloader that can load a M7 firmware that runs from DDR (not from internal ITCM).
Not problem in creating a TEZI image which contains the M7 firmware.

But how can I handle this with Torizon OTA feature?
It’s not an Operating System update (that is OSTree), but I think it’s a bootloader update.
Is this true?
How can I push/load this bootloader in Torizon Cloud (it’s not a customized bootloader, it’s a Toradex bootloader plus a M7 firmware).

I don’t know if Torizon OS 7 is able to run the same firmware from DDR.

[ToradexAI]

Hi, I’m the Toradex AI Assistant. The Toradex Support team will follow up with you shortly!

Did you like my answer?

---

Hello,

To update the M7 firmware on your Verdin iMX8M-Plus using Torizon OTA features, you have a couple of options depending on how the firmware is integrated with the bootloader.

1. Bootloader Update Approach:

   • If the M7 firmware is part of the bootloader package, you can handle it as a bootloader update. Torizon Cloud allows you to update the bootloader, which can include additional firmware like the M7. However, bootloader updates are generally more risky and should be performed with caution.
   • Ensure that your Torizon Cloud account has the tdx-bootloader package source added. You can check this in the “Packages” section of your Torizon Cloud account.
   • Select the appropriate bootloader package version for your device and perform the update through Torizon Cloud. This process is similar to updating the OS or applications but must be done separately due to the risks involved.

2. Subsystem Update Approach:

   • If the M7 firmware can be treated as a separate subsystem, you can use the Subsystem Updates feature of Torizon Cloud. This involves configuring the M7 as a secondary device connected to the primary system (your SoM).
   • Create a 99-secondaries.toml file in the /etc/sota/conf.d directory to define the M7 as a secondary device.
   • Develop an action handler script that specifies how the system should update the M7 firmware. This script can automate the update process using a container or other mechanisms.
   • Use Torizon Cloud to manage and deploy the firmware updates to the M7 as a secondary device.

Both methods require careful preparation and testing to ensure that the update process is reliable and does not disrupt the system’s operation. Consider the integration level of the M7 firmware with the bootloader and the specific requirements of your application when choosing the approach.

Best regards,
Toradex AI Assistant

Please, also check if these links can help you solve your problem:

Torizon OS kernel panic when running M7 application on Verdin IMX8M Plus

I’m trying to get a HMP application running on our Verdin IMX8M Plus SoM. I’m following the instructions as outlined here: Heterogeneous Multi-core Processing (HMP) Documentation Overview | Toradex Developer Center. Thus far I’ve successfully acquired and installed the MCUXpresso SDK for Linux (SDK_2.15.000_MIMX8ML8xxxKZ) and the ARM toolchain and have compiled and loaded the “hello_world” demo application. If I run the application from Uboot (run m4boot) all seems well, I see “hello world” p…

Torizon cloud secure updates to 100's of units.

[i.staehli]

Hello @vix! I think we are working on similar topics.

I am able to update the M7 firmware with the subsystem update, explained in this webinar: Secure Over-the-Air Updates | Updating Cortex-M and External Devices

However, I only got it running with these setups:

• It works with Torizon OS 7.1.0 using RemoteProc and .elf file
• It works with Torizon OS 6.8.1 using U-Boot and .bin file

[jeremias.tx]

Greetings @vix,

As @i.staehli has pointed out, I would recommend viewing that webinar as a starting point for how to update M-core firmware using our OTA update feature.

Best Regards,
Jeremias

[vix]

Hi @i.staehli

thank you very much!
Yes, I think we’ve been working on similar topic.
At the moment my scenario is Torizon OS 6.8.1 + U-Boot + .bin file.

Here is the necessary steps, based on what I understood:

• customize Torizon OS with:
  • custom /etc/sota/conf.d/50-secondaries.toml - see here
  • custom /etc/sota/secondaries.json based on /usr/lib/sota/secondaries.json and adding m7-firmware secondary
  • a new action handler /etc/ota-subsystem/m7-firmware-update.sh
• upload the payload to Torizon Cloud, thus creating a custom update package - see here

Can you confirm the above steps?

For the action handler, I can take inspiration by this one that is for .elf file + remoteproc.
But I think that for the .bin file is much simpler

#!/usr/bin/bash
#
# Custom action handler for Cortex M7 .bin firmware updates
#

case "$1" in
    get-firmware-info)
        # Perform normal processing for this action.
        echo "ACTION HANDLER: get-firmware-info"
        exit 64
        ;;
    install)
        # Perform normal processing for this action.
        [ -d /var/m7-firmware ] || mkdir -p /var/m7-firmware
        [ -f /var/m7-firmware/firmware.bin ] && mv /var/m7-firmware/firmware.bin /var/m7-firmware/firmware.bin.rollback
        cp "${SECONDARY_FIRMWARE_PATH}" /var/m7-firmware/firmware.bin
        echo '{"status": "ok", "message": "New firmware installed"}'
        exit 0 
        ;;
    complete-install)
        # Perform normal processing for this action.
        echo "ACTION HANDLER: complete-install"
        exit 64
        ;;
    *)
        # Perform normal processing for this action.
        echo "ACTION HANDLER: ERROR event $1 not supported"
        exit 64
        ;;
esac

Does the action handler sond good to you?
Is it needed an additional step to reboot the device )so that u-boot can load the new firmware)?

[i.staehli]

@vix Yes, everything that you mentioned looks good.

A reboot is needed afterwards, which I did manually. I heard that if you want to do synchronous updates it won’t work if you reboot in the action handler, because it will interrupt the other updates. So I guess you need to also update the OS, which will then reboot.

Note: I found out that it also works with Torizon OS 7.1.0. You need to add the fw_setenv tdxargs "clk-imx8mm.mcore_booted=1" that is explained here How to Use RemoteProc | Toradex Developer Center

We are going to use the method with the RemoteProc and .elf file, which seems more straightforward. Although, the boot time is a bit worse.

[vix]

@i.staehli

I heard that if you want to do synchronous updates it won’t work if you reboot in the action handler, because it will interrupt the other updates. So I guess you need to also update the OS, which will then reboot.

Maybe toradex can help here, because triggering a final OS update only to have a final reboot (needed by a firmware update) is a little bit over-engineered IMHO.

@i.staehli

You need to add the fw_setenv tdxargs "clk-imx8mm.mcore_booted=1" that is explained here How to Use RemoteProc | Toradex Developer Center

I know, and I do this in my custom build of Torizon OS (torizoncore-builder).
Where do you add it?

[i.staehli]

@vix I added it in the tcbuild.yaml like this

  kernel:
    # >> Custom kernel arguments.
    arguments:
      - clk-imx8mp.mcore_booted=1 # prevents Linux from disabling the root clock of the Cortex M7

[vix]

@i.staehli

I added it in the tcbuild.yaml

Yes, this is what I do with Torizon OS 6.

Now I’m not able to create a custom package in Torizon Cloud that contains the firmware (using V2 web UI - Toradex asked to use V2 and no more V1).

[i.staehli]

To upload it to Torizon Cloud I used

torizoncore-builder platform push --credentials credentials.zip --hardwareid cortex-firmware --description "M7 Hello World demo" --package-name m7-firmware --package-version 0.0 ./hello_world_cm7.bin

[jeremias.tx]

Maybe toradex can help here, because triggering a final OS update only to have a final reboot (needed by a firmware update) is a little bit over-engineered IMHO.

If you want your subsystem update to require a reboot in order to complete then just have the install method send back a status of need-completion. For reference see how this is done in our bootloader action handler script here: meta-toradex-torizon/recipes-sota/config/files/bl_actions.sh at scarthgap-7.x.y · torizon/meta-toradex-torizon · GitHub

This will prompt the system to perform a reboot to complete the pending update. Keep in mind though after the reboot the update client will call the complete-install method of your action handler. Add logic to this if you think it’s needed to determine the success of your update.

Best Regards,
Jeremias

[vix]

I created the package in Torizon Cloud, and I see it

But I can’t trigger an update for this component to my device.
Here is the log from aktualizr

Feb 13 08:22:24 verdin-imx8mp-15128266 systemd[1]: Started Aktualizr SOTA Client.
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Aktualizr version tdx-9444e804 starting
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Reading config: "/usr/lib/sota/conf.d/20-sota-device-cred.toml"
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Reading config: "/usr/lib/sota/conf.d/30-rollback.toml"
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Reading config: "/usr/lib/sota/conf.d/40-hardware-id.toml"
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Reading config: "/etc/sota/conf.d/50-secondaries.toml"
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Reading config: "/usr/lib/sota/conf.d/60-polling-interval.toml"
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Reading config: "/etc/sota/conf.d/61-custom-polling-interval.toml"
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Reading config: "/usr/lib/sota/conf.d/70-reboot.toml"
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Reading config: "/etc/sota/conf.d/99-offline-updates.toml"
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Use existing SQL storage: "/var/sota/sql.db"
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Couldn`t import data: empty path received
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Initializing docker-compose Secondaries...
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Use existing SQL storage: "/var/sota/storage/docker-compose/sql.db"
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Adding Secondary with ECU serial: 1b6............aa w>
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Initializing torizon-generic Secondaries...
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Use existing SQL storage: "/var/sota/storage/bootloader/sql.db"
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: No valid Director metadata found in storage: Could not load latest root
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: No valid Image metadata found in storage: Could not load latest root
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Adding Secondary with ECU serial: c445afd......cb6f9 w>
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Primary ECU serial: 4305996.................8d2f0886028d with hardware I>
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Device ID: 4efec097-.......-b962-e34d82275c68
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Device Gateway URL: https://dgw.torizon.io
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Certificate subject: CN=4efec097-.........b962-e34d82275c68
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Certificate issuer: CN=ota-devices-CA
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Certificate valid from: Feb  5 08:56:40 2025 GMT until: Feb  5 08:56:40 2125 GMT
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: PROXY: initializing...
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: PROXY: using TCP port 8850.
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: Offline Updates are enabled
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: PROXY: starting thread.
Feb 13 08:22:24 verdin-imx8mp-15128266 aktualizr-torizon[955]: PROXY: listening to connections...
Feb 13 08:22:26 verdin-imx8mp-15128266 aktualizr-torizon[955]: Event: SendDeviceDataComplete
Feb 13 08:22:27 verdin-imx8mp-15128266 aktualizr-torizon[955]: Event: UpdateCheckComplete, Result - No updates available

This is what I added to secondaries.json

{
    "partial_verifying": false,
    "ecu_hardware_id": "verdin-imx8mp-m7-firmware",
    "full_client_dir": "/var/sota/storage/m7-firmware",
    "ecu_private_key": "sec.private",
    "ecu_public_key": "sec.public",
    "firmware_path": "/var/sota/storage/m7-firmware/firmware_imxcm7.bin",
    "target_name_path": "/var/sota/storage/m7-firmware/target_name",
    "metadata_path": "/var/sota/storage/m7-firmware/metadata",
    "action_handler_path": "/etc/ota-subsystem/m7-firmware-update.sh"
}

I think that "ecu_hardware_id" in secondaries.json should be the same passed with --hardwareid to torizoncore-builder, and seen in “Supported Component” column in the Torizon Cloud web UI.

/* EDIT */
I see that on my device, the folder /var/sota/storage has only the ‘toradex’ folders bootloader and docker-compose.
No m7-firmware folder (and metadata subfolder).
@drew.tx when is this folder created?
I don’t find info on how to create it manually.

I think I missed something.

[vix]

@drew.tx I think I got the point.
When I log as root to the SoM, I see that the file /etc/sota/secondaries.json is not the one I customized in my Torizon OS build.
Instead /etc/sota/conf.d/50-secondaries.toml is the one I customized (so I’m sure the Torizon OS has been updated) to read

[uptane]
secondary_config_file = "/etc/sota/secondaries.json"

The secondaries.json I customized is present in /usr/etc/sota/secondaries.json.
The one in /etc/sota/secondaries.json is the same as those in /usr/lib/sota/secondaries.json

Is there an issue in “mounting” /etc/sota/secondaries.json?

After I copied /usr/etc/sota/secondaries.json to /etc/sota/secondaries.json and stopped/started aktualizr, now the log seems ok

Feb 13 12:50:03 verdin-imx8mp-15128266 aktualizr-torizon[49126]: Initializing torizon-generic Secondaries...
Feb 13 12:50:04 verdin-imx8mp-15128266 aktualizr-torizon[49126]: created: /var/sota/storage/m7-firmware
Feb 13 12:50:04 verdin-imx8mp-15128266 aktualizr-torizon[49126]: created: /var/sota/storage/m7-firmware/metadata
Feb 13 12:50:04 verdin-imx8mp-15128266 aktualizr-torizon[49126]: Bootstrap empty SQL storage
Feb 13 12:50:04 verdin-imx8mp-15128266 aktualizr-torizon[49126]: Bootstraping DB to version 26
Feb 13 12:50:04 verdin-imx8mp-15128266 aktualizr-torizon[49126]: No valid Director metadata found in storage: Could not load latest root
Feb 13 12:50:04 verdin-imx8mp-15128266 aktualizr-torizon[49126]: No valid Image metadata found in storage: Could not load latest root
Feb 13 12:50:04 verdin-imx8mp-15128266 aktualizr-torizon[49126]: Adding Secondary with ECU serial: 88b2c2ea78ed302f1daad88dd15af69980a73d16f283c80df7ebc417ce1e4e6c>

and I’ve been able to update the firmware once.
But when I tried to installa nother version, here is the log:

Feb 13 13:49:01 verdin-imx8mp-15128266 aktualizr-torizon[934]: New updates found in Director metadata. Checking Image repo metadata...
Feb 13 13:49:01 verdin-imx8mp-15128266 aktualizr-torizon[934]: Downloading new Image repo Snapshot metadata because verification of local copy failed: Snapshot metadata hash verification failed
Feb 13 13:49:02 verdin-imx8mp-15128266 aktualizr-torizon[934]: Downloading new Image repo Targets metadata because verification of local copy failed: Snapshot hash mismatch for targets metadata
Feb 13 13:49:03 verdin-imx8mp-15128266 aktualizr-torizon[934]: 1 new update found in both Director and Image repo metadata.
Feb 13 13:49:03 verdin-imx8mp-15128266 aktualizr-torizon[934]: Event: UpdateCheckComplete, Result - Updates available
Feb 13 13:49:04 verdin-imx8mp-15128266 aktualizr-torizon[934]: Event: DownloadProgressReport, Progress at 1%
Feb 13 13:49:04 verdin-imx8mp-15128266 aktualizr-torizon[934]: Event: DownloadProgressReport, Progress at 2%
Feb 13 13:49:04 verdin-imx8mp-15128266 aktualizr-torizon[934]: Event: DownloadProgressReport, Progress at 3%
Feb 13 13:49:04 verdin-imx8mp-15128266 aktualizr-torizon[934]: Event: DownloadProgressReport, Progress at 7%
...
Feb 13 13:49:04 verdin-imx8mp-15128266 aktualizr-torizon[934]: Event: DownloadProgressReport, Progress at 98%
Feb 13 13:49:04 verdin-imx8mp-15128266 aktualizr-torizon[934]: Event: DownloadProgressReport, Progress at 99%
Feb 13 13:49:04 verdin-imx8mp-15128266 aktualizr-torizon[934]: Event: DownloadProgressReport, Progress at 100%
Feb 13 13:49:04 verdin-imx8mp-15128266 aktualizr-torizon[934]: Event: DownloadTargetComplete, Result - Success
Feb 13 13:49:04 verdin-imx8mp-15128266 aktualizr-torizon[934]: Event: AllDownloadsComplete, Result - Success
Feb 13 13:49:04 verdin-imx8mp-15128266 aktualizr-torizon[934]: Waiting for Secondaries to connect to start installation...
Feb 13 13:49:05 verdin-imx8mp-15128266 aktualizr-torizon[934]: Downloading new Image repo Snapshot metadata because verification of local copy failed: Snapshot metadata hash verification failed
Feb 13 13:49:05 verdin-imx8mp-15128266 aktualizr-torizon[934]: Downloading new Image repo Targets metadata because verification of local copy failed: Snapshot hash mismatch for targets metadata
Feb 13 13:49:06 verdin-imx8mp-15128266 aktualizr-torizon[934]: Event: InstallStarted
Feb 13 13:49:06 verdin-imx8mp-15128266 aktualizr-torizon[934]: No update to install on Primary
Feb 13 13:49:06 verdin-imx8mp-15128266 aktualizr-torizon[934]: Action-handler "/etc/ota-subsystem/m7-firmware-update.sh" message: New firmware installed
Feb 13 13:49:06 verdin-imx8mp-15128266 aktualizr-torizon[934]: Event: InstallTargetComplete, Result - Success
Feb 13 13:49:06 verdin-imx8mp-15128266 aktualizr-torizon[934]: Event: AllInstallsComplete, Result - OK
Feb 13 13:49:07 verdin-imx8mp-15128266 aktualizr-torizon[934]: Event: PutManifestComplete, Result - Success

Are the messages Snapshot metadata hash verification failed and Snapshot hash mismatch for targets metadata expected?

I see that /etc/sota/secondaries.json is preserved after the reboot.
But it’s not updated by a Torizon OS update if a file with the same name already exists.

[drew.tx]

Hi @vix,

Any file in /etc that has been modified on the device will not be updated by an OSTree revision update. This is a standard OSTree feature and you can find more details in the OSTree docs. That is my guess as to why your file was not updated. OStree keeps the pristine config files in /usr/etc and if you have modified a file in /etc on the device then you need to explicitly update it to match the one under /usr/etc.

It looks like you effectively did this when you mentioned copying the file above. Once you get all this setup into a proper torizoncore-builder project then you should have no need to manually edit those files on the device and will not see this behavior.

As for the hash mismatch, I don’t know what would cause that. Perhaps @jeremias.tx has an answer handy. I’ll also ask internally to see if anyone has suggestions. It may just be an artifact of the manual changes you were doing on the device.

Drew

[jeremias.tx]

Are the messages Snapshot metadata hash verification failed and Snapshot hash mismatch for targets metadata expected?

This is completely normal, if you read the full log line in it’s entirety:

Downloading new Image repo Snapshot metadata because verification of local copy failed: Snapshot metadata hash verification failed

All it’s saying is that it couldn’t verify the local copy of the metadata therefore it needs to download newer metadata. If it was hard error the update would not have proceeded.

Best Regards,
Jeremias

[vix]

@drew.tx

It looks like you effectively did this when you mentioned copying the file above.

I copied the files 50-secondaries.toml and secondaries.json because it’s what I found in the toradex documentation here.
And I copied them on the same device already provisioned where I tested everything, before including them in a Torizon OS customization.

Lesson learned: do not use a provisioned device to test.

[vix]

I’ve been able to update m7 firmware three or four times properly.
But then I initiated an update that bricked my SoM.
I had the console open, so I can give the logs.
This is from aktualizr

root@verdin-imx8mp-15128266:/var/rootdirs/home/torizon# journalctl -fu aktualizr-torizon
Feb 14 13:13:39 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Certificate issuer: CN=ota-devices-CA
Feb 14 13:13:39 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Certificate valid from: Feb 14 13:12:49 2025 GMT until: Feb 14 13:12:49 2125 GMT
Feb 14 13:13:39 verdin-imx8mp-15128266 aktualizr-torizon[1960]: PROXY: initializing...
Feb 14 13:13:39 verdin-imx8mp-15128266 aktualizr-torizon[1960]: PROXY: using TCP port 8850.
Feb 14 13:13:39 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Offline Updates are enabled
Feb 14 13:13:39 verdin-imx8mp-15128266 aktualizr-torizon[1960]: PROXY: starting thread.
Feb 14 13:13:39 verdin-imx8mp-15128266 aktualizr-torizon[1960]: PROXY: listening to connections...
Feb 14 13:13:41 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: SendDeviceDataComplete
Feb 14 13:13:41 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Current versions in storage and reported by OSTree do not match
Feb 14 13:13:43 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: UpdateCheckComplete, Result - No updates available
Feb 14 13:18:43 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Current versions in storage and reported by OSTree do not match
Feb 14 13:18:44 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Current version for ECU ID: 790ffbe2827d7ee2cc1bd555113e55ab4089387da529e540bd7597d90c980bd9 is unknown
Feb 14 13:18:44 verdin-imx8mp-15128266 aktualizr-torizon[1960]: New updates found in Director metadata. Checking Image repo metadata...
Feb 14 13:18:44 verdin-imx8mp-15128266 aktualizr-torizon[1960]: 1 new update found in both Director and Image repo metadata.
Feb 14 13:18:44 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: UpdateCheckComplete, Result - Updates available
Feb 14 13:18:44 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Current version for ECU ID: 790ffbe2827d7ee2cc1bd555113e55ab4089387da529e540bd7597d90c980bd9 is unknown
Feb 14 13:18:45 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: DownloadProgressReport, Progress at 1%
Feb 14 13:18:45 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: DownloadProgressReport, Progress at 2%
Feb 14 13:18:45 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: DownloadProgressReport, Progress at 3%
Feb 14 13:18:45 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: DownloadProgressReport, Progress at 4%
Feb 14 13:18:45 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: DownloadProgressReport, Progress at 5%
....
Feb 14 13:18:48 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: DownloadProgressReport, Progress at 97%
Feb 14 13:18:48 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: DownloadProgressReport, Progress at 98%
Feb 14 13:18:48 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: DownloadProgressReport, Progress at 99%
Feb 14 13:18:48 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: DownloadProgressReport, Progress at 100%
Feb 14 13:18:49 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: DownloadTargetComplete, Result - Success
Feb 14 13:18:49 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: AllDownloadsComplete, Result - Success
Feb 14 13:18:49 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Current version for ECU ID: 790ffbe2827d7ee2cc1bd555113e55ab4089387da529e540bd7597d90c980bd9 is unknown
Feb 14 13:18:49 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Waiting for Secondaries to connect to start installation...
Feb 14 13:18:50 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: InstallStarted
Feb 14 13:18:50 verdin-imx8mp-15128266 aktualizr-torizon[1960]: No update to install on Primary
Feb 14 13:18:50 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Action-handler "/etc/ota-subsystem/m7-firmware-update.sh" message: New firmware installed
Feb 14 13:18:51 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: InstallTargetComplete, Result - Success
Feb 14 13:18:51 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: AllInstallsComplete, Result - OK
Feb 14 13:18:51 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Current versions in storage and reported by OSTree do not match
Feb 14 13:18:51 verdin-imx8mp-15128266 aktualizr-torizon[1960]: Event: PutManifestComplete, Result - Success

and it seems ok.

But this is the log from the next Torizon boot

U-Boot SPL 2022.04-6.8.1+git.a31fdd543bee (Jan 01 1970 - 00:00:00 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
Training FAILED
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
DDR configured as single rank
SEC0:  RNG instantiated
Normal Boot
WDT:   Started watchdog@30280000 with servicing (60s timeout)
Trying to boot from BOOTROM
Boot Stage: Primary boot
Find img info 0x&4802a600, size 888
Need continue download 1024
Download 951296, Total size 952656
NOTICE:  BL31: v2.6(release):lf_v2.6-g3c1583ba0a
NOTICE:  BL31: Built : 00:00:00, Jan  1 1970


U-Boot 2022.04-6.8.1+git.a31fdd543bee (Jan 01 1970 - 00:00:00 +0000)

CPU:   i.MX8MP[6] rev1.1 1600 MHz (running at 1200 MHz)
CPU:   Industrial temperature grade (-40C to 105C) at 47C
Reset cause: POR
DRAM:  2 GiB
Core:  89 devices, 23 uclasses, devicetree: separate
WDT:   Started watchdog@30280000 with servicing (60s timeout)
MMC:   FSL_SDHC: 1, FSL_SDHC: 2
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
Model: Toradex 0064 Verdin iMX8M Plus Quad 2GB WB IT V1.1A
Serial#: 15128266
get_tdx_eeprom: cannot find EEPROM by node
MISSING TORADEX CARRIER CONFIG BLOCKS
get_tdx_eeprom: cannot find EEPROM by node
SEC0:  RNG instantiated

 BuildInfo:
  - ATF 3c1583b

flash target is MMC:2
Net:   eth1: ethernet@30be0000, eth0: ethernet@30bf0000 [PRIME]
Fastboot: Normal
Normal Boot
Hit any key to stop autoboot:  0
Failed to load '/ostree/deploy/torizon/var/firmware_imxcm7.bin'
## Starting auxiliary core stack = 0x00800400, pc = 0x00000000...
switch to partitions #0, OK
mmc2(part 0) is current device
Scanning mmc 2:1...
Found U-Boot script /boot.scr
973 bytes read in 1 ms (950.2 KiB/s)
## Executing script at 50280000
6705 bytes read in 2 ms (3.2 MiB/s)
88627 bytes read in 2 ms (42.3 MiB/s)
102 bytes read in 2 ms (49.8 KiB/s)
Applying Overlay: verdin-imx8mp_hdmi_overlay.dtbo
2091 bytes read in 3 ms (680.7 KiB/s)
Applying Overlay: verdin-imx8mp_hmp_overlay.dtbo
2503 bytes read in 3 ms (814.5 KiB/s)
Applying Overlay: custom-kargs_overlay.dtbo
200 bytes read in 2 ms (97.7 KiB/s)
13404218 bytes read in 47 ms (272 MiB/s)
6318867 bytes read in 23 ms (262 MiB/s)
   Uncompressing Kernel Image
## Flattened Device Tree blob at 50200000
   Booting using the fdt blob at 0x50200000
   Loading Device Tree to 00000000bfac0000, end 00000000bfaf8fff ... OK
Modify /vipsi@38500000:status disabled

Starting kernel ...

[    1.012762] clk: failed to reparent hsio_axi to sys_pll2_500m: -16
[    1.023678] clk: failed to reparent hsio_axi to sys_pll2_500m: -16
[    1.036602] Unable to handle kernel paging request at virtual address ffff80000a4f4fff
[    1.044551] Mem abort info:
[    1.047359]   ESR = 0x0000000096000007
[    1.051110]   EC = 0x25: DABT (current EL), IL = 32 bits
[    1.056425]   SET = 0, FnV = 0
[    1.059484]   EA = 0, S1PTW = 0
[    1.062625]   FSC = 0x07: level 3 translation fault
[    1.067543] Data abort info:
[    1.070427]   ISV = 0, ISS = 0x00000007
[    1.074262]   CM = 0, WnR = 0
[    1.077241] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000049c61000
[    1.083944] [ffff80000a4f4fff] pgd=10000000bffff003, p4d=10000000bffff003, pud=10000000bfffe003, pmd=10000000400e6003, pte=0000000000000000
[    1.096485] Internal error: Oops: 0000000096000007 [#1] PREEMPT SMP
[    1.102754] Modules linked in:
[    1.105813] CPU: 1 PID: 102 Comm: kworker/u8:1 Not tainted 5.15.148-6.8.1+git.1cbf48124747 #1-TorizonCore
[    1.115383] Hardware name: Toradex Verdin iMX8M Plus WB on CEMB 100134 Board (DT)
[    1.119683] mmc0: new ultra high speed SDR104 SDIO card at address 0001
[    1.122866] Workqueue: events_unbound deferred_probe_work_func
[    1.135312] pstate: 00000005 (nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    1.142280] pc : rproc_handle_resources.constprop.0+0xb8/0x1a4
[    1.148121] lr : rproc_boot+0x45c/0x620
[    1.151961] sp : ffff80000a81b9e0
[    1.155277] x29: ffff80000a81b9e0 x28: ffff80000a4f5003 x27: 00000000ffffffff
[    1.162419] x26: ffff8000099031e0 x25: 0000000000000000 x24: ffff000000af5038
[    1.169566] x23: ffffffffffffffff x22: ffff80000a4f5000 x21: 00000000000003fd
[    1.176712] x20: ffff000000af5000 x19: 0000000000000000 x18: fffffffffffc48d0
[    1.183860] x17: 000000040044ffff x16: 00500072b5503510 x15: fffffffffff848cf
[    1.191004] x14: 0000000000000001 x13: 0000000000000000 x12: 0000000000000003
[    1.198148] x11: 0101010101010101 x10: 0000000000000037 x9 : 0000000000000000
[    1.205295] x8 : ffff000000c0bf00 x7 : 0000000000000000 x6 : 000000000000003f
[    1.212443] x5 : 0000000000000040 x4 : ffff80000a81b910 x3 : 0000000000000000
[    1.219586] x2 : 0000000000000400 x1 : ffff80000a1326e0 x0 : ffff80000a4f5000
[    1.226733] Call trace:
[    1.229180]  rproc_handle_resources.constprop.0+0xb8/0x1a4
[    1.234673]  rproc_boot+0x45c/0x620
[    1.238164]  rproc_add+0xd0/0x170
[    1.241482]  imx_rproc_probe+0x510/0x710
[    1.245410]  platform_probe+0x68/0xe0
[    1.249079]  really_probe+0xbc/0x460
[    1.252659]  __driver_probe_device+0x100/0x160
[    1.257107]  driver_probe_device+0x40/0x120
[    1.261297]  __device_attach_driver+0xbc/0x160
[    1.265743]  bus_for_each_drv+0x7c/0xe0
[    1.269582]  __device_attach+0xac/0x1f0
[    1.273425]  device_initial_probe+0x14/0x20
[    1.277612]  bus_probe_device+0x9c/0xb0
[    1.281454]  deferred_probe_work_func+0x94/0xf0
[    1.285988]  process_one_work+0x1d0/0x4a0
[    1.290004]  worker_thread+0x2c0/0x490
[    1.293760]  kthread+0x150/0x160
[    1.296994]  ret_from_fork+0x10/0x20
[    1.300578] Code: f94037e0 aa1a03e2 aa1803e1 97d05d42 (b8776ac1)
[    1.306678] ---[ end trace 89de641f8b26b9d3 ]---
[    1.311297] Kernel panic - not syncing: Oops: Fatal exception
[    1.317046] SMP: stopping secondary CPUs
[    1.321284] Kernel Offset: disabled
[    1.324773] CPU features: 0x0,00002001,20000846
[    1.329307] Memory Limit: none
[    1.332366] Rebooting in 5 seconds..

You can see the message
Failed to load '/ostree/deploy/torizon/var/firmware_imxcm7.bin'
and I think that after trying to load rpmsg driver a fatal error happens and so the system reboots.

Can you suggest something to make this kind of update much more robust?

[drew.tx]

I can see where those docs need some improvement. I’ll chat with the team to see if we can make it more clear.

I don’t think the issue is that the device was provisioned, just that, if you are manually modifying a device, then you probably want to reflash with the Easy Installer before you start deploying tcb-created images OTA.

Drew

[drew.tx]

Hi @vix

I see the error but there does not appear to be any info as to why it failed to load. Does the file exist? Is it correct? Do you have filesystem corruption?

I don’t know why loading the module would cause the kernel to oops but I can’t say that I’ve tried this exact scenario. It is concerning though. I’ll see if anyone internally has any ideas and report back.

Drew

[jeremias.tx]

Hi @vix,

Could you share what commands you have running in U-Boot to boot/start the M-core on your device?

The crash you showed seems somewhat similar to another issue we had in the past. But this was specific to 1.1B revision modules and based on your logs you seem to have a 1.1A. Also the fix for the 1.1B was to set reinitalize a region of memory in U-Boot like so mw.w 0x550ff000 0 64. We already added this command to our documentation, so if you copied the commands there you should have this already.

If you trying to load your m7-binary without the update mechanism does the system boot fine then? If yes this could indicate something went wrong during the update procedure.

Best Regards,
Jeremias