Cortex M7 Subsystem Update Action handler

Hi there,

I am struggling with the action handler of my M7 subsystem update. I am using Uboot, remoteproc is not an option atm.

The update itself works (copying of binary into correct location, then reboot), but on TorizonCloud, the update shows up as failed. My suspicion is that since i reboot in the action handler (in the install step), the update is seen as failed by aktualizr. So what would be the correct way to do this? When can i actually reboot?

case "$1" in
    get-firmware-info)
        # Perform normal processing for this action.
        echo "ACTION HANDLER: get-firmware-info"
        exit 64
        ;;
    install)
        # Perform normal processing for this action.
        M7_FIRMWARE=$1
  
        echo "Installing ${M7_FIRMWARE}"
        cp "${M7_FIRMWARE}" /var/m7-firmware/m7-cortex.bin


        echo '{"status": "need-completion", "message": "rebooting soon"}'
        /usr/sbin/reboot
        exit 0
        ;;
    complete-install)
        # Perform normal processing for this action.
        echo "ACTION HANDLER: complete-install"
        exit 64
        ;;
    *)
        # Perform normal processing for this action.
        echo "ACTION HANDLER: ERROR event $1 not supported"
        exit 64
        ;;
esac

torizon@verdin-imx8mp-15604840:~$ sudo tdx-info

Software summary
------------------------------------------------------------
Bootloader:               U-Boot
Kernel version:           6.6.54-7.1.0-g3493ccd66900 #1-Torizon SMP PREEMPT Tue Dec 17 21:04:41 UTC 2024
Kernel command line:      root=LABEL=otaroot rootfstype=ext4 quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3 ostree=/ostree/boot.1/torizon/8542e3fe3c580e148eff80535c0677db89fb057703cedffb3b88b7018d3cc5f2/0 clk-imx8mp.mcore_booted=1
Distro name:              NAME="Torizon OS"
Distro version:           VERSION_ID=7.1.0-build.4
Distro variant:           VARIANT="Docker"
Hostname:                 verdin-imx8mp-15604840
------------------------------------------------------------

Hardware info
------------------------------------------------------------
HW model:                 Toradex Verdin iMX8M Plus WB on Verdin Development Board
Toradex version:          0058 V1.1B
Serial number:            15604840
Processor arch:           aarch64
------------------------------------------------------------

Hi, I’m the Toradex AI Assistant.

I might make mistakes. Our support team will step in if needed. If I answered your question, please tick the “Solution” checkbox. For follow-up questions to me, please use @ToradexAI in your post.

Did you like my answer? :+1: :-1:


Hello! Thank you for describing your current setup in detail.

You’re correct in your assumption: triggering a reboot directly inside the action handler (for example, in your install step) will interfere with how Aktualizr manages the subsystem update process. If the system reboots unexpectedly during the install phase, Aktualizr considers the update as failed because it never receives a proper completion signal from the action handler.

Correct Way to Handle Subsystem Updates

Here’s the recommended approach for Torizon subsystem updates:

  1. Install Action:

    • Copy the firmware to the appropriate location.
    • Do not reboot here!
    • Respond with a JSON status indicating that completion (and thus, a reboot) is needed:
      echo '{"status": "need-completion", "message": "Installation requires a reboot."}'
      exit 0
      
    • At this point, Aktualizr will schedule the required reboot.
  2. Reboot Control:

    • Let Aktualizr/Torizon OS manage the reboot. It will automatically reboot the device if the install action handler returns "need-completion" status.
  3. Complete-Install Action:

    • This action is called automatically after the system reboots.
    • Here you should verify your installation (e.g., check if the firmware file is present and valid).
    • If everything is correct, respond with:
      echo '{"status": "ok", "message": "Firmware update complete."}'
      exit 0
      
    • If verification fails, respond with "failed" and a suitable message.

Example Update to Your Script

case "$1" in
    install)
        M7_FIRMWARE="$2"
        echo "Installing ${M7_FIRMWARE}"
        cp "${M7_FIRMWARE}" /var/m7-firmware/m7-cortex.bin
        echo '{"status": "need-completion", "message": "Installation requires a reboot."}'
        exit 0
        ;;
    complete-install)
        # Verify the firmware installation here
        if [ -f /var/m7-firmware/m7-cortex.bin ]; then
            echo '{"status": "ok", "message": "Firmware update complete."}'
            exit 0
        else
            echo '{"status": "failed", "message": "Firmware not found after reboot."}'
            exit 1
        fi
        ;;
    # ... other cases ...
esac

Key Point

  • Never call reboot directly within your handler. The platform will handle the reboot cycle after the "need-completion" status is returned.

For further information, see:

Let me know if you need more details or if you’d like help with another aspect of this workflow!

Best regards,
Toradex AI Assistant.

I removed the reboot command, but I ran into another issue:

Jun 10 14:58:04 verdin-imx8mp-15604840 aktualizr-torizon[1263]: Event: DownloadTargetComplete, Result - Success
Jun 10 14:58:04 verdin-imx8mp-15604840 aktualizr-torizon[1263]: Event: AllDownloadsComplete, Result - Success
Jun 10 14:58:04 verdin-imx8mp-15604840 aktualizr-torizon[1263]: Invalid role in root.json
Jun 10 14:58:04 verdin-imx8mp-15604840 aktualizr-torizon[1263]: Invalid role in root.json
Jun 10 14:58:04 verdin-imx8mp-15604840 aktualizr-torizon[1263]: Current version for ECU ID: c15f602655b1db1c8c0a82ebf4fac86f049bd8aab5044fea9336e5bc642a19d3 is unknown
Jun 10 14:58:04 verdin-imx8mp-15604840 aktualizr-torizon[1263]: Waiting for Secondaries to connect to start installation...
Jun 10 14:58:05 verdin-imx8mp-15604840 aktualizr-torizon[1263]: Invalid role in root.json
Jun 10 14:58:05 verdin-imx8mp-15604840 aktualizr-torizon[1263]: Invalid role in root.json
Jun 10 14:58:05 verdin-imx8mp-15604840 aktualizr-torizon[1263]: Invalid role in root.json
Jun 10 14:58:05 verdin-imx8mp-15604840 aktualizr-torizon[1263]: Invalid role in root.json
Jun 10 14:58:06 verdin-imx8mp-15604840 aktualizr-torizon[1263]: Event: InstallStarted
Jun 10 14:58:06 verdin-imx8mp-15604840 aktualizr-torizon[1263]: No update to install on Primary
Jun 10 14:58:06 verdin-imx8mp-15604840 aktualizr-torizon[1263]: Action-handler "/etc/sota/action-handler.sh" (action=install) output could not be parsed (expecting JSON string)
Jun 10 14:58:06 verdin-imx8mp-15604840 aktualizr-torizon[1263]: Event: InstallTargetComplete, Result - Error
Jun 10 14:58:06 verdin-imx8mp-15604840 aktualizr-torizon[1263]: Event: AllInstallsComplete, Result - m7-firmware:GENERAL_ERROR
Jun 10 14:58:06 verdin-imx8mp-15604840 aktualizr-torizon[1263]: Current versions in storage and reported by OSTree do not match
Jun 10 14:58:06 verdin-imx8mp-15604840 aktualizr-torizon[1263]: Event: PutManifestComplete, Result - Success


What am I missing?
I see that there is some json error, maybe this is related?

Update: I currently have this code:

case "$1" in
    get-firmware-info)
        # Perform normal processing for this action.
        echo "ACTION HANDLER: get-firmware-info"
        exit 64
        ;;
    install)
        # Perform normal processing for this action.
        [ -d /var/m7-firmware ] || mkdir -p /var/m7-firmware
        [ -f /var/m7-firmware/m7-cortex_tcm.tar ] && mv /var/m7-firmware/m7-cortex_tcm.tar /var/m7-firmware/m7-cortex_tcm.tar.rollback

        M7_FIRMWARE_TAR=${SECONDARY_FIRMWARE_PATH}

        BINARIES_OUTPUT_DIRECTORY="./m7-binaries"

        mkdir -p "${BINARIES_OUTPUT_DIRECTORY}"

        tar -xvf "${M7_FIRMWARE_TAR}" -C "${BINARIES_OUTPUT_DIRECTORY}"

        # Find binaries
        M7_BINARY_FILE_NAME_TCM=$(find "${BINARIES_OUTPUT_DIRECTORY}" -type f -name '*_tcm.bin' -printf '%f\n' | head -n 1)
        M7_BINARY_FILE_NAME_DDR=$(find "${BINARIES_OUTPUT_DIRECTORY}" -type f -name '*_ddr.bin' -printf '%f\n' | head -n 1)

        if [[ -z "$M7_BINARY_FILE_NAME_TCM" || -z "$M7_BINARY_FILE_NAME_DDR" ]]; then
            exit 1
        fi

        cp ${BINARIES_OUTPUT_DIRECTORY}/${M7_BINARY_FILE_NAME_TCM} /var/m7-firmware/m7-cortex_tcm.bin
        cp ${BINARIES_OUTPUT_DIRECTORY}/${M7_BINARY_FILE_NAME_DDR} /var/m7-firmware/m7-cortex_ddr.bin

        echo '{"status": "need-completion", "message": "rebooting soon"}'
        exit 0
        ;;
    complete-install)
        # Perform normal processing for this action.
        echo "ACTION HANDLER: complete-install"
        exit 0
        ;;
    *)
        # Perform normal processing for this action.
        echo "ACTION HANDLER: ERROR event $1 not supported"
        exit 64
        ;;
esac

which somehow runs into this error:

Jun 10 15:22:08 verdin-imx8mp-15604840 aktualizr-torizon[1240]: Action-handler "/etc/sota/action-handler.sh" (action=install) output could not be parsed (expecting JSON string)
Jun 10 15:22:08 verdin-imx8mp-15604840 aktualizr-torizon[1240]: Event: InstallTargetComplete, Result - Error

I’m guessing the echo will get interpreted, or does something else trigger some parsing which causes the error?
Edit: With higher loglevel:

Action-handler "/etc/sota/action-handler.sh" (action=install) started
Action-handler "/etc/sota/action-handler.sh" (action=install) finished with exit code 0
Action-handler "/etc/sota/action-handler.sh" (action=install) output could not be parsed (expecting JSON string)
JSON parse errors* Line 1, Column 1
  Syntax error: value, object or array expected.

Okay so I found out that the tar -xvf actually outputs some stuff to stdout, which is then parsed, causing the error. When redirecting the output to /dev/null, the command seems to work. However, I am still missing a reboot.

Jun 10 16:13:44 verdin-imx8mp-15604840 aktualizr-torizon[1250]: Action-handler "/etc/sota/action-handler.sh" message: rebooting soon
Jun 10 16:13:44 verdin-imx8mp-15604840 aktualizr-torizon[1250]: Event: InstallTargetComplete, Result - Error
Jun 10 16:13:44 verdin-imx8mp-15604840 aktualizr-torizon[1250]: Event: AllInstallsComplete, Result - NEED_COMPLETION

No more JSON issues, but there is still no reboot… Does aktualizr need to be somehow configured to reboot with the need-completion status?

Greetings @TimSiegrist,

Okay so I found out that the tar -xvf actually outputs some stuff to stdout, which is then parsed, causing the error.

Just to confirm your observation is correct here. The only stdout output from your action handler should be the json string. Any other unexpected output will cause issues. You need to make sure any commands/utilities in your script do not produce additional stdout output. This includes any error messages/logs if a command in your script were to fail.

Does aktualizr need to be somehow configured to reboot with the need-completion status?

Please refer to our action handler for bootloader updates to see how a reboot can be requested. Specifically this section here: meta-toradex-torizon/recipes-sota/config/files/bl_actions.sh at scarthgap-7.x.y · torizon/meta-toradex-torizon · GitHub

Best Regards,
Jeremias

Hi @TimSiegrist

pay attention that the cp command is not guaranteed to be atomic in Linux.
I mean here

cp "${M7_FIRMWARE}" /var/m7-firmware/m7-cortex.bin

And if the reboot happens in the middle of cp you can have empty or corrupted firmware (I saw this).

You cannot use mv synce the source folder is read only.
You must add an explicit sync befor reboot.

@jeremias.tx
Thanks for the answer.
I can request a reboot by creating the /run/need-reboot file, and now the update is successful.
I also added the sync, thanks for the input @vix
Cheers
Tim

Glad we were able to assist you.

Best Regards,
Jeremias

Hi @jeremias.tx

I have a follow-up question regarding the action handler and Aktualizr, in regards to a OS update.
If we perform an update with Torizon Cloud and select both the OS and a subsystem, how is the reboot handled? Does Aktualizr recognize that the subsystem and the OS both need reboots and will just reboot once? Or should we expect two reboots, one from the OS and one from the subsystem?

Best Regards,
Tim

In the latest version of Aktualizr the behavior is as follows. The update client should perform all the updates requested. Then if any of the updates performed reported that they require a reboot to finish, then a reboot to the system will be requested.

In your specific case. The OS update always installs first and sets an internal flag to reboot later once all the other updates have been installed. Then the update client will move onto the secondaries like your subsystem, in no particular order if there are multiple secondaries. In your action handler since the reboot sentinel file (/run/need-reboot) is created, this will trigger a reboot shortly afterwards. If your subsystem did not create this file or did not require a reboot, then the system will still reboot due to the internal flag that was set by the OS part of the update.

In summary, I only expect one reboot in your case. Is that okay for your system or did you need two reboots for some reason?

Best Regards,
Jeremias

Hi @jeremias.tx

One reboot is totally fine (2 reboots would be a bit strange). We need the reboot since we are booting the M7 from uboot now, in the best case everything is updated and then one reboot happens. As far as I see, this is this case now. We still want to create the reboot sentinel file in the subsystem in case we want to update only that one though.

Best Regards
Tim

One reboot is totally fine (2 reboots would be a bit strange).

Alright, so sounds like everything is okay then correct?

Yes, with the setup now everything works as intended.
Best regards
Tim

Glad we were able to provide assistance.

Best Regards,
Jeremias