Update in progress stuck

Hello

I just did an OS update OTA and it got stuck with “Update in progress”.

I looked at the output of aktualizr, but I don’t know what the problem is:

torizon@verdin-imx8mp-15604835:~$ journalctl -fu aktualizr*
Feb 03 14:57:49 verdin-imx8mp-15604835 aktualizr-torizon[955]: Certificate valid from: Jan 19 13:27:25 2026 GMT until: Jan 19 13:27:25 2126 GMT
Feb 03 14:57:49 verdin-imx8mp-15604835 aktualizr-torizon[955]: Offline Updates are disabled
Feb 03 14:57:49 verdin-imx8mp-15604835 aktualizr-torizon[955]: Registering with D-Bus
Feb 03 14:57:51 verdin-imx8mp-15604835 aktualizr-torizon[955]: Event: SendDeviceDataComplete
Feb 03 14:57:51 verdin-imx8mp-15604835 aktualizr-torizon[955]: Current versions in storage and reported by OSTree do not match
Feb 03 14:57:51 verdin-imx8mp-15604835 aktualizr-torizon[955]: Event: UpdateCheckComplete, Result - No updates available
Feb 03 15:02:51 verdin-imx8mp-15604835 aktualizr-torizon[955]: Current versions in storage and reported by OSTree do not match
Feb 03 15:02:51 verdin-imx8mp-15604835 aktualizr-torizon[955]: Event: UpdateCheckComplete, Result - No updates available
Feb 03 15:07:51 verdin-imx8mp-15604835 aktualizr-torizon[955]: Current versions in storage and reported by OSTree do not match
Feb 03 15:07:51 verdin-imx8mp-15604835 aktualizr-torizon[955]: Event: UpdateCheckComplete, Result - No updates available

I can’t cancel the update or install a new one. What else should I look at to know what the problem might be?

Kind regards

Ian

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.


Based on the logs you’ve shared, the message “Current versions in storage and reported by OSTree do not match” is not an error but rather an informational statement(1). This message appears when Aktualizr’s SQL storage contains different version information than what OSTree reports, which can happen in several scenarios(1).

Understanding the Log Message

The message occurs because Aktualizr stores version information only if it came from an update(1). If you’ve never performed an OS/OSTree update through Aktualizr, the storage would be empty while OSTree reports the current version(1). This mismatch is expected and not inherently problematic(1).

Troubleshooting Your Stuck Update

Since your logs show “No updates available” but you mention the update is stuck at “Update in progress,” there may be a disconnect between the Torizon Cloud interface and the actual device state. Here are steps to investigate:

1. Check OSTree Status

To understand your current OSTree deployment status, you can examine the OSTree state directly. OSTree manages multiple bootable filesystem versions and handles atomic updates(2).

2. Review Complete Aktualizr Logs

Your current logs only show the polling activity. To get a complete picture, check the full Aktualizr logs around the time the update was initiated:

sudo journalctl -u aktualizr* --since "1 hour ago"

3. Verify Update Configuration

Aktualizr is configured via .toml files in /usr/lib/sota/conf.d/ and /etc/sota/conf.d/(3). The default polling frequency is every 5 minutes(3), which matches your log timestamps.

4. Check for Pending Reboots

Torizon OS is configured to automatically reboot after successful OS updates(3). If automatic reboot is disabled, the update process might appear stuck. Check if there’s a pending reboot:

systemctl status ostree-pending-reboot.path

Next Steps

  1. Restart Aktualizr: Try restarting the Aktualizr service to refresh its state:

    sudo systemctl restart aktualizr-torizon
    
  2. Check Device Status: Verify your device’s connection status in the Torizon Cloud interface to ensure it’s properly communicating with the server.

  3. Review Full Logs: Examine the complete Aktualizr logs from when the update was initiated to identify any error messages or status changes that might not appear in the current polling logs.

The “Current versions in storage and reported by OSTree do not match” message itself is not the cause of your stuck update - it’s a normal informational message(1).

Best regards, Toradex AI Assistant.


Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

Here are more logs from aktualizr:

torizon@verdin-imx8mp-15604835:~$ sudo journalctl -u aktualizr* --since "1 hour ago"
Password:
Feb 03 14:47:15 verdin-imx8mp-15604835 systemd[1]: Started Aktualizr SOTA Client.
Feb 03 14:47:15 verdin-imx8mp-15604835 (-torizon)[771]: aktualizr-torizon.service: Referenced but unset environment variable evaluates to an empty string: AKTUALIZR_CMDLINE_PARAMETERS
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Aktualizr version tdx-a8e5ac1a4 starting
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Reading config: "/usr/lib/sota/conf.d/20-sota-device-cred.toml"
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Reading config: "/usr/lib/sota/conf.d/30-rollback.toml"
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Reading config: "/usr/lib/sota/conf.d/40-hardware-id.toml"
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Reading config: "/usr/lib/sota/conf.d/50-secondaries.toml"
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Reading config: "/usr/lib/sota/conf.d/60-polling-interval.toml"
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Reading config: "/usr/lib/sota/conf.d/70-reboot.toml"
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Use existing SQL storage: "/var/sota/sql.db"
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Couldn`t import data: empty path received
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Initializing docker-compose Secondaries...
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Use existing SQL storage: "/var/sota/storage/docker-compose/sql.db"
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: No valid Director metadata found in storage: Could not load latest root
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: No valid Image metadata found in storage: Could not load latest root
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Adding Secondary with ECU serial: 54c0f96627a1955aa39a2ca874c5f07f0503a11ebafc1bfb489ddcdbc8cc03cc with hardware ID: docker-compose
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Initializing torizon-generic Secondaries...
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Use existing SQL storage: "/var/sota/storage/bootloader/sql.db"
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: No valid Director metadata found in storage: Could not load latest root
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: No valid Image metadata found in storage: Could not load latest root
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Adding Secondary with ECU serial: e6ad8d356c9ad3fdd738789735c51b6a28193a67bbb8fcc79e34efe5bbf33c6e with hardware ID: verdin-imx8mp-bootloader
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Initializing torizon-generic Secondaries...
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Use existing SQL storage: "/var/sota/storage/fuse/sql.db"
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: No valid Director metadata found in storage: Could not load latest root
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: No valid Image metadata found in storage: Could not load latest root
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Adding Secondary with ECU serial: 70d3f09b764ec6ae71217e34b2a0898b22858c3844494eeaa82b86d77d93d7b5 with hardware ID: verdin-imx8mp-fuses
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Initializing torizon-generic Secondaries...
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: Use existing SQL storage: "/var/sota/storage/m7-firmware/sql.db"
Feb 03 14:47:15 verdin-imx8mp-15604835 aktualizr-torizon[771]: No valid Image metadata found in storage: The timestamp metadata was expired.

Here is the output for the pending reboot:

torizon@verdin-imx8mp-15604835:~$ systemctl status ostree-pending-reboot.path
* ostree-pending-reboot.path - Aktualizr OSTree Update Automatic Reboot
     Loaded: loaded (/usr/lib/systemd/system/ostree-pending-reboot.path; enabled; preset: enabled)
     Active: active (waiting) since Tue 2026-02-03 14:47:06 UTC; 29min ago
   Triggers: * ostree-pending-reboot.service

Notice: journal has been rotated since unit was started, output may be incomplete.

Restarting aktualizr doesn’t work, the update still doesn’t install.

Analysis of Your Update Issue

Based on your additional logs, I can see several important details about your system’s state. The logs show that Aktualizr is initializing multiple secondaries including docker-compose, bootloader, fuses, and M7 firmware components (1). The “No valid Director metadata found in storage” and “No valid Image metadata found in storage” messages indicate that the device hasn’t successfully completed the initial metadata synchronization with the update server (1).

Key Observations

1. Metadata Issues

Your logs show multiple instances of “Could not load latest root” which suggests the device is having trouble establishing proper communication with the Torizon Cloud update infrastructure (1).

2. Automatic Reboot Service Status

The ostree-pending-reboot.path service is active and waiting, which is the correct state. This service monitors for pending OSTree updates and triggers automatic reboots when needed (1).

3. Multiple Secondary Components

Your system is configured with several secondary update components:

  • docker-compose secondary
  • bootloader secondary
  • fuses secondary
  • M7 firmware secondary

This indicates a complex update configuration that may require all components to be properly synchronized (1).

Troubleshooting Steps

1. Check OSTree Status Directly

Run the following command to see the actual OSTree deployment status:

sudo ostree admin status

This will show you the current and pending deployments, helping identify if an update is actually staged.

2. Verify Network Connectivity

Ensure your device can reach the Torizon Cloud servers:

curl -I https://dgw.torizon.io

3. Check for Blocked Updates

Torizon OS provides mechanisms to allow and block updates (2). Verify if updates are currently blocked:

sudo systemctl status aktualizr-torizon

4. Enable Debug Logging

To get more detailed information about what’s happening, you can temporarily increase the logging level. Create a custom configuration file (1):

sudo mkdir -p /etc/sota/conf.d/
cat <<EOF | sudo tee /etc/sota/conf.d/99-custom-loglevel.toml
[logger]
loglevel = 1
EOF

Then restart Aktualizr:

sudo systemctl restart aktualizr-torizon

5. Manual Update Check

You can force Aktualizr to check for updates immediately rather than waiting for the polling interval:

sudo systemctl kill -s SIGUSR1 aktualizr-torizon

Understanding the Update Process

Torizon updates are designed to be power-cut tolerant and atomic operations (2). The system uses OSTree for reliable updates with automatic rollback capabilities if issues occur (2). When an update is “stuck,” it’s often due to metadata synchronization issues rather than the actual update process failing.

Next Steps

  1. Check OSTree status to understand the current deployment state
  2. Enable debug logging to get more detailed information about the update process
  3. Verify network connectivity to ensure communication with Torizon Cloud
  4. Monitor the logs after enabling debug mode to identify specific error conditions

If these steps don’t resolve the issue, the problem may be related to the specific update package or server-side configuration that requires further investigation.

Best regards, Toradex AI Assistant.


Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

Greetings @i.staehli,

Could you provide the device UUID of this specific device, as well as the email of the account this device is provisioned to.

Also, if possible could you leave this specific device in this state in the meanwhile? It will allow our team to investigate what could have happened to this device to put it into this state.

Finally, do you happen to know when the last time this device performed an update?

Best Regards,
Jeremias

Hi @jeremias.tx

I digged a bit more into the issue and found a way to reproduce it. What I did is:

  1. Have the software version A installed on the device

  2. Install the software version B with Torizon Cloud OTA

  3. Manually rollback the software to version A

    1. Check the installed versions with ostree admin status

    2. Rollback with sudo ostree admin deploy to software A

    3. Reboot the device

  4. Install the software version B with Torizon Cloud OTA (again)

    1. The device will be stuck in “Update in progress”

Maybe the issue is that I requested to install two times the same software (version B) one after the other? I saw that it’s not allowed to install twice the same software via Torizon Cloud.

Finally, the “solution” we found to get out of this state is to delete the device from Torizon Cloud and provision it again. At the moment when developing in-house we can do this easily, but later when the devices are in the field with the customers this will no longer be an option.

Manually rollback the software to version A

Why did you do this step? Is this a step you will also do when the device is in the field?

The solution would be to never do this kind of step. All changes/updates to the OS should be managed through the update client proper. If you change the OS outside of an update, then the update client doesn’t accurately know what happened.

Basically what happened is the following:

  • You install version B via OTA update.
  • Update succeeds and the update client stores in it’s persistent database that version B is what is installed.
  • You manually rollback to version A.
  • This action did not go through the update client. As far as the update client knows, the device still has version B installed.
  • You do another OTA update to version B.
  • The device gets the update request from the server, but goes “wait I already have version B installed, why should I do anything?”.
  • The device doesn’t do anything because it thinks it doesn’t need to.
  • The server gets stuck with “Update in progress”, because it’s waiting for a result from the device. But the device doesn’t have a result to give because it didn’t do anything.

In summary, if you plan to manage a software component (i.e the OS), with our OTA updates. You should only let OTA updates change the state of that component. Otherwise you may get weird results like this.

Best Regards,
Jeremias

@jeremias.tx Thanks for your explanation.

I rolled back manually, because I bricked the device with a corrupt update OTA (I will open another issue for this).

In the field we wouldn’t do this, so I guess it won’t be an issue. I was wondering why it got stuck and there is no option to cancel, but that is not a normal use case as you explained.

Kind regards,

Ian

Glad we were able to help clarify.

Best Regards,
Jeremias

Hello @jeremias.tx,

I was about to open a new issue when I came across this thread — it looks like I’m experiencing the same problem as Ian.

I believe this issue has been around for a while. The only workaround I found so far was to remove and re-provision the device entirely…

The issue:

After pushing an update from Torizon Cloud, the device remains stuck on Update in progress, even though aktualizr-torizon reports no update available.

Here is the current service status:

$ sudo systemctl status aktualizr-torizon
● aktualizr-torizon.service - Aktualizr SOTA Client
     Loaded: loaded (/etc/systemd/system/aktualizr-torizon.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2026-01-14 09:36:05 CET; 1 month 4 days ago
   Main PID: 1363 (aktualizr-toriz)
      Tasks: 6 (limit: 4079)
     Memory: 22.1M
     CGroup: /system.slice/aktualizr-torizon.service
             └─ 1363 /usr/bin/aktualizr-torizon --enable-data-proxy

Feb 17 21:36:13 verdin-imx8mp-14XXXXXX aktualizr-torizon[1363]: Invalid role in root.json
Feb 17 21:36:14 verdin-imx8mp-14XXXXXX aktualizr-torizon[1363]: Event: UpdateCheckComplete, Result - No updates available
Feb 18 03:36:13 verdin-imx8mp-14XXXXXX aktualizr-torizon[1363]: Current versions in storage and reported by OSTree do not match
Feb 18 03:36:13 verdin-imx8mp-14XXXXXX aktualizr-torizon[1363]: Invalid role in root.json
Feb 18 03:36:13 verdin-imx8mp-14XXXXXX aktualizr-torizon[1363]: Invalid role in root.json
Feb 18 03:36:14 verdin-imx8mp-14XXXXXX aktualizr-torizon[1363]: Event: UpdateCheckComplete, Result - No updates available
Feb 18 09:36:13 verdin-imx8mp-14XXXXXX aktualizr-torizon[1363]: Current versions in storage and reported by OSTree do not match
Feb 18 09:36:13 verdin-imx8mp-14XXXXXX aktualizr-torizon[1363]: Invalid role in root.json
Feb 18 09:36:13 verdin-imx8mp-14XXXXXX aktualizr-torizon[1363]: Invalid role in root.json
Feb 18 09:36:14 verdin-imx8mp-14XXXXXX aktualizr-torizon[1363]: Event: UpdateCheckComplete, Result - No updates available
Feb 18 09:36:13 verdin-imx8mp-14777566 aktualizr-torizon[1363]: Invalid role in root.json
Feb 18 09:36:14 verdin-imx8mp-14777566 aktualizr-torizon[1363]: Event: UpdateCheckComplete, Result - No updates available

I also checked ostree admin status, which confirms the update was received but remains pending — the device never actually switched to the new version:

$ ostree admin status
  torizon 48248beac685ba411d02a22efb544a327da5a6b7bdadb61604d5XXXXXXXXXXXX.0 (pending)
    Version: 6.8.4+build.40-tcbuilder.20251216100717
    origin refspec: 48248beac685ba411d02a22efb544a327da5a6b7bdadb61604d5XXXXXXXXXXXX
* torizon 7da2d07a16cb9f4d1d1db46e21af494342baaf12d5932defbf42XXXXXXXXXXXXX.0
    Version: 6.8.0+build.22-tcbuilder.20250102150417
    origin refspec: 7da2d07a16cb9f4d1d1db46e21af494342baaf12d5932defbf42XXXXXXXXXXXXX

The logs also show a recurring Invalid role in root.json warning and a mismatch between the versions in storage and those reported by OSTree, which may be related.

Any guidance on how to resolve this without having to re-provision the device would be greatly appreciated.

Thank you!

Best,
M

Hi @unablesalt,

I was about to open a new issue when I came across this thread — it looks like I’m experiencing the same problem as Ian.

But, the question is did you do the same cause as Ian? If not, I wouldn’t assume the cause is the same even if the effect is similar, otherwise we may make incorrect assumptions and investigations.

So to clarify, did you do something similar to Ian where the OS/OSTree component of this system was changed/tampered with outside of the control of Torizon Cloud/Aktualizr?

If yes, then the answer would be similar as before. Basically don’t do that. If not, then we must figure out what your cause was.

First of all, according to your ostree admin status there is a new deployment pending. Can you confirm whether this deployment came from Aktualizr or not? Your Aktualizr logs don’t seem to mention anything about a “pending” update, which is what I would expect if this did come from Aktualizr.

If you reboot does the pending OSTree deployment get applied?

Can you recall or reproduce the steps that got you into this situation? Without knowing this I can only guess or theorize how this device might have gotten into this state.

Best Regards,
Jeremias

Hello @jeremias.tx,

To clarify: No, I did not perform any manual operations on the OS or OSTree components like Ian. I haven’t run any ostree admin commands or tampered with the file system outside of the standard Torizon Cloud/Aktualizr workflow.

This state was reached solely through a standard update pushed from the Torizon Cloud.

Regarding the “steps to reproduce,” it is as simple as it gets:

  1. I selected a device in the Torizon Cloud dashboard;

  2. I selected and pushed a standard Operating System update;

  3. The device downloaded the update, but it never successfully transitioned to the new version and remains stuck on “Update in progress”.

Regarding the logs not showing the “pending” update: the log snippet I provided was captured long after the update was originally pushed. I only noticed the device was stuck in “Update in progress” on the dashboard several days later and decided to open (complete) a ticket.

If you reboot does the pending OSTree deployment get applied?

Since the two devices currently in this state are in the field in another country, I’m hesitant to experiment with them remotely, sending someone on-site if they go completely offline is not an option. However, I can provide their UUIDs if you’d like to investigate the logs or status from the Cloud side.

Best,
M

Just to clarify, the 6.8.4+build.40-tcbuilder.20251216100717 that is pending according to ostree admin status. Did this come from the update before the update that got stuck in progress? Or did this come from the update that got stuck in progress?

Are you able to configure the log level of Aktualizr on these devices to be more verbose? As documented here: Torizon Update Client - Aktualizr Commands & Settings | Toradex Developer Center

We’ll need more details than what is shown. Also if you do configure the log level please make sure to share the entire log output of Aktualizr from the moment it starts.

As for current theories. The update is stuck “in progress”. This means the server has sent the update information to the device and the device as “seen” this information. Now the fact that your logs are showing No updates available, means somehow the device didn’t “see” the update, which would be nearly impossible since the server is only “in progress” if the device has seen the update information. Or, the device decided that it didn’t need to perform the update since it already had the package installed, but this would require doing something similar to what Ian did, which you said you did not do. Finally, it could mean Aktualizr did do the update but somehow never told the server it did the update, which would mean the server does not know to move out of the “in progress” state.

The last theory seems the most likely based on what you’ve told me so far. You have an OSTree deployment that is pending, but not applied for some reason. Your Aktualizr logs don’t make mention of a “pending” update which means Aktualizr thinks nothing is going on at the moment. Now the issue is that since Atkualizr thinks no update is happening at the moment, it won’t send a result to the server, which is needed to get the server out of the “in progress” state. Honestly, your device is in a very weird state, and I can’t imagine what led it to being like this.

That’s all I can determine at the moment. To know what actually happened would probably require the Aktualizr logs from the exact moment the update that got stuck “in progress” was occurring. Though I assume those logs are long in the past and not obtainable anymore.

All that said, the only thing we could do at the moment without re-provisioning is forcibly cancel the updates to the affected devices from the server-side. Though there is some risk as these updates are not designed to be forcibly cancelled in such a way. This could have unforeseen strange side-effects in the future on these devices.

However, I can provide their UUIDs if you’d like to investigate the logs or status from the Cloud side.

Could you provide the UUIDs for the affected devices and the email for the Torizon Cloud account they are provisioned to. There might be more information on the server-side, but this does seem to be mostly an issue with whatever happened on the device-side to put it into this state.

Also, if you want us to force-cancel the update we will need the UUIDs anyways.

Best Regards,
Jeremias