Offline updates - best way to apply pending changes without aktualizr service running

ollamh · July 12, 2024, 5:12pm

Hello,

I’m working on the offline updates with Verdin imx8mp board and customized Torizon OS based on 6.4 image.

I stopped aktualizr-torizon service as I want to control when to run updates. I provisioned the OS via tcbuild.yaml file and provided /etc/sota/conf.d/99-offline-updates.toml file with this contents:

[uptane]
enable_offline_updates = true
enable_online_updates = false
offline_updates_source = "/tmp/update"  # update directory contains images and metadata folders
force_install_completion = true

What is happening when I run updates:

I verify the updates via small python code on top of the aktualizr-torizon call to ensure the update is available with provided files and set the lock so Aktualizr doesn’t proceed with the updates.
I run aktualizr-torizon again without the lock so it proceeds with primary and secondary updates. From the logs I see it goes well until the moment where I see this string

2024-07-12 13:43:22,996 - subprocess_logger - INFO - STDOUT: Event: InstallStarted
2024-07-12 13:43:23,083 - subprocess_logger - INFO - STDOUT: Installing package using ostree package manager
2024-07-12 13:43:27,716 - subprocess_logger - INFO - STDOUT: Commit metadata kargs=quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3
2024-07-12 13:43:35,229 - subprocess_logger - INFO - STDOUT: Freed objects: 642.8?kB
2024-07-12 13:43:35,268 - subprocess_logger - INFO - STDOUT: Performing sync()
2024-07-12 13:43:35,309 - subprocess_logger - INFO - STDOUT: Event: InstallTargetComplete, Result - Success
2024-07-12 13:43:35,309 - subprocess_logger - INFO - STDOUT: Updating containers via docker-compose
2024-07-12 13:43:35,320 - subprocess_logger - INFO - STDOUT: OSTree update pending. This is a synchronous update transaction.
2024-07-12 13:43:38,987 - subprocess_logger - INFO - STDOUT: Event: InstallTargetComplete, Result - Error
2024-07-12 13:43:39,023 - subprocess_logger - INFO - STDOUT: Event: AllInstallsComplete, Result - INTERNAL_ERROR
2024-07-12 13:43:39,024 - subprocess_logger - INFO - STDOUT: Update install completed. Releasing the update lock...
2024-07-12 13:43:39,026 - subprocess_logger - INFO - STDOUT: Exiting aktualizr so that pending updates can be applied after reboot

I expected the system reboot to happen because we have OS changes pending and it seems that the Aktualizr should trigger it as I defined the force reboot flag according to the docs aktualizr/docs/ota-client-guide/modules/ROOT/pages/aktualizr-config-options.adoc at master · advancedtelematic/aktualizr · GitHub and which was mentioned in some of the posts here.

If I do the reboot, it goes well, but no docker container is up and running until I run the aktualizr update process again, which completes the update process. At this point I see docker images listing having current and previous versions of docker images.

To me it seems that if I don’t run any specific command on startup, it won’t finish pending update and start docker containers by itself, because the aktualizr-torizon service is intentionally disabled, do I understand the situation correctly?

Also, will the touch /run/need-reboot command work if I run it manually so that ostree-pending-reboot.path service will trigger the reboot?

And one more question is - may I consider that the update process was completed successfully even if I see these lines in the log?

Event: InstallTargetComplete, Result - Error
Event: AllInstallsComplete, Result - INTERNAL_ERROR

Thank you!

ToradexAI · July 12, 2024, 5:13pm

Hi, I’m the Toradex AI Assistant. I’m in early beta, and I would appreciate your feedback.
The Toradex Support team will follow up with you shortly!

Did you like my answer?

Hello,

For your scenario with the Verdin iMX8MP board and a customized TorizonCore 6.4, managing offline updates requires careful handling, especially when the aktualizr-torizon service is stopped to manually control update timings.

Running Updates Without the aktualizr-torizon Service: When you stop the aktualizr-torizon service to manually trigger updates, you assume responsibility for the entire update process, including error handling and rebooting. The force_install_completion setting in your 99-offline-updates.toml configuration file is not a standard option for the aktualizr configuration, which might be why you’re not seeing the expected automatic reboot behavior. Aktualizr, by default, requires a reboot to apply an OSTree-based system update fully. However, this reboot must be triggered externally if aktualizr-torizon is not managing the process automatically.
Applying Pending Changes: If the update process has completed with an OSTree update pending, a reboot is indeed necessary to apply these changes. The absence of running Docker containers after the reboot suggests that the Docker service might not be set to start automatically, or the update process requires an additional step to re-enable or restart the Docker service.
Manual Reboot Triggering: The touch /run/need-reboot command is a mechanism used by some systems to flag that a reboot is required. However, for it to trigger an automatic reboot, there must be a service or script actively monitoring this file and then initiating the reboot. The ostree-pending-reboot.path service you mentioned is not a standard part of the TorizonCore or Aktualizr documentation. Typically, the reboot process should be managed by your update scripts or manually if you’re controlling updates outside of the automated aktualizr-torizon workflow.
Interpreting Log Messages: Seeing InstallTargetComplete, Result - Error and AllInstallsComplete, Result - INTERNAL_ERROR in your logs indicates that the update process encountered an error. Without the aktualizr-torizon service’s automatic handling, you’ll need to manually investigate these errors. Check the detailed aktualizr logs for more information on the error. It’s crucial to resolve these errors before considering the update process successful.
Best Practices: When managing updates manually, especially with offline updates, ensure that you have a robust error handling and recovery process. This includes checking for update success, handling errors, triggering necessary reboots, and verifying that all services,

Please, also check if these links can help you solve your problem:

How to Use Secure Offline Updates with Torizon OS | Toradex Developer Center.

jeremias.tx · July 12, 2024, 11:05pm

Greetings @ollamh,

Just to summarize, you’re doing an offline synchronous update (OS + container update). Based on your previous threads I assume you’re executing aktualizr-torizon once, correct?

You then tried an update but it didn’t reboot as execpted, after manually rebooting you had to run aktualizr-torizon on the next boot to finish the process.

Did I more or less summarize the situation correctly?

Assuming I understood you correctly, I did a quick test and was able to reproduce your findings. After some investigation here’s what I found:

System not rebooting automatically
- Seems like when the once run mode is used the aktualizr-torizon will check and install any available offline updates, but then it does not call the function in the code that will actually trigger the reboot. Most likely a bug.
Needing to run aktualizr-torizon after reboot to finish the update.
- This is expected. When there is an update that requires a reboot of the system, aktualizr-torizon needs to run after the reboot to finish the update and do some final checks on the system. This includes recording the success/failure of the update and the new installed version. This is how aktualizr-torizon maintains consistency with it’s internal database. Failing to do this could put the system in a weird state causing issues later on.

I will report this all internally for further discussion with our R&D team.

In the meantime here’s what I would suggest. After you call aktualizr-torizon once you’ll need to reboot the system yourself. You can do so either just calling the reboot command, or touch /run/need-reboot will work as well I checked this myself.

After the reboot, whether the update succeeded or not, you need to run aktualizr-torizon again so the update can officially “finish” and aktualizr-torizon will be put into a good state again. This is very important.

Does that all make sense?

And one more question is - may I consider that the update process was completed successfully even if I see these lines in the log?

Yeah these are just some badly worded log messages we need to improve on, these can be ignored.

One final note, you don’t need to have force_install_completion = true in your toml, we actually already have this set in our default configs:

sudo cat /lib/sota/conf.d/70-reboot.toml
[uptane]
force_install_completion = true
[bootloader]
reboot_command = "/usr/bin/touch /run/need-reboot"

Best Regards,
Jeremias

jeremias.tx · August 2, 2024, 9:58pm

Hi @ollamh,

Did my response help clear things up, or do you still require further support on this topic?

Best Regards,
Jeremias

ollamh · August 5, 2024, 7:56am

Hello @jeremias.tx ,

Sorry for a response delay, I was AFK for some time.
Yes, I got it, I’ve created a systemd service which is enabled by the certain file presence on the disk, which I touch before rebooting the system via touch /run/need-reboot. All seems to be working fine with this approach, I’d say that it is even better imho, because we have more freedom in between first update stage completed and the system reboot.

Thank you for your invaluable help!

jeremias.tx · August 5, 2024, 6:17pm

Glad I was able to assist.