Problem with Torizon OTA - no component to update

I’ve got problem connected with Torizon OTA – during upgrades via OTA something went wrong – upgrade got stuck, one day later after delete and added device again it passed (got successful message in terminal) , but at OTA webpage I can’t upgrade board - after update initiation I can’t select component to update – there are no option to select, as it was before. The screenshot showing the problem:

Greetings @ml_man,

This is quite an odd situation that I’ve never seen before. Just to understand your steps were more or less the following:

  • Provisioned device to Torizon OTA
  • Deleted device
  • Added again (how did you do this step exactly?)
  • After adding/provisioning device again you got the issue pictured

Is that more or less correct?

My only theory is that perhaps you did something strange in the “added device again” step. If you start from complete scratch (i.e. fresh TorizonCore install) does this issue still happen?

Best Regards,
Jeremias

Hi @jeremias.tx ,

Yes, I made as you mentioned:

  • I have added device to Torizon OTA, as it’s shown in QuickStart Guide (Adding New Device to Torizon OTA),

  • Updated containers couple times (successfully),

  • After one update got stuck for about one day (I tried to reboot device, it was working properly, but in Torizon OTA update was still in progress),

  • I decided to delete device and add it again (in the same way as in QuickStart Guide - in the Apalis shell I got communicate that it’s successfully added, it showed in Torizon OTA with new name, but the problem shown in the picture has happened.

After your reply I got the idea to re-install system on IMX8 and try to add it to Torizon OTA again, because I haven’t done this before.

All right, re-installation of the system helped, after uploading reference TorizonCore image via Easy Installer the device was added to Torizon OTA properly and I could update it using .yaml file via OTA successfully.

Glad to hear starting from scratch was able to get things back into a good state. Though I am still curious how you got your device into that weird state that you showed in your initial post. If you ever run into this again please let us know so that we can look into it since the device shouldn’t get like this in the first place.

Best Regards,
Jeremias

1 Like

Hello,

the problem is going to occur again - after uploading new .yaml file update failed, so I decided to get back to previous, working configuration, and after select it the device is updating from 1 hour, as you can see here :

I can communicate with Apalis through UART, but working only one container (not two) and nothing happens on the touch screen, as previous. Last time in the same scenario I got the error with missing update option described in first post.

Here are my .yaml files, based on yours from quickstart guide. After uploading 16th I got upload failed, and after uploading 15th thich was working previous, the uploading got stuck:

We’ll try to reproduce your observations based on the compose files that you supplied. Though I also have a few questions about your environment:

  • What version of TorizonCore are you running on your device?
  • On this device that is stuck can you get the logs from the update client? You can do this by running journalctl -f -u aktualizr*.
  • I noticed in your compose files you’re Weston container is coming from mlorek/base-vivante:latest. Why are you using a custom Weston container image over the one we provide?

Best Regards,
Jeremias

Hi, here are answers for your question:

  • I’m using 5.4.161-5.6.0+git.0f0011824921 at this moment,
  • There are the logs from aktualizr*:
Oct 10 10:25:00 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:25:01 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:25:01 apalis-imx8-07029113 aktualizr-torizon[1039]: Director metadata update failed: The root metadata was expired.
Oct 10 10:25:01 apalis-imx8-07029113 aktualizr-torizon[1039]: Event: UpdateCheckComplete, Result - Error
Oct 10 10:25:01 apalis-imx8-07029113 aktualizr-torizon[1039]: Current versions in storage and reported by OSTree do not match
Oct 10 10:25:02 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:25:03 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:25:04 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:25:04 apalis-imx8-07029113 aktualizr-torizon[1039]: Put manifest request failed: 60 SSL peer certificate or SSH remote key was not OK HTTP 0
Oct 10 10:25:04 apalis-imx8-07029113 aktualizr-torizon[1039]: Event: PutManifestComplete, Result - Error
Oct 10 10:29:13 apalis-imx8-07029113 aktualizr-torizon[1039]: PROXY: sending device data to Torizon OTA.
Oct 10 10:29:13 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:29:15 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:29:16 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:29:16 apalis-imx8-07029113 aktualizr-torizon[1039]: Event: SendDeviceDataComplete
Oct 10 10:30:04 apalis-imx8-07029113 aktualizr-torizon[1039]: Current versions in storage and reported by OSTree do not match
Oct 10 10:30:04 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:30:06 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:30:07 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:30:07 apalis-imx8-07029113 aktualizr-torizon[1039]: Put manifest request failed: 60 SSL peer certificate or SSH remote key was not OK HTTP 0
Oct 10 10:30:07 apalis-imx8-07029113 aktualizr-torizon[1039]: Error sending manifest!
Oct 10 10:30:07 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:30:08 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:30:10 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:30:10 apalis-imx8-07029113 aktualizr-torizon[1039]: Director metadata update failed: The root metadata was expired.
Oct 10 10:30:10 apalis-imx8-07029113 aktualizr-torizon[1039]: Event: UpdateCheckComplete, Result - Error
Oct 10 10:30:10 apalis-imx8-07029113 aktualizr-torizon[1039]: Current versions in storage and reported by OSTree do not match
Oct 10 10:30:10 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:30:11 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:30:12 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:30:12 apalis-imx8-07029113 aktualizr-torizon[1039]: Put manifest request failed: 60 SSL peer certificate or SSH remote key was not OK HTTP 0
Oct 10 10:30:12 apalis-imx8-07029113 aktualizr-torizon[1039]: Event: PutManifestComplete, Result - Error
Oct 10 10:34:13 apalis-imx8-07029113 aktualizr-torizon[1039]: PROXY: sending device data to Torizon OTA.
Oct 10 10:34:13 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:34:15 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:34:16 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:34:16 apalis-imx8-07029113 aktualizr-torizon[1039]: Event: SendDeviceDataComplete
Oct 10 10:35:12 apalis-imx8-07029113 aktualizr-torizon[1039]: Current versions in storage and reported by OSTree do not match
Oct 10 10:35:13 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:35:14 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:35:15 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:35:15 apalis-imx8-07029113 aktualizr-torizon[1039]: Put manifest request failed: 60 SSL peer certificate or SSH remote key was not OK HTTP 0
Oct 10 10:35:15 apalis-imx8-07029113 aktualizr-torizon[1039]: Error sending manifest!
Oct 10 10:35:15 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:35:17 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:35:18 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:35:18 apalis-imx8-07029113 aktualizr-torizon[1039]: Director metadata update failed: The root metadata was expired.
Oct 10 10:35:18 apalis-imx8-07029113 aktualizr-torizon[1039]: Event: UpdateCheckComplete, Result - Error
Oct 10 10:35:18 apalis-imx8-07029113 aktualizr-torizon[1039]: Current versions in storage and reported by OSTree do not match
Oct 10 10:35:18 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:35:19 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:35:21 apalis-imx8-07029113 aktualizr-torizon[1039]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Oct 10 10:35:21 apalis-imx8-07029113 aktualizr-torizon[1039]: Put manifest request failed: 60 SSL peer certificate or SSH remote key was not OK HTTP 0
Oct 10 10:35:21 apalis-imx8-07029113 aktualizr-torizon[1039]: Event: PutManifestComplete, Result - Error
  • I’m using my version of Weston container to examine if it’s working to pull it from my dockerhub, but at this moment it’s cloned torizon/weston-vivante:2 with no changes.

I noticed that you have added option to abort update - it’s very useful. I have aborted the update and try to pass standard .yaml file from quickstart guide, but at Torizon OTA it’s still updating, on screen nothing happened and via UART I can communicate with IMX8 - so I need to pass reference image via Easy Installer, as previous.

Using the compose files you supplied us we tried to reproduce your observations on our side. First of all both your compose files failed to update. Looking at the logs it seems the healthcheck failed:

May 19 19:10:09 apalis-imx8-06980202 aktualizr-torizon[6751]: Running command: /usr/bin/docker-compose --file /var/sota/storage/docker-compose/docker-compose.yml.tmp -p torizon up --detach --remove-orphans
May 19 19:10:11 apalis-imx8-06980202 aktualizr-torizon[7955]: Creating network "torizon_default" with the default driver
May 19 19:10:11 apalis-imx8-06980202 aktualizr-torizon[7955]: Creating torizon_weston_1 ...
May 19 19:10:12 apalis-imx8-06980202 aktualizr-torizon[7955]: Creating torizon_weston_1 ... done
May 19 19:10:12 apalis-imx8-06980202 aktualizr-torizon[7955]: ERROR: for kiosk  Container "6992510a1c9d" is unhealthy.
May 19 19:10:12 apalis-imx8-06980202 aktualizr-torizon[7955]: Encountered errors while bringing up the project.
May 19 19:10:13 apalis-imx8-06980202 aktualizr-torizon[6751]: Error running docker-compose up
May 19 19:10:13 apalis-imx8-06980202 aktualizr-torizon[6751]: Rolling back container update
May 19 19:10:13 apalis-imx8-06980202 aktualizr-torizon[6751]: Running docker-compose up
May 19 19:10:13 apalis-imx8-06980202 aktualizr-torizon[6751]: Running command: /usr/bin/docker-compose --file /var/sota/storage/docker-compose/docker-compose.yml -p torizon up --detach --remove-orphans

You may need to slightly alter or adapt your compose files to work properly with OTA updates.

Other than that the bigger issue is that we were unable to reproduce your error you showed in the picture where your device got stuck. Looking at the logs you provided I see a bunch of curl errors:

curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK

These errors seem to suggest something is wrong with the security certificates that the devices uses to communicate with our update servers. Did you do anything on this device that might have caused this? Or anything that might have affected the communication between this device and the server?

If you’re not sure try this. Flash this device again with a fresh installation of TorizonCore take note of everything you do up until you see this issue, then give me all the steps of what you did.

Something happened to this device and it’s not obvious what this is yet.

Best Regards,
Jeremias

Hi @jeremias.tx ,

Thank you for your replay and time spent on testing my problem. I didn’t do anything with my network which may caused problems with security certificates. I’ll flash this device again with a fresh installation of TorizonCore and try to document if the problem will occur again. Also I’ll check if there might be some some issues with my network.

Best Regards,
Michał

Please let us know if you find out anymore about the issue. Since we’re unable to reproduce this it’s difficult for us to investigate further without new details.