Tcb-builder build - provisioning for online updates

Technical Support
,
1

Hello,

I’m building an image with changes to the base torizon os image:

input:
  easy-installer:
    toradex-feed:
      version: 6.5.0
      release: quarterly
      machine: verdin-imx8mp
      distro: torizon
      variant: torizon-core-docker
      build-number: 8

The customisation step can be ignored, I’m currently only adding a test.txt during debug.

By following the documentation as far as:

Installing-provisioning-data

I have my outputs defined as:

output:
  easy-installer:
    local: my-base-<VERSION>-ap-secure
    name: "my-base-<VERSION>-ap-secure"
    description: "my OS base image"
  provisioning:
    mode: "online"
    shared-data: shared-data.tar.gz
    online-data: <ONLINE_DATA_STRING>

We’re using some of the scripts like tcb-env-setup.sh provided by the IDE extension, so can not rule out the issue is somewhere in there - when the Aktualizr starts on the device, the following files are missing:
/var/sota/import/pkey.pem
/var/sota/import/director/root.json

○ aktualizr-torizon.service - Aktualizr SOTA Client
     Loaded: loaded (/usr/lib/systemd/system/aktualizr-torizon.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Tue 2024-05-28 06:45:09 UTC; 39min ago
             ├─ ConditionPathExists=|/var/sota/import/pkey.pem was not met
             └─ ConditionPathExists=|/var/sota/import/director/root.json was not met

May 28 06:45:09 verdin-imx8mp-xxxxxxxx systemd[1]: Aktualizr SOTA Client was skipped because all trigger condition checks failed.

There are no other logs coming from build to help. The /var/sota/ directory does not get created by the build tool. The director/root.json is in the shared-data.tar.gz, but the pkey.pem appears that is needs to be generated by the build command:
(https)://github.com/search?q=repo%3Atoradex%2Faktualizr+pkey.pem&type=code

It appears the director/root.json is also created by the builder tool, and the /var/sota/ directory:
(https)://github.com/toradex/aktualizr/blob/1523e48df0c0ef68c8c05c519ef4c834ece7b46c/src/libaktualizr/storage/invstorage.h#L188

This section of the documentation says “Make sure that you performed the proper configurationof the device for using online or offline updates…” but the link to “proper configuration” links to configuring for offline-data only - at the top of the documentation saying " If you just want to use the Torizon Remote Updates with default settings, the pre-installed software on a Torizon OS image abstracts the settings of Aktualizr for you, therefore you can skip this article."

Does anyone know how I can generate these pkey.pem, and director/root.json? Or if there is some other steps I have missed? Any help would be great :slight_smile:

Thank you,
Diarmuid

2

Hi, I’m the Toradex AI Assistant. I’m in early beta, and I would appreciate your feedback.
The Toradex Support team will follow up with you shortly!

Did you like my answer?

:+1: :-1:

Hello,

For your scenario, where you’re aiming to provision your devices for online updates using TorizonCore Builder, it’s crucial to ensure that all necessary provisioning data, including pkey.pem and director/root.json, are correctly integrated into your custom Torizon OS image.

The pkey.pem and director/root.json files are essential for the secure operation of Aktualizr, the client responsible for handling updates in Torizon. These files should be part of the provisioning data you prepare for your devices. Here’s how to address the issues you’re encountering:

  1. Provisioning Data Preparation: Ensure that the provisioning data (shared-data.tar.gz and the online data string) you’re using includes the necessary pkey.pem and director/root.json. These files are typically prepared and provided as part of the online provisioning data. If these files are missing, you’ll need to revisit the provisioning data generation step to ensure they’re correctly included.

  2. Applying Provisioning Data: When using the TorizonCore Builder to apply provisioning data, ensure that the provisioning section of your configuration correctly references the provisioning data files. It sounds like you’ve done this correctly, but it’s worth double-checking the paths and file names to ensure they match what’s expected.

  3. Manual Verification: After building your image but before deploying it, you can manually inspect the output image to verify that the /var/sota/import/ directory and its expected contents (pkey.pem and director/root.json) are present. This can be done by mounting the image on your development machine and navigating to the appropriate directory.

  4. Debugging the Build Process: If the files are still missing after verifying the steps above, there may be an issue with the build process itself or with the scripts you’re using, such as tcb-env-setup.sh. To debug this, you can add verbose logging to the build process to track each step and identify where the provisioning data might be getting lost or not applied correctly.

  5. Reviewing Documentation: While the documentation primarily references offline updates, the process for online updates shares similarities, especially regarding the provisioning data’s application. Ensure that any steps specific to online updates, such as generating or including pkey.pem and director/root.json, are not overlooked.

If after following these steps, you’re still encountering issues with missing pkey.pem and director/root.json, it may indicate a more specific problem with the provisioning

Please, also check if these links can help you solve your problem:

TorizonCore Builder Tool “build” command | Toradex Developer Center.

3

Greetings @dk-coligo,

Just to understand, you tried to configure auto-provisioning for your image and it did not work from what I see.

Couple of questions, when you create your custom image with TorizonCore Builder, how are you deploying it to your device? With Easy Installer?

Could you provide the logs from the auto-provisioning service as detailed here: Production Programming in Torizon | Toradex Developer Center

Finally does the the file /var/sota/auto-provisioning.json exist on your custom image? You don’t need to share the contents of this.

Best Regards,
Jeremias

4

Hi @jeremias.tx,

yes, auto provisioning with online updates only for now.

Deploying with Easy installer via USB storage plugged into the device. Image flashes, and greenboot completes without error:

● greenboot-healthcheck.service - greenboot Health Checks Runner
     Loaded: loaded (/usr/lib/systemd/system/greenboot-healthcheck.service; enabled; vendor preset: enabled)
     Active: active (exited) since Wed 2024-05-29 05:46:43 UTC; 4min 14s ago
    Process: 683 ExecStart=/usr/libexec/greenboot/greenboot check (code=exited, status=0/SUCCESS)
   Main PID: 683 (code=exited, status=0/SUCCESS)

the output of journalctl -f -u auto-provisioning is:

root@verdin-imx8mp-xxxxxxxx:/var/rootdirs/home/torizon# journalctl -f -u auto-provisioning
May 29 05:46:50 verdin-imx8mp-xxxxxxxx systemd[1]: Automatically provision the device to the Platform Services was skipped because of a failed condition check (ConditionPathExists=/var/sota/auto-provisioning.json).

And the whole directory /var/sota/ does not exist.

root@verdin-imx8mp-xxxxxxxx:/var/rootdirs/home/torizon# ls /var/  
cache  lib  lock  log  rootdirs  roothome  run	spool  tmp  usrlocal  volatile

Running the connection string from Provision Device on torizon.io dashboard does connect the device, so it’s not a network issue:

root@verdin-imx8mp-xxxxxxxx:/var/rootdirs/home/torizon# telnet app.torizon.io 443
Connected to app.torizon.io

When I run the /bin/aktualizr-torizon directly, it will create the /var/sota dir with storage and sql, but still complains:

root@verdin-imx8mp-xxxxxxxx:/var/rootdirs/home/torizon# /bin/aktualizr-torizon 
Aktualizr version tdx-d71f18a1 starting
Reading config: "/usr/lib/sota/conf.d/20-sota-device-cred.toml"
Reading config: "/usr/lib/sota/conf.d/30-rollback.toml"
Reading config: "/usr/lib/sota/conf.d/40-hardware-id.toml"
Reading config: "/usr/lib/sota/conf.d/50-secondaries.toml"
Reading config: "/usr/lib/sota/conf.d/60-polling-interval.toml"
Reading config: "/usr/lib/sota/conf.d/70-reboot.toml"
Bootstrap empty SQL storage
created: /var/sota
Bootstraping DB to version 26
Couldn`t import data: empty path received
Couldn't import client certificate: "/var/sota/import/client.pem" doesn't exist.
Couldn't import client TLS key: "/var/sota/import/pkey.pem" doesn't exist.
...

If I extract the boot image on my local machine, in the torizon-core-docker-verdin-imx8mp.ota/ostree/deploy/torizon/ the /var/ in here does not contain /sota/ and there are another /var/ and /var-local/ which are empty in the /deploy/ dir.

The base image at build time came from here:

torizon.tcbuilder.backend.build - DEBUG - Feed URL: https://artifacts.toradex.com/artifactory/torizoncore-oe-prod-frankfurt/kirkstone-6.x.y/release/8/verdin-imx8mp/torizon/torizon-core-docker/oedeploy/torizon-core-docker-verdin-imx8mp-Tezi_6.5.0+build.8.tar
5

The fact you don’t even have a /var/sota directory is a bit concerning. If you flash the input base image you used you can see even this base image has a /var/sota by default:

torizon@verdin-imx8mp-06849059:~$ ls /var/
cache  lib  local  lock  log  rootdirs  roothome  run  sota  spool  tmp  usrlocal  volatile

I then tried to reproduce this by using a tcbuild.yml similar to what you shared at the start:

input:
  easy-installer:
    toradex-feed:
      version: 6.5.0
      release: quarterly
      machine: verdin-imx8mp
      distro: torizon
      variant: torizon-core-docker
      build-number: 8

output:
  easy-installer:
    local: foo
    name: "bar"
    description: "foobar"
    provisioning:
      mode: "online"
      shared-data: "shared-data.tar.gz"
      online-data: "<SECRET>"

For me the resulting image was fine and it was able to auto-provision itself without issue. Are you sure the image from TorizonCore Builder built without issue? All the observations you are sharing with me makes it look like your custom image does not have any provisioning data at all. Does the issue occur again if you do the entire process from the beginning,?

In the output folder created by TorizonCore Builder is there a file in there named provisioning-data.tar.gz?

Best Regards,
Jeremias

6

UPDATE

Oh wait I think I see the issue!

Notice the difference in our output sections:

output:
  easy-installer:
    local: my-base-<VERSION>-ap-secure
    name: "my-base-<VERSION>-ap-secure"
    description: "my OS base image"
  provisioning:
    mode: "online"
    shared-data: shared-data.tar.gz
    online-data: <ONLINE_DATA_STRING>

versus:

output:
  easy-installer:
    local: foo
    name: "bar"
    description: "foobar"
    provisioning:
      mode: "online"
      shared-data: "shared-data.tar.gz"
      online-data: "<SECRET>"

In my case I had the provisioning section indented an extra level. This is the intended format. I modified my file to be like yours where the extra indent level is not there. Surprisingly the build command still completes without error, but it produces an output file without the provisioning data. This seems to be a bug as the tool should throw an error if the format is not correct.

Please change your tcbuild.yml file to reflect mine with the extra indent on provisioning: and see if that helps.

Best Regards,
Jeremias

1 Like
8

Amazing, yes this worked! Thank you

9

Perfect, glad I was able to assist. Thank you for reporting this bug so I can bring it to the attention of our developers.

Best Regards,
Jeremias