Torizoncore-builder can't pull containers because of access permissions

Hello, How are you ?

Board: toradex verdin imx8mp plus.
In the tcbuild.yml file config, i see we can pull containers and install them in our output core image by simply passing a docker-compose.yml file and username/password (optionally).

bundle:
      # >> Choose one of the options:
      # >> (1) Specify a docker-compose file whose referenced images will be downloaded.
      # >>     Properties platform, username, password and registry are optional.
      compose-file: files/docker-compose.yml
      # platform: linux/arm/v7
      #username: ""
      #password: "${DOCKER_REGISTRY_PULL_SA}"
      #registry: https://europe-docker.pkg.dev/v2/xxxxxxxxxxx/docker-registry-xxxxxxxxxxxxxxx

In our case, we are pulling our containers from google cloud and we use a token to grant access.
We run this command to authenticate to GCP (google cloud platform).

export GOOGLE_APPLICATION_CREDENTIALS="/tmp/service_account.json"
gcloud auth activate-service-account --key-file="$GOOGLE_APPLICATION_CREDENTIALS" -q

The service_account.json looks like this:

{
  "type": "service_account",
  "project_id": "our_product",
  "private_key_id": "c4e052d2990d092ba2bxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0Bxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxhXxDBui7ILodA0E=\n-----END PRIVATE KEY-----\n",
  "client_email": "registry-pull-xxxxxxxxxxxxxxxxx@xxxxxxxxxxxxxxxxx.iam.gserviceaccount.com",
  "client_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/registry-pull-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.iam.gserviceaccount.com"
}

After that, we can pull our containers without problems.

The question is: How can i do that with the Torizoncore-builder ? is there a way to pass the token in the tcbuild.yml ?
Do you have any good idea ?

Thanks a lot !

Hello @Samir1,

Thanks for reaching out.
Would it be possible to send your docker-compose.yml file?

Hello,
Here is my docker-compose.yml file :

version: "3"

services:
  config:
    image: europe-docker.pkg.dev/xxxxxx/docker-registry-xxxxxxxxx/xxxxxx-config:1.2.2-arm64
    restart: always
    network_mode: "host"
    privileged: true
    environment:
      PRODUCT: xxxxxx
      VERSION: 1.2.2
      ENV: prod
      SSH_HOST: localhost
      SSH_USER: torizon
      SSH_KEY_B64: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
      DOCKER_REGISTRY_PULL_SA: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      TOKEN_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      DATA_DIR: /app/data
      WIFI_DEV: wlan0
      WIFI_DIRECT_NETWORK: 192.168.42.1/24
      DHCP_LEASE: 20,50
      CRYPTO_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    volumes:
      - /etc/hostname:/local/hostname
      - api_logs:/app/data/api_logs

volumes:
  api_logs:
  api_data:

thanks !

I don’t think TorizonCore Builder would be able to easily access this container registry with this authentication method. Currently Torizoncore Builder can authenticate container registries via the following methods:

  • Username/Password login
  • Authentication via provided CA certificate

In this case it looks like you need to authenticate using a specific google cloud tool along with a json file that looks specific to google cloud as well. Currently TorizonCore Builder doesn’t handle anything google cloud specific, so I would think this is currently not possible. Unless there’s a more generic way to authenticate access to google cloud registry, though I’m not familiar enough with google cloud to know.

A possible workaround would be to first pull the container image from your google cloud registry to your development machine. Then you could create a local registry on your machine and then have TorizonCore Builder target that instead. Basically something like what is described here: Try to make a customized TorizonCore with Pre-provisioned Docker-Images - #12 by jeremias.tx

This should work in theory, though it is a bit clunky. But until proper google cloud authentication support could be added to TorizonCore Builder, this might be your only practical option.

Best Regards,
Jeremias

2 Likes

Hello @jeremias.tx,
This Try to make a customized TorizonCore with Pre-provisioned Docker-Images - #12 by jeremias.tx worked fine when i created a local registry.

Unfortunately, that’s not what we want.
I wonder if there is a way to run post scripts after flashing the base image on the Toradex?

Thanks !

I wonder if there is a way to run post scripts after flashing the base image on the Toradex?

There is support in Toradex Easy Installer to run a script after an image has been flashed. Read about wrapup.sh here: Toradex Easy Installer Configuration Files | Toradex Developer Center

Otherwise you could also create a systemd service that executes a script on first boot or something similar.

What exactly is your plan here with such a script?

Best Regards,
Jeremias

Hello,
The idea is to copy docker-compose.yml and install.sh files to /home/torizon and run this commands from wrapup.sh :

bash ~/home/torizon/install.sh

install.sh looks like :

#!/bin/bash

package_architecture=$(arch)
if [[ "$package_architecture" == "aarch64" ]]
then
    package=https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-419.0.0-linux-arm.tar.gz
else
    package=https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-419.0.0-linux-x86_64.tar.gz
fi
wget $package -O gcloud.tar.gz
tar xf gcloud.tar.gz
./google-cloud-sdk/install.sh -q

source google-cloud-sdk/path.bash.inc # this cmd to use gcloud command line
# authenticate to gcp
gcloud auth activate-service-account --key-file="/tmp/service_account.json" -q
# pull and install containers
docker-compose up -d

To test wrapup.sh, i added this lines :

echo "test" >> /ostree/deploy/torizon/var/rootdirs/home/torizon/test.txt
echo "test" >> /home/torizon/test1.txt

but it does not work !

Can you tell me why it doesn’t work? and give me an example of use please?

I see now that I understand what you are trying to do here, maybe the wrapup.sh method is not ideal.

First of all what you need to realize is that wrapup.sh is being executed in Easy Installer not in TorizonCore. Easy Installer is in RAM not in the eMMC like TorizonCore would be. So you can’t just write to the filesystem of TorizonCore since Easy Installer is separate in RAM. In theory you could try to mount the eMMC device to have access to the newly flashed TorizonCore filesystem. But even then I don’t know if you’d have the ability to modify it freely.

Secondly, the install.sh script you have isn’t going to work in Easy Installer. Your script replies on various binaries and utilities like docker-compose which aren’t in Easy Installer to begin with. And again this would be executing from the Easy Installer filesystem not the TorizonCore one.

Due to these complications I would recommend you create a systemd service on TorizonCore that executes your script upon first boot. That way everything gets installed and executed from the perspective of TorizonCore which is where you want these things to end up.

Best Regards,
Jeremias

1 Like

Hello @jeremias.tx , how are you ?
I came to the same observation as you, can you tell me how to create this systemd service please ?

Is there a way to customize torizoncore images to add systemd services ?

I take any type of solution even if it requires modifying the torizon-core-docker-verdin-imx8mp.ota.tar.zst to add or edit services manually.

thanks !

Here’s general documentation on how to create a systemd service: Basic Linux Commands | Toradex Developer Center

After that here’s how to capture the changes and produce a TorizonCore image that has your changes: Capturing Changes in the Configuration of a Board on TorizonCore | Toradex Developer Center

Please note that everything should be in /etc in the filesystem for it to be captured properly. This includes the script that your systemd service will call.

Best Regards,
Jeremias

Hello @jeremias.tx,

First time, I captured the changes in /etc successfully, but after that, i did some changes in the machine, i added user torizon to sudoers :

echo "torizon  ALL=(ALL) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/torizon

and executed this command again:

torizoncore-builder isolate --remote-host 192.168.10.34 --remote-username torizon --remote-password pass --changes-directory changes --force

I get this error :

Traceback (most recent call last):
  File "/builder/torizoncore-builder", line 221, in <module>
    mainargs.func(mainargs)
  File "/builder/tcbuilder/cli/isolate.py", line 55, in isolate_subcommand
    ret = isolate.isolate_user_changes(changes_dir,
  File "/builder/tcbuilder/backend/isolate.py", line 163, in isolate_user_changes
    indx = output.index("Password: ")
ValueError: 'Password: ' is not in list

Do you know why i get this error ? how to solve this ?

Hello,
I was able to fix the problem by running the command:

sudo usermod -aG sudo torizon

Now I wonder why when I make modifications again and I execute the isolate command then the union command, I do not see my modifications in the .tar.zst ?

First time : isolate + build + union … working fine
Second time: (after doing changes in /etc/) isolate + union … is not working, isolate + build + union is applying only the last isolated changes in /etc.

What did I miss or misunderstand please?

Thanks !

Second time: (after doing changes in /etc/) isolate + union … is not working, isolate + build + union is applying only the last isolated changes in /etc.

The issue here is an understanding of how the isolate command works. So before you started you ran the unpack command on a base image, correct? When you do this the TorizonCore Builder tool has a fresh image as a reference. Then when you run the isolate command it compares the filesystem on the target device to the reference from unpack. It finds the differences and captures them.

However, once this is done the reference is no longer of the base image it’s of the base image plus your initial changes from the first isolate. So now if you were to run isolate the second time it will only capture the differences between the first and second isolate. Not the differences between the base and the second isolate.

Therefore, what you need to do is run unpack again on the base image. This resets TorizonCore Builder’s reference to a fresh filesystem. That way when you run isolate it will capture the differences between the fresh filesystem and whatever you have on your target device.

Or better yet use the method with the build command instead, since the build command always runs unpack before doing anything to ensure a clean starting reference.

Best Regards,
Jeremias

Hello @jeremias.tx ,

Thank you so much !

Now I understand how isolate works.

One last note before closing this post, do you have any idea why I can’t capture the changes in /etc/sudoers.d/ when i add torizon to sudoers with this command?

echo “torizon ALL=(ALL) NOPASSWD:ALL” | sudo tee /etc/sudoers.d/torizon

Thanks !

One last note before closing this post, do you have any idea why I can’t capture the changes in /etc/sudoers.d/ when i add torizon to sudoers with this command?

The issue here is that you’ve configured sudoers such that the torizon user doesn’t require a password input for sudo. When TorizonCore Builder runs for some commands like isolate it needs to execute commands remotely on the target device via ssh. Some of these commands are executed with sudo. When TorizonCore Builder processes the output from the target device it filters out the Password: prompt that comes when you execute commands with sudo. But since you’ve disabled this there is no Password: string to filter, which completely messes up the output handling and parsing.

Basically don’t do this change, also it’s ill-advised to remove the password input for sudo commands.

Best Regards,
Jeremias

Hello @jeremias.tx , @rudhi.tx,

I solved the problem, i’m using systemd to run services at boot.

Thank you so much !

Glad we were able to help!