Issues with bundling docker containers in torizon OS image

Technical Support
, , ,
1

Hello,

I am attempting to create a “production” image that combines our customised Torizon OS based image with our docker application using guidelines from this page: Pre-provisioning Docker Containers onto a Torizon OS image | Toradex Developer Center.
The goal is to have docker app “baked” into the image, so that no download is required.
Our docker app is in a private repository on docker hub.
Initially I used the bundle option in tcb.yaml file (excerpt below):

output:
  ostree:
    branch: "cabinet-fw-update"
    commit-subject: "FW update package"
    commit-body: ""
  easy-installer:
    local: output/custom-tcbuild-overlays-provisioning
    autoinstall: true
    autoreboot: true
    accept-licence: true
    provisioning:
      mode: "online"
      shared-data: "shared-data.tar.gz"
      online-data: "***"
    bundle:
      compose-file: <sub-dir>/docker-compose.yml
      username: <mylogin>
      password: <docker_pat>

However, when i build the image with
torizoncore-builder build --file $TCBUILD_FILE
and then download the image on the board using Tezi, after the first boot it seems that docker-compose still pulls the docker images from the docker hub registry (e.g. I tested this by booting the board for the first time, and then cutting the network connection, which failed the docker-compose service, as it failed to connect to docker.io registry). But at least correct docker-compose file is on the image…

Next i tried the torizoncore builder workflow (2nd option in the webpage provided), but I fail to even create the bundle:

torizoncore-builder bundle docker-compose.yml --login <mylogin> <pat> --bundle-directory docker-bundle
Creating Docker Container bundle...
NOTE: TCB no longer expands environment variables present in the compose file.

Starting DIND container
Using Docker host "tcp://127.0.0.1:22376"
Connecting to Docker Daemon at "tcp://127.0.0.1:22376"
Attempting to log in to registry 'default' with username=<mylogin>
Fetching container image <repository>/<image-name>:2.2.0 in service <image-name>
Stopping DIND container
Error: container images download failed: 404 Client Error for https://127.0.0.1:22376/v1.40/images/<repository>/<image-name>:2.2.0/json: Not Found ("no such image: <repository>/<image-name>:2.2.0: No such image: <repository>/<image-name>:2.2.0")

It seems that the bundle command for some reason looks at my local registry instead of the docker hub registry?
i tried with --login-to hub.docker.com and some variations of that, all of which fail. I’m not really sure what i’m doing wrong…

Tools:
Board: Verdin AM62
torizoncorebuilder : 3.13
OS: Ubuntu 24.04.2 LTS
Tezi base image: torizon-docker-verdin-am62-Tezi_7.1.0+build.4.tar

2

Hi, I’m the Toradex AI Assistant.

I might make mistakes. Our support team will step in if needed. If I answered your question, please tick the “Solution” checkbox. For follow-up questions to me, please use @ToradexAI in your post.

Did you like my answer? :+1: :-1:

Hello! Thank you for your detailed explanation. You’re very close, but there are a few key points to address regarding the proper bundling of Docker containers from a private Docker Hub repository into your Torizon OS image.

1. Deprecated Parameters in Your tcbuild.yaml

The username and password parameters under the bundle section are deprecated and no longer supported in recent versions of TorizonCore Builder. This means authentication information there is ignored, which is why your images are not pre-bundled as expected and the system tries to pull images on first boot.

More info:

2. Correct Approach for Bundling with Private Repositories

Use the torizoncore-builder bundle command, NOT the bundle section in tcbuild.yaml, for private container registries. The recommended workflow is:

  • First, create a bundle directory containing your private Docker images using the correct bundle command.
  • Then, reference that pre-created bundle directory in your tcbuild.yaml file for your final image build.

Step-by-step Instructions:

Step 1: Ensure Your docker-compose.yml References the Correct Images

  • Make sure your images are referenced as yourdockerhubuser/imagename:tag.

Step 2: Create a Pre-bundled Directory with Authentication

Run the following command to create the bundle directory for your private Docker Hub repo:

torizoncore-builder bundle docker-compose.yml --login <mylogin> <docker_pat> --bundle-directory docker-bundle

If you want to be explicit about Docker Hub as registry, you can use:

torizoncore-builder bundle docker-compose.yml --login-to hub.docker.com <mylogin> <docker_pat> --bundle-directory docker-bundle

Details:

  • Replace <mylogin> and <docker_pat> with your Docker Hub username and your Personal Access Token (PAT) which must have permission to pull from your private repo.

Step 3: Reference the Bundle Directory in Your tcbuild.yaml

After the bundle is created, modify your build config as follows:

output:
  ostree:
    branch: "cabinet-fw-update"
    commit-subject: "FW update package"
    commit-body: ""
  easy-installer:
    local: output/custom-tcbuild-overlays-provisioning
    autoinstall: true
    autoreboot: true
    accept-licence: true
    provisioning:
      mode: "online"
      shared-data: "shared-data.tar.gz"
      online-data: "***"
      bundle:
        dir: docker-bundle/

Simpler sample here:

Step 4: Build Your Final Image

Now run:

torizoncore-builder build --file $TCBUILD_FILE

This should create an image with your Docker images pre-bundled. Upon first boot, Docker Compose uses the bundled images and does not attempt to pull them from the network.

Troubleshooting Tips:

  • Make sure you’re using the correct tag for your container (2.2.0 in your example).
  • Your PAT must have read (pull) access to the repo.
  • The error "404 Client Error ... Not Found" means the image or tag could not be found or accessed. Double-check spelling, tag, and permissions.

If you follow these steps, your offline deployment should succeed. After flashing the image with TEZI, you can test that your application starts fine without an Internet connection.

References:

Let me know if you need further clarification or if you encounter errors with these steps.

Best regards,
Toradex AI Assistant.

3

@ToradexAI
Re Troubleshooting tips:

Container tag is correct
PAT has pull permissions
Spelling is correct.

When the same docker compose file is used on our system with the same user credentials, images are pulled without any issues.

4

Greetings @stasgil,

I just tried this myself with my own private registry hosted on Dockerhub and it works as expected.

and then download the image on the board using Tezi, after the first boot it seems that docker-compose still pulls the docker images from the docker hub registry

Hmm that doesn’t sound right. If the build command truly succeeded without errors then both the compose file and container images should be present in the final image. Also when bundling via the build command the same logic is used as in the bundle command. So it would be odd if one had an error but the other didn’t considering the common code that would be called.

Do you have the logs from the build command execution?

It seems that the bundle command for some reason looks at my local registry instead of the docker hub registry?

What makes you think the bundle command is using a local registry? I can see in your logs the line:

Attempting to log in to registry 'default' with username=<mylogin>

The default registry in this case is Dockerhub. Are you perhaps confused by the references to 127.0.0.1:22376 in your logs? This is not the registry this is the port where your Docker daemon is located for command requests.

As for your issue for some reason your container image can’t be located by that name. If it was a login/access issue the error would say something like:

Not Found ("pull access denied for <registry>/<image-name>, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

Maybe trying enabling verbose logs for TorizonCore Builder and see if it provides more details. Otherwise it’s hard to say for sure why your private container image can’t be fetched.

Best Regards,
Jeremias

5

Hi Jeremias,

Re bundle command using a local registry - yes it was my confusion, i’m still in the process of learning when it comes to docker (and how torizoncorebuilder works)

So the AI response about password / username fields in the tcbuild.yaml being deprecated was false?

Thanks for the verbose flag tip. I have found the issue with my bundle command - by default it looks for platform linux/arm64, while my docker containers were build with linux/arm64/v8 (so i guess GET command was failing because it thought images didnt exist). I have rebuild the containers with linux/arm64 and the bundle command succeeded, and I was able to produce an image with containers using the sequence of torizoncore-builder commands (deploy, union, combine) that booted the containers while being offline.

However, I would prefer to use a tcbuild.yaml file for the build. And this is still not working, even though I have now specified the platform as well. When I say not working, i mean that the build seems to succeed and I can even see the tarball with containers in the image directory. However when i boot the board with this image on it, it still tries to pull containers from dockerhub.
I think the older version of the tcbuild file allowed to specify the bundle directory instead of a docker-compose file. Is this still the case ? (The only version of the tcbuild yaml schema i can find on the web seems to be outdated)
Please find the torizoncore-builder build log(before i deleted the contents of the storage folder, so it did some extra downloads) as well as journalctrl boot log of the built image attached.
torizoncorebuilder_yaml.log (77.0 KB)
jtcl_boot.log (127.6 KB)

6

So the AI response about password / username fields in the tcbuild.yaml being deprecated was false?

Yes the AI was not correct here, or at least I’m not sure what it was referring to.

I think the older version of the tcbuild file allowed to specify the bundle directory instead of a docker-compose file. Is this still the case ?

No the latest version still allows that. What schema are you referencing? The schema found in the TorizonCore Builder Github repository is the most up-to-date: torizoncore-builder/tcbuilder/backend/tcbuild.schema.yaml at bullseye · torizon/torizoncore-builder · GitHub

You can see under the bundle section you can provide the compose file or a path to a bundle directory. We even document using either method here: Pre-provisioning Docker Containers onto a Torizon OS image | Toradex Developer Center

it still tries to pull containers from dockerhub.

Now as for this issue this is strange. Based of the logs you provided I can see TorizonCore Builder did pull the container images as listed in your compose file. It then provided the compose file and the tarball containing your container images in the image directory as you said. Yet despite this the device still tries to pull images from Docker hub.

I just did a quick test and bundled my container image that is stored in a private registry on Dockerhub and it worked as expected:

# My tcbuild.yaml
input:
  easy-installer:
    local: torizon-docker-verdin-imx8mp-Tezi_7.2.0+build.13.tar

output:
  easy-installer:
    local: test
    bundle:
      compose-file: docker-compose.yaml
      username: <username>
      password: <password>

When I flashed the result to my device via Easy Installer I can see my container image is present and it did not try to pull from Dockerhub (I removed the ethernet before booting so it could not have even attempted to pull anything):

torizon@verdin-imx8mp-06849059:~$ journalctl -n 100 -f -u docker-compose
Jan 08 18:23:11 verdin-imx8mp-06849059 systemd[1]: Starting Docker Compose service with docker compose...
Jan 08 18:23:11 verdin-imx8mp-06849059 systemd[1]: Started Docker Compose service with docker compose.
Jan 08 18:23:11 verdin-imx8mp-06849059 docker-compose[902]: time="2025-01-08T18:23:11Z" level=warning msg="/var/sota/storage/docker-compose/docker-compose.yml: `version` is obsolete"
Jan 08 18:23:11 verdin-imx8mp-06849059 docker-compose[902]:  Network torizon_default  Creating
Jan 08 18:23:11 verdin-imx8mp-06849059 docker-compose[902]:  Network torizon_default  Created
Jan 08 18:23:11 verdin-imx8mp-06849059 docker-compose[902]:  Container torizon-app1-1  Creating
Jan 08 18:23:12 verdin-imx8mp-06849059 docker-compose[902]:  Container torizon-app1-1  Created
Jan 08 18:23:12 verdin-imx8mp-06849059 docker-compose[902]:  Container torizon-app1-1  Starting
Jan 08 18:23:12 verdin-imx8mp-06849059 docker-compose[902]:  Container torizon-app1-1  Started

torizon@verdin-imx8mp-06849059:~$ docker images
REPOSITORY                     TAG       IMAGE ID       CREATED         SIZE
<container image name>         arm64     b038788ddb22   20 months ago   9.14kB

On your device what happens if you run docker images are there no container images present at all on your device?

Via your logs I can see a docker-storage.tar.xz is produced. This should get unpacked to /ostree/deploy/torizon/var/lib/docker/ on the device during flashing. Is this not happening during flashing for you?

I also compared our logs to see if there is any discrepancies. In my logs I noticed the following:

Attempting to log in to registry 'default' with username=<username>
Fetching container image <container image> in service app1
  Pulling from <container image>
  70f5ac315c5a: Pull complete
  Digest: sha256:efebf0f7aee69450f99deafe11121afa720abed733943e50581a9dc7540689c8
  Status: Downloaded newer image for <container image>

In your logs I can see the first 2 lines, but I don’t see the last 4. Did you omit this from your logs? These logs are present on the default log level and this was with TorizonCore Builder 3.13.0 as well.

That said, I’m still not entirely sure why the OS image produced via the build command is trying to pull what should be the pre-bundled container images in your case. I can’t seem to reproduce what you’re observing as well.

Would it be possible for you to provide me your OS image that was produced via the build command? It may help if I could more closely analyze what is happening.

Best Regards,
Jeremias

7

Hi Jeremias,

Okay i found the issue, it was my fault all along, one of the changes I made a while back bind mounted a different directory on top of the default /var/lib/docker root directory, and it slipped my mind because as a colleague used to say - SW i wrote half a year ago is by now an off the shelf software :smiley:

Thank you for your testing and time, and apologies for wasting some of it.

8

bind mounted a different directory on top of the default /var/lib/docker root directory

Oh well that makes sense then with regards to what you were seeing. In any case, I’m glad you found this and were able to resolve the issue.

Best Regards,
Jeremias