Python NullResource error when running torizoncore-builder build

Technical Support
, ,
1

I’m running into a problem using torizoncore-builder. I have a Docker container and device tree overlay that I’m trying to add to TorizonCore 5.3.0 build 7, but torizoncore-builder build is failing:

user@debian:~/build-0.9$ . ./tcb-env-setup.sh -t 3.1.0
Setting up TorizonCore Builder with version 3.1.0.

Pulling TorizonCore Builder...
3.1.0: Pulling from torizon/torizoncore-builder
Digest: sha256:e39ab04132c817a148013f1d735eb264831225a0d40efd465b186e640fd7ae7d
Status: Image is up to date for torizon/torizoncore-builder:3.1.0
docker.io/torizon/torizoncore-builder:3.1.0
Done!

Setup complete! TorizonCore Builder is now ready to use.
********************
Important: When you run TorizonCore Builder, the tool can only access the files inside the current working directory. Files and directories outside of the current working directory, or links to files and directories outside of the current working directory, won't be visible to TorizonCore Builder. So please make sure that, when running TorizonCore Builder, all files and directories passed as parameters are within the current working directory.
Your current working directory is: /home/user/build-0.9
********************
For more information, run 'torizoncore-builder -h' or go to https://developer.toradex.com/knowledge-base/torizoncore-builder-tool
user@debian:~/build-0.9$ docker images
REPOSITORY                    TAG             IMAGE ID       CREATED         SIZE
localhost:5000/vcu            0.9             9d2876580dd6   24 hours ago    124MB
torizon/torizoncore-builder   3.1.0           1a5402c8275b   2 months ago    878MB
torizon/debian                2-bullseye      c68a0b90ef78   2 months ago    62.4MB
debian                        bullseye-slim   b95c659b3dcc   3 months ago    80.3MB
registry                      2               1fd8e1b0bb7e   5 months ago    26.2MB
torizon/binfmt                20200807        893244d86992   13 months ago   37.5MB
docker                        19.03.8-dind    c814ba3a41a3   17 months ago   236MB
user@debian:~/build-0.9$ docker ps
CONTAINER ID   IMAGE        COMMAND                  CREATED        STATUS          PORTS                    NAMES
b12ac3a8d6cd   registry:2   "/entrypoint.sh /etc…"   2 months ago   Up 29 minutes   0.0.0.0:5000->5000/tcp   registry
user@debian:~/build-0.9$ uname -a
Linux debian 5.10.0-8-amd64 #1 SMP Debian 5.10.46-5 (2021-09-23) x86_64 GNU/Linux
user@debian:~/build-0.9$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye
user@debian:~/build-0.9$ torizoncore-builder build --force --file tcbuild-r0.yml
Building image as per configuration file 'tcbuild-r0.yml'...

=>> Handling input section
Fetching URL 'https://artifacts.toradex.com/artifactory/torizoncore-oe-prod-frankfurt/dunfell-5.x.y/release/7/colibri-imx6/torizon-upstream/torizon-core-docker/oedeploy/torizon-core-docker-colibri-imx6-Tezi_5.3.0+build.7.tar' into '/tmp/torizon-core-docker-colibri-imx6-Tezi_5.3.0+build.7.tar'
[========================================]
Download Complete!
Downloaded file name: '/tmp/torizon-core-docker-colibri-imx6-Tezi_5.3.0+build.7.tar'
No integrity check performed because checksum was not specified.
Unpacking Toradex Easy Installer image.
Copying Toradex Easy Installer image.
Unpacking TorizonCore Toradex Easy Installer image.
Importing OSTree revision 36ad904617b170339b6ded7b9dce87ed8cf0f76473b897fdd832d91e82eb1ddc from local repository...
1090 metadata, 12725 content objects imported; 412.4 MB content written
Unpacked OSTree from Toradex Easy Installer image:
  Commit checksum: 36ad904617b170339b6ded7b9dce87ed8cf0f76473b897fdd832d91e82eb1ddc
  TorizonCore Version: 5.3.0+build.7

=>> Handling customization section

=> Handling device-tree subsection

=> Selecting custom device-tree 'device-trees/dts-arm32/imx6dl-colibri-eval-v3.dts'
'imx6dl-colibri-eval-v3.dts' compiles successfully.
warning: removing currently applied device tree overlays
Device tree imx6dl-colibri-eval-v3.dtb successfully applied.

=> Adding device-tree overlay 'device-trees/overlays/vcu-r0-overlay.dts'
'vcu-r0-overlay.dts' compiles successfully.
/tmp/tmpq2kk3pz4: Device Tree Blob version 17, size=62518, boot CPU=0, string block size=4946, DT structure block size=57516
'vcu-r0-overlay.dtbo' can successfully modify the device tree 'imx6dl-colibri-eval-v3.dtb'.
Overlay vcu-r0-overlay.dtbo successfully applied.

=>> Handling output section
Applying changes from STORAGE/dt.
Commit 1adc1180b8530f32d805baad7bf9c20be4e8509457ede42df0075c19253a1b64 has been generated for changes and is ready to be deployed.
Deploying commit ref: tcbuilder-20210928212611
Pulling OSTree with ref tcbuilder-20210928212611 from local archive repository...
  Commit checksum: 1adc1180b8530f32d805baad7bf9c20be4e8509457ede42df0075c19253a1b64
  TorizonCore Version: 5.3.0+build.7-tcbuilder.20210928212611
  Default kernel arguments: quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash

1090 metadata, 12726 content objects imported; 412.5 MB content written
Pulling done.
Deploying OSTree with checksum 1adc1180b8530f32d805baad7bf9c20be4e8509457ede42df0075c19253a1b64
Deploying done.
Copy files not under OSTree control from original deployment.
Packing rootfs...
Packing rootfs done.
Updating TorizonCore image in place.
Bundling images to directory bundle_20210928212618_796717.tmp
Removing output directory 'vcu-r0-0.9' due to build errors
An unexpected Exception occured. Please provide the following stack trace to
the Toradex TorizonCore support team:

Traceback (most recent call last):
  File "/builder/torizoncore-builder", line 204, in <module>
    mainargs.func(mainargs)
  File "/builder/tcbuilder/cli/build.py", line 422, in do_build
    build(args.config_fname, args.storage_directory,
  File "/builder/tcbuilder/cli/build.py", line 408, in build
    raise exc
  File "/builder/tcbuilder/cli/build.py", line 399, in build
    handle_output_section(
  File "/builder/tcbuilder/cli/build.py", line 284, in handle_output_section
    handle_bundle_output(
  File "/builder/tcbuilder/cli/build.py", line 330, in handle_bundle_output
    "host_workdir": common.get_host_workdir()[0],
  File "/builder/tcbuilder/backend/common.py", line 413, in get_host_workdir
    container = docker_client.containers.get(container_id)
  File "/usr/local/lib/python3.9/dist-packages/docker/models/containers.py", line 889, in get
    resp = self.client.api.inspect_container(container_id)
  File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 16, in wrapped
    raise errors.NullResource(
docker.errors.NullResource: Resource ID was not provided

Here are the contents of tcbuild-r0.yml:

input:
  easy-installer:
    toradex-feed:
      version: "5.3.0"
      release: quarterly
      machine: colibri-imx6
      distro: torizon-upstream
      variant: torizon-core-docker
      build-number: "7"

customization:
  device-tree:
    include-dirs:
      - device-trees/include/
      - device-trees/dts-arm32/
    custom: device-trees/dts-arm32/imx6dl-colibri-eval-v3.dts
    overlays:
      add:
        - device-trees/overlays/vcu-r0-overlay.dts

output:
  easy-installer:
    local: vcu-r0-0.9
    name: "vcu-r0-0.9"
    description: "VCU firmware 0.9 for r0"
    bundle:
      compose-file: docker-compose.yml
      platform: linux/arm/v7
      registry: localhost:5000

And docker-compose.yml:

version: '2.4'
services:
  vcu:
    image: localhost:5000/vcu:0.9

I’ve also tried building using torizon-core bundle, but I received the same error:

user@debian:~/build-0.9$ torizoncore-builder bundle docker-compose.yml --platform linux/arm/v7 --dind-param="--insecure-registry=localhost:5000"
An unexpected Exception occured. Please provide the following stack trace to
the Toradex TorizonCore support team:


Traceback (most recent call last):
  File "/builder/torizoncore-builder", line 204, in <module>
    mainargs.func(mainargs)
  File "/builder/tcbuilder/cli/bundle.py", line 94, in do_bundle
    bundle(bundle_dir=args.bundle_directory,
  File "/builder/tcbuilder/cli/bundle.py", line 46, in bundle
    host_workdir = common.get_host_workdir()
  File "/builder/tcbuilder/backend/common.py", line 413, in get_host_workdir
    container = docker_client.containers.get(container_id)
  File "/usr/local/lib/python3.9/dist-packages/docker/models/containers.py", line 889, in get
    resp = self.client.api.inspect_container(container_id)
  File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 16, in wrapped
    raise errors.NullResource(
docker.errors.NullResource: Resource ID was not provided

Is there something else that I’m missing with my commands or configuration files?

Here’s the hardware and software I’m using:

Colibri iMX6DL 512MB IT V1.1A
Colibri Evaluation Board V3.2B
TorizonCore 5.3.0 build 7
TorizonCore Builder 3.1.0
Debian 11

2

Greetings @richgriswold,

I was able to find a way to reproduce this on my side. However I had to unnaturally force the situation to reproduce this and I’ve been unable to find a way to reproduce this issue naturally. This is to say I’m not sure why this is happening specifically to you.

But let’s try and figure this out together.

So the bit of code where this issue is occurring is here: torizoncore-builder/common.py at bullseye · toradex/torizoncore-builder · GitHub

Basically the chunk of code I referenced is trying to get the Container ID of the running instance of TorizonCore Builder. However for some reason in your case it’s not able to get a valid Container ID.

Could you please do the following for me on your development computer where you are running TorizonCore Builder:

  • Run this command:
docker run --entrypoint /bin/bash --rm -it -v /deploy -v $(pwd):/workdir -v storage:/storage --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:3.1.0
  • The above command should put you into a bash shell inside the TorzionCore Builder container.
  • Now what is the output of cat /proc/self/cgroup when you run this inside the container shell?

For reference here’s what I get:

cat /proc/self/cgroup
12:perf_event:/docker/4763feee455d2fce333d1496a640434675ac72fe614f58c0f930f3dc69f1d586
11:cpuset:/docker/4763feee455d2fce333d1496a640434675ac72fe614f58c0f930f3dc69f1d586
10:freezer:/docker/4763feee455d2fce333d1496a640434675ac72fe614f58c0f930f3dc69f1d586
9:hugetlb:/docker/4763feee455d2fce333d1496a640434675ac72fe614f58c0f930f3dc69f1d586
8:pids:/docker/4763feee455d2fce333d1496a640434675ac72fe614f58c0f930f3dc69f1d586
7:rdma:/
6:memory:/docker/4763feee455d2fce333d1496a640434675ac72fe614f58c0f930f3dc69f1d586
5:devices:/docker/4763feee455d2fce333d1496a640434675ac72fe614f58c0f930f3dc69f1d586
4:net_cls,net_prio:/docker/4763feee455d2fce333d1496a640434675ac72fe614f58c0f930f3dc69f1d586
3:cpu,cpuacct:/docker/4763feee455d2fce333d1496a640434675ac72fe614f58c0f930f3dc69f1d586
2:blkio:/docker/4763feee455d2fce333d1496a640434675ac72fe614f58c0f930f3dc69f1d586
1:name=systemd:/docker/4763feee455d2fce333d1496a640434675ac72fe614f58c0f930f3dc69f1d586

The 4763feee455d2fce333d1496a640434675ac72fe614f58c0f930f3dc69f1d586 is the Container ID that the tool is trying to determine. If you run docker ps on your development machine while the TorizonCore Builder container is running the container ID should match. For whatever reason in your case the Tool is not able to extract a valid container ID from /proc/self/cgroup. Which is why I would like for you to run through the above steps and let me know what you get.

Best Regards,
Jeremias

3

I ran the commands you listed, but cat /proc/self/cgroup doesn’t return the container ID:

user@debian:~$ docker run --entrypoint /bin/bash --rm -it -v /deploy -v $(pwd):/workdir -v storage:/storage --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:3.1.0
root@debian:/workdir# cat /proc/self/cgroup
0::/
root@debian:/workdir#

Running the command in a standard Debian container returns the same results:

user@debian:~$ docker run --rm -it debian /bin/bash -i
root@2c8a9660c23e:/# cat /proc/self/cgroup
0::/
root@2c8a9660c23e:/#

Next, I tried dumping /proc/self/cgroup on a different Linux system from the torizoncore-builder container and from the standard Debian container, and on that system I got results similar to yours from both containers, so it seems like something is messed up with Docker on the first Linux system.

4

I found a question on Stack Overflow where someone else is running into the same problem with Docker on Debian 11: https://stackoverflow.com/questions/69002675/. In my case, the Linux system where /proc/self/cgroup is returning the correct information is running Debian 10, and the one where it isn’t is running Debian 11. It looks like there may be ways to work around the change in Debian 11, but for now I’ll switch back to Debian 10.

Edit: After a bit more digging, I found that the container ID is still exposed in /proc/self/mountinfo:

user@debian:~$ docker run --entrypoint /bin/bash --rm -it -v /deploy -v $(pwd):/workdir -v storage:/storage --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:3.1.0
root@debian:/workdir# cat /proc/self/mountinfo
587 430 0:42 / / rw,relatime master:222 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/7UXPC4DMWOMAKZOPT2AU5TO42F:/var/lib/docker/overlay2/l/N23BUIPF6JSDAUEXPF3OVGNH63:/var/lib/docker/overlay2/l/WBCQ5TMZHES7JVXWO47UM55YG5:/var/lib/docker/overlay2/l/J43KCBC7576AQRYPCDWBHJ5PIA:/var/lib/docker/overlay2/l/IQSRCWDU7ARSW5SSYAA2MYR2ZT:/var/lib/docker/overlay2/l/PPCZZPQHTCQ4IW75U62SOSZJZ4:/var/lib/docker/overlay2/l/BL5EDRXDUHXPNG5ECDMVSUU6LP:/var/lib/docker/overlay2/l/O2MR4SA67GGKA3BX6SXBQNET5P:/var/lib/docker/overlay2/l/SYK2HVX6QTSMN3LTSEMC6MEK2V:/var/lib/docker/overlay2/l/A77ZL2BP77DWS5QH3RFNNVP4BP:/var/lib/docker/overlay2/l/NWBBA52E5JI2343JBIZOO7MEC4:/var/lib/docker/overlay2/l/WK47J7O3FBS6S2KA4CHI2AOX76:/var/lib/docker/overlay2/l/SCN6UKOU6DKEDFUAK3XY3ZD5VA:/var/lib/docker/overlay2/l/ZTTWJGFWJPK7E4PSRI7WU6B4IK:/var/lib/docker/overlay2/l/76JGUU6DDVGMCBLG65OKGPX6QV:/var/lib/docker/overlay2/l/GHNOJFO7OIQ5LZI3JDAUD4K2EP:/var/lib/docker/overlay2/l/B3VAX4JLENH4BJV635U7UBDJRC,upperdir=/var/lib/docker/overlay2/f790066af0926dde260f6d436f8bf1bbb4449fdf8bc1c74a38bc7c06b4be927f/diff,workdir=/var/lib/docker/overlay2/f790066af0926dde260f6d436f8bf1bbb4449fdf8bc1c74a38bc7c06b4be927f/work
588 587 0:57 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
589 587 0:58 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
590 589 0:60 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
591 587 0:20 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs rw
592 591 0:27 / /sys/fs/cgroup ro,nosuid,nodev,noexec,relatime - cgroup2 cgroup rw
593 589 0:56 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw
594 589 0:61 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,size=65536k
595 587 8:1 /home/user /workdir rw,relatime - ext4 /dev/sda1 rw,errors=remount-ro
596 587 8:1 /var/lib/docker/volumes/c90738bc179ec28d104cf44fb54625ff35efa0ff782508837cb5b41de5055cf9/_data /deploy rw,relatime master:1 - ext4 /dev/sda1 rw,errors=remount-ro
597 587 8:1 /var/lib/docker/volumes/storage/_data /storage rw,relatime master:1 - ext4 /dev/sda1 rw,errors=remount-ro
598 587 8:1 /var/lib/docker/containers/edaf7ad2c9a2c3b64ae9b22c9faeb2245f7ededd71ae44caab23e964f1a65769/resolv.conf /etc/resolv.conf rw,relatime - ext4 /dev/sda1 rw,errors=remount-ro
599 587 8:1 /var/lib/docker/containers/edaf7ad2c9a2c3b64ae9b22c9faeb2245f7ededd71ae44caab23e964f1a65769/hostname /etc/hostname rw,relatime - ext4 /dev/sda1 rw,errors=remount-ro
600 587 8:1 /var/lib/docker/containers/edaf7ad2c9a2c3b64ae9b22c9faeb2245f7ededd71ae44caab23e964f1a65769/hosts /etc/hosts rw,relatime - ext4 /dev/sda1 rw,errors=remount-ro
601 587 0:23 /docker.sock /run/docker.sock rw,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,size=302752k,mode=755
431 589 0:60 /0 /dev/console rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
432 588 0:57 /bus /proc/bus ro,relatime - proc proc rw
433 588 0:57 /fs /proc/fs ro,relatime - proc proc rw
434 588 0:57 /irq /proc/irq ro,relatime - proc proc rw
445 588 0:57 /sys /proc/sys ro,relatime - proc proc rw
495 588 0:57 /sysrq-trigger /proc/sysrq-trigger ro,relatime - proc proc rw
496 588 0:62 / /proc/asound ro,relatime - tmpfs tmpfs ro
512 588 0:63 / /proc/acpi ro,relatime - tmpfs tmpfs ro
513 588 0:58 /null /proc/kcore rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
526 588 0:58 /null /proc/keys rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
540 588 0:58 /null /proc/timer_list rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
541 588 0:58 /null /proc/sched_debug rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
542 591 0:64 / /sys/firmware ro,relatime - tmpfs tmpfs ro
root@debian:/workdir#

And from another terminal:

user@debian:~$ docker ps --no-trunc
CONTAINER ID                                                       IMAGE                               COMMAND                                            CREATED          STATUS          PORTS                    NAMES
edaf7ad2c9a2c3b64ae9b22c9faeb2245f7ededd71ae44caab23e964f1a65769   torizon/torizoncore-builder:3.1.0   "/bin/bash"                                        10 seconds ago   Up 10 seconds                            elated_kirch
b12ac3a8d6cd51ebb82968f3d1c3491113df47f7e56839411351d57c2ad5b213   registry:2                          "/entrypoint.sh /etc/docker/registry/config.yml"   2 months ago     Up 45 hours     0.0.0.0:5000->5000/tcp   registry
user@debian:~$

I’m not sure if this is guaranteed to always work this way, though.

5

Huh interesting, looking at the thread you linked it seems this will become an issue on any other distribution that also uses cgroups v2. I’ll report this internally to the team since this looks like it will become an issue going forward, with how we currently implement TorizonCore Builder.

For now yes please stick with Debian 10 while we work on a more future-proof solution. Thank you for reporting this issue and making us aware of this hole in our implementation.

Best Regards,
Jeremias

6

Hello everyone,
I was having the exact same problem.
Adding the “–cgroupns host” to the docker run helped in showing the cgroups inside the container, but the cgroup entries are now in a different format.

To get it working, in addition to the cgroupns switch, I had to:

  • Uninstall docker docker-compose inside the torizoncore-builder image (pip uninstall docker docker-compose)
  • Install both back (pip install docker docker-compose)
  • Modify line 422 of /builder/tcbuilder/backend/common.py from
    container_id = cgroup_name[cgroup_name.rfind("/")+1:-1]
    to
    container_id = cgroup_name[cgroup_name.rfind("/")+1:-1].replace("docker-", "").replace(".scope", "")
2 Likes
7

Hi @mag,

Thank you for the alternative solution. The issue here is still in our team’s backlog but, I’ll pass along your suggestion. Much appreciated!

Best Regards,
Jeremias