TCB built from sources cannot create lockbox

I have downloaded a freshest TCB:

git log -1
commit 9e66e8483aedab7927a8d3f9bfc61d1ada48cf9d (HEAD -> bullseye, tag: 3.6.0-ea-20221009, origin/bullseye, origin/HEAD)

The I have prepared a slightly modified version of TCB to support private registries in “platform” command.
However in later stages (unmodified by me, nothing to do with uptane) I found an issue in executing “platform lockbox” command.

docker run --rm -it -v /deploy --name torizoncore-builder-test -v $(pwd):/workdir -v $(pwd)/changes1/usr/etc/:/etc/ -v storage:/storage --net=host -v /var/run/docker.sock:/var/run/docker.sock torizoncore-builder:fixed platform lockbox --credentials credentials.zip --registry-certs="$(pwd)/certs.d" lockbox

Please find the logs below. First line confirms that a change around using private registries was successful.

2022-10-17 08:24:57,821 - torizon.tcbuilder.backend.platform - INFO - Canonicalized file 'output_provisioned/docker-compose.lock.yml' has been generated.
2022-10-17 08:24:57,821 - torizon.tcbuilder.backend.platform - INFO - Pushing 'docker-compose.lock.yml' with package version 2022-10-17 to OTA server. You should keep this file under your version control system.
== uptane-sign stdout:
2022-10-17 08:24:57,843 - torizon.tcbuilder.backend.platform - DEBUG - No java installations was detected.
Please go to http://www.java.com/getjava/ and download
== uptane-sign stderr:
2022-10-17 08:24:57,843 - torizon.tcbuilder.backend.platform - WARNING - /usr/bin/uptane-sign: line 278: awk: command not found
2022-10-17 08:24:57,843 - root - ERROR - Error (1) running uptane command "uptane-sign" with arguments "['init', '--credentials', '/workdir/credentials.zip', '--repo', '/deploy/tuf-repo', '--verbose']"
2022-10-17 08:24:57,844 - root - DEBUG - Traceback (most recent call last):
  File "/builder/torizoncore-builder", line 217, in <module>
    mainargs.func(mainargs)
  File "/builder/tcbuilder/cli/platform.py", line 444, in do_platform_push
    platform.push_compose(credentials=credentials, target=args.target, version=version,
  File "/builder/tcbuilder/backend/platform.py", line 1123, in push_compose
    run_uptane_command(["uptane-sign", "init",
  File "/builder/tcbuilder/backend/platform.py", line 998, in run_uptane_command
    raise TorizonCoreBuilderError(
tcbuilder.errors.TorizonCoreBuilderError: Error (1) running uptane command "uptane-sign" with arguments "['init', '--credentials', '/workdir/credentials.zip', '--repo', '/deploy/tuf-repo', '--verbose']"

It looks that there is an issue with symbolic links. Those point to unexistent /etc/alternatives/* path.
I have prepared a quick workaround, however

RUN apt-get -q -y update && apt-get -q -y --no-install-recommends install gawk \
    && rm -rf /var/lib/apt/lists/* \
    && rm /usr/bin/awk && ln -s /usr/bin/gawk /usr/bin/awk \
    && rm /usr/bin/java && ln -s /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/bin/java

Then another Java problem have appeared:

== uptane-sign stderr:
2022-10-17 08:30:50,523 - torizon.tcbuilder.backend.platform - WARNING - Exception in thread "main" java.lang.InternalError: Failed to load default.policy

As I did not have a time to find the source on an issue I have used a manual passing a docker-compose.yml.lock file to TorizonPlatformServices what enabled me to create a Lockbox.

Please try to rebuild TCB from scratch and try to confirm the issue.

Greetings @marek.kucinski,

I can’t seem to reproduce this. I took the latest TorizonCore Builder commit as seen on Github (the commit hash matches what you showed). I then just ran platform lockbox and it worked.

Furthermore looking at the error log you produced something doesn’t make sense. In the traceback produced you can see the call stack going into the function do_platform_push. But this function is only called when doing something like platform push why is this being called when doing platform lockbox?

Best Regards,
Jeremias

You are right… The problem is not related to the lockbox, but with platform push

docker run --rm -it -v /deploy --name torizoncore-builder-test -v $(pwd):/workdir -v $(pwd)/changes1/usr/etc/:/etc/ -v storage:/storage --net=host -v /var/run/docker.sock:/var/run/docker.sock torizoncore-builder:fixed platform push --credentials credentials.zip --canonicalize --force --verbose output_provisioned/docker-compose.yml

I will do a clean build (unmodified) and provide you the status if the problem persists.
Sorry for bothering…

I did checkout the freshest snapshot. Then added certificates (address was replaced with xxx).

ADD certs/registry_own.crt /usr/local/share/ca-certificates/registry.crt
ADD certs/company_ca.crt /usr/local/share/ca-certificates/company_ca.crt

RUN mkdir -p /etc/docker/certs.d/xxx/ && \
    cp /usr/local/share/ca-certificates/registry.crt /etc/docker/certs.d/ca.crt && \
    cp /usr/local/share/ca-certificates/registry.crt /etc/docker/certs.d/xxx/ca.crt && \

RUN apt-get -q -y update && apt-get -q -y --no-install-recommends install \
    git ca-certificates \
    && rm -rf /var/lib/apt/lists/* \
    && update-ca-certificates

Please see the git status proof:

Your branch is up to date with 'origin/bullseye'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   torizoncore-builder.Dockerfile

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        certs/

I have created super simple docker-compose file which uses all docker-compose features we do use. It is purely fake - just to show non-private registry images example:

services:
  alpine:
    container_name: alpine
    image: alpine:latest
    ipc: host
    restart: always
    tty: true
    volumes:
    - ipcvolume:/externalCommunication/ipc:rw
  test_postgres:
    container_name: postgres
    image: postgres:latest
    ipc: host
    restart: always
    stdin_open: true
    tty: true
    cap_add:
    - SYS_TIME
    - MKNOD
    - SYS_ADMIN
    device_cgroup_rules:
    - b 8:* rmw
    devices:
    - /dev/gpiochip0
    - /dev/gpiochip1
    - /dev/gpiochip5
    - /dev/spidev0.0
    - /dev/input/event0
    - /dev/input/event1
    - /dev/rtc0
    security_opt:
    - seccomp:/etc/docker/seccomp/time.json
    ports:
    - 80:8080/tcp
    - 443:443/tcp
version: '2.4'
volumes:
  ipcvolume: {}

Then I run TCB manually from command line:

docker run --rm -it -v /deploy --name torizoncore-builder-test -v $(pwd):/workdir -v $(pwd)/changes1/usr/etc/:/etc/ -v storage:/storage --net=host -v /var/run/docker.sock:/var/run/docker.sock torizoncore-builder:fresh platform push --credentials credentials.zip --canonicalize --force --verbose docker-compose-test.yml
2022-10-18 07:00:42,414 - torizon.tcbuilder.backend.registryops - DEBUG - No token cached for scope repository:library/alpine:pull
2022-10-18 07:00:42,420 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): registry-1.docker.io:443
2022-10-18 07:00:44,132 - urllib3.connectionpool - DEBUG - https://registry-1.docker.io:443 "GET /v2/library/alpine/manifests/latest HTTP/1.1" 401 157
2022-10-18 07:00:44,140 - torizon.tcbuilder.backend.registryops - DEBUG - WWW-Authenticate header='Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/alpine:pull"'
2022-10-18 07:00:44,143 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): auth.docker.io:443
2022-10-18 07:00:45,014 - urllib3.connectionpool - DEBUG - https://auth.docker.io:443 "GET /token?service=registry.docker.io&scope=repository%3Alibrary%2Falpine%3Apull HTTP/1.1" 200 None
2022-10-18 07:00:45,016 - torizon.tcbuilder.backend.registryops - DEBUG - Using cached token for scope repository:library/alpine:pull
2022-10-18 07:00:45,017 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): registry-1.docker.io:443
2022-10-18 07:00:45,795 - urllib3.connectionpool - DEBUG - https://registry-1.docker.io:443 "GET /v2/library/alpine/manifests/latest HTTP/1.1" 200 1638
2022-10-18 07:00:45,797 - torizon.tcbuilder.backend.registryops - DEBUG - Manifest of 'library/alpine', 'latest' has digest 'sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad'
2022-10-18 07:00:45,798 - torizon.tcbuilder.backend.registryops - DEBUG - No token cached for scope repository:library/postgres:pull
2022-10-18 07:00:45,799 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): registry-1.docker.io:443
2022-10-18 07:00:46,545 - urllib3.connectionpool - DEBUG - https://registry-1.docker.io:443 "GET /v2/library/postgres/manifests/latest HTTP/1.1" 401 159
2022-10-18 07:00:46,547 - torizon.tcbuilder.backend.registryops - DEBUG - WWW-Authenticate header='Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/postgres:pull"'
2022-10-18 07:00:46,549 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): auth.docker.io:443
2022-10-18 07:00:47,306 - urllib3.connectionpool - DEBUG - https://auth.docker.io:443 "GET /token?service=registry.docker.io&scope=repository%3Alibrary%2Fpostgres%3Apull HTTP/1.1" 200 None
2022-10-18 07:00:47,309 - torizon.tcbuilder.backend.registryops - DEBUG - Using cached token for scope repository:library/postgres:pull
2022-10-18 07:00:47,310 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): registry-1.docker.io:443
2022-10-18 07:00:48,157 - urllib3.connectionpool - DEBUG - https://registry-1.docker.io:443 "GET /v2/library/postgres/manifests/latest HTTP/1.1" 200 1862
2022-10-18 07:00:48,158 - torizon.tcbuilder.backend.registryops - DEBUG - Manifest of 'library/postgres', 'latest' has digest 'sha256:769422529210357f359e7e5620246028837f84fabe92c838ca2aef75fe2e0741'
2022-10-18 07:00:48,172 - torizon.tcbuilder.backend.platform - INFO - Canonicalized file 'docker-compose-test.lock.yml' has been generated.
2022-10-18 07:00:48,172 - torizon.tcbuilder.backend.platform - INFO - Pushing 'docker-compose-test.lock.yml' with package version 2022-10-18 to OTA server. You should keep this file under your version control system.
== uptane-sign stdout:
2022-10-18 07:00:48,241 - torizon.tcbuilder.backend.platform - DEBUG - No java installations was detected.
Please go to http://www.java.com/getjava/ and download
== uptane-sign stderr:
2022-10-18 07:00:48,241 - torizon.tcbuilder.backend.platform - WARNING - /usr/bin/uptane-sign: line 278: awk: command not found

2022-10-18 07:00:48,241 - root - ERROR - Error (1) running uptane command "uptane-sign" with arguments "['init', '--credentials', '/workdir/credentials.zip', '--repo', '/deploy/tuf-repo', '--verbose']"
2022-10-18 07:00:48,242 - root - DEBUG - Traceback (most recent call last):
  File "/builder/torizoncore-builder", line 217, in <module>
    mainargs.func(mainargs)
  File "/builder/tcbuilder/cli/platform.py", line 440, in do_platform_push
    platform.push_compose(credentials=credentials, target=args.target, version=version,
  File "/builder/tcbuilder/backend/platform.py", line 1116, in push_compose
    run_uptane_command(["uptane-sign", "init",
  File "/builder/tcbuilder/backend/platform.py", line 991, in run_uptane_command
    raise TorizonCoreBuilderError(
tcbuilder.errors.TorizonCoreBuilderError: Error (1) running uptane command "uptane-sign" with arguments "['init', '--credentials', '/workdir/credentials.zip', '--repo', '/deploy/tuf-repo', '--verbose']"

The canonicalized form is created properly - just the exact image sha256 is placed within docker-compose file in favor to “:” format.

diff docker-compose-test.yml docker-compose-test.lock.yml
4c4
<     image: alpine:latest
---
>     image: alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad
11,16d10
<     container_name: postgres
<     image: postgres:latest
<     ipc: host
<     restart: always
<     stdin_open: true
<     tty: true
20a15
>     container_name: postgres
31,32c26,27
<     security_opt:
<     - seccomp:/etc/docker/seccomp/time.json
---
>     image: postgres@sha256:769422529210357f359e7e5620246028837f84fabe92c838ca2aef75fe2e0741
>     ipc: host
35a31,35
>     restart: always
>     security_opt:
>     - seccomp:/etc/docker/seccomp/time.json
>     stdin_open: true
>     tty: true

What I believe proofs that the problem is within official TCB’s Dockerfile itself.

Still not getting this issue with platform push. If you take the latest TorizonCore Builder as is from: GitHub - toradex/torizoncore-builder: TorizonCore Builder is a tool that allows the customization of TorizonCore images.

Build it locally and then use platform push do you see this error? Because it does not look like you are doing this. In your error log notice the lines:

== uptane-sign stdout:
2022-10-18 07:00:48,241 - torizon.tcbuilder.backend.platform - DEBUG - No java installations was detected.
Please go to http://www.java.com/getjava/ and download

In our latest Dockerfile you can see we install the Java dependencies right here: torizoncore-builder/torizoncore-builder.Dockerfile at bullseye · toradex/torizoncore-builder · GitHub

So why in your setup is Java not found? Did you forget to update/re-base the Dockerfile based on our latest changes?

Best Regards,
Jeremias

I did a fresh checkout in another directory.
Sadly - I have to modify the Dockerfile as our company proxies all outgoing traffic and therefore TLS verification fails when downloading sources for garage. See below what happens:

Step 14/46 : RUN cd aktualizr && mkdir build/ && cd build/ &&     cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_DEB=ON -DBUILD_SOTA_TOOLS=ON           -DSOTA_DEBIAN_PACKAGE_DEPENDS=openjdk-11-jre-headless           -DBUILD_OSTREE=ON           -DWARNING_AS_ERROR=OFF .. &&     make -j"$(nproc)" package 
...
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)>
make[2]: *** [src/sota_tools/CMakeFiles/garage-sign.dir/build.make:76: src/sota_tools/CMakeFiles/garage-sign] Error 1
make[1]: *** [CMakeFiles/Makefile2:4175: src/sota_tools/CMakeFiles/garage-sign.dir/all] Error 2
...
make: *** [Makefile:182: all] Error 2
The command '/bin/sh -c cd aktualizr && mkdir build/ && cd build/ &&     cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_DEB=ON -DBUILD_SOTA_TOOLS=ON           -DSOTA_DEBIAN_PACKAGE_DEPENDS=openjdk-11-jre-headless           -DBUILD_OSTREE=ON           -DWARNING_AS_ERROR=OFF .. &&     make -j"$(nproc)" package' returned a non-zero code: 2

That’s why I do have to add our CA before the build. This is the only change - adding certificates to trusted store.

Regarding Dockerfile - in fact - java and awk are installed, however python cannot find it. I think the problem lays here:

bash-5.1# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/builder
bash-5.1# find / -iname java
/usr/lib/jvm/java-11-openjdk-amd64/bin/java
/usr/share/doc/bison/examples/java
/usr/share/java
/usr/bin/java
/var/lib/dpkg/alternatives/java
bash-5.1# file /usr/bin/java
/usr/bin/java: broken symbolic link to /etc/alternatives/java

Something still doesn’t seem right. This is what I get when I build the latest TorizonCore Builder from Github:

docker run --rm -it --entrypoint /bin/bash torizoncore-builder:local
root@a4366eca97f0:/workdir# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/builder
root@a4366eca97f0:/workdir# find / -iname java
/var/lib/dpkg/alternatives/java
/etc/alternatives/java
/etc/ssl/certs/java
/usr/lib/jvm/java-11-openjdk-amd64/bin/java
/usr/bin/java
/usr/share/doc/bison/examples/java
/usr/share/java
root@a4366eca97f0:/workdir# file /usr/bin/java
/usr/bin/java: symbolic link to /etc/alternatives/java

It looks like the Java installation on your side is not correct or broken somehow. But how did this happen? We’re doing container builds using nearly the same Dockerfile, the resulting container image should be identical.

Best Regards,
Jeremias

Hi Jeremias,
I did a completely fresh build setup on a different machine. I can acknowledge that the build from scratch works. Then I started to bisect differences. Then I have realized

docker run --rm -it -v /deploy --name torizoncore-builder-test -v $(pwd):/workdir -v $(pwd)/changes1/usr/etc/:/etc/ -v storage:/storage --net=host -v /var/run/docker.sock:/var/run/docker.sock torizoncore-builder:fresh platform push --credentials credentials.zip --canonicalize --force --verbose docker-compose-test.yml

especially:

-v $(pwd)/changes1/usr/etc/:/etc/

brakes the contents of the /etc/ folder. I believe it is related to the groups and access rights.
Now it works.
Thank you for the time involved in helping me tracing this bug. Topic may be closed.

Thank you for confirming that nothing was amiss.