TorizonCore Builder - Docker issue

Technical Support
, , ,
1

Hello,

I am in the process to customize the default torizon image. I am applying a custom device tree, add kernel modules and provision docker containers.

Here starts the issue. I get follow error when the docker part starts:

Creating Docker Container bundle...
Starting DIND container
Using Docker host "tcp://127.0.0.1:22376"
Connecting to Docker Daemon at "tcp://127.0.0.1:22376"
Stopping DIND container
An unexpected Exception occured. Please provide the following stack trace to
the Toradex TorizonCore support team:


Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 353, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 181, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f321ccf2490>: Failed to establish a new connection: [Errno 111] Connection refused

There are couple more exceptions after this.

I am using this image “torizon-core-docker-apalis-imx8x-Tezi_5.3.0+build.7.tar” and an Apalis imx8x module.

Even the thsi command from the pre-provisioning article throws the same errors: “torizoncore-builder bundle --platform=linux/arm64 files/docker-compose.yml --bundle-directory bundle”

Any idea on what i need to change? Docker hello-world test works.

2

It seems you don’t have the connection to the docker registry you provide in the compose file. As the torizon builder tries to pull the image from the registry specified in the compose file, you should have access to that and also provide the credentials for private repositories to torizon builder.

I would recommend you try to pull the image and see if you can access the image. Also check if you pass the credentials to the torizon builder

3

I can pull the hello-world demo as a test.

In my compose file i am pulling the weston-vivante:2 image and the kiosk image from the example file provided by the builder page. I presume those are free to pull.

4

Greetings @c.gantner,

I don’t encounter any issues when running the example on this page: Pre-provisioning Docker Containers onto a TorizonCore image | Toradex Developer Center

What kind of machine are you running TorizonCore Builder on? Is it a Linux machine or a Windows machine running WSL 2?

Also judging by the stack trace it looks like communication with the docker-in-docker (DIND) container could not be established. Do you have any kind of firewall or network restrictions? As you can see from the command it is trying to communicate via tcp://127.0.0.1:22376.

Best Regards,
Jeremias

5

Hello Jeremias,

The machine i use is a linux PC with MX linux 19.4 (debian buster) installed on it.

I saw that the issue is happening when entering DIND. I will start to search for the issue on google.
Network is configured as default, i did not change anything.

I still have 2 issues at the moment during the build process, but i will make separate threads for that.

LE: From where does the script get the port and ip of the daemon? (i saw that the port should be 2376)

Thanks,
Cristian

6

The torizoncore-builder tool is simply connecting to the localhost via an arbitrary port to try and communicate with the host Docker daemon. We just arbitrarily choose 22376 because it’s a high numbered port that isn’t very likely to have any conflicts.

You can see the logic for the bundle command here: https://github.com/toradex/torizoncore-builder/blob/bullseye/tcbuilder/backend/bundle.py

Best Regards,
Jeremias

7

Resolved the connection issue by switching from MX Linux to Linux Mint.
After this the built went as expected.

Thank you for your help.

8

Interesting, well glad to see you were able to resolve the issue.

If you ever figure out what the issue with MX Linux was, it would be appreciated if you could share. This might help others who use this or a similar distribution.

Best Regards,
Jeremias

9

I am experiencing the same issue:
Windows 10
WSL2 with Ubuntu

I was following the instructions at:

cd ~/tcbworkdir
# Get the setup script and run it
wget https://raw.githubusercontent.com/toradex/tcb-env-setup/master/tcb-env-setup.sh
source tcb-env-setup.sh -- -p 8080:8080
# Download the base image
wget https://artifacts.toradex.com/artifactory/torizoncore-oe-prod-frankfurt/dunfell-5.x.y/release/11/apalis-imx8/torizon/torizon-core-docker/oedeploy/torizon-core-docker-apalis-imx8-Tezi_5.5.0+build.11.tar
# Get the sample docker-compose.yml
wget https://docs.toradex.com/108979-docker-compose-example-arm64.yml
mv 108979-docker-compose-example-arm64.yml docker-compose.yml
# Make sure I can run the arm version on windows
docker run --rm -it --privileged torizon/binfmt
docker run --rm -it arm32v7/debian arch
# Output from it is WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested
# armv7l
# Just to make sure, see what we have in the current directory
ls -F
# docker-compose.yml  tcb-env-setup.sh  torizon-core-docker-apalis-imx8-Tezi_5.5.0+build.11.tar
torizoncore-builder bundle --platform=linux/arm64 docker-compose.yml --bundle-directory bundle

Unfortunately the bundle command fails with an error

Creating Docker Container bundle...
Starting DIND container
Using Docker host "tcp://127.0.0.1:22376"
Connecting to Docker Daemon at "tcp://127.0.0.1:22376"
Stopping DIND container
An unexpected Exception occured. Please provide the following stack trace to
the Toradex TorizonCore support team:


Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 358, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f1d88c41580>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='127.0.0.1', port=22376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1d88c41580>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/usr/local/lib/python3.9/dist-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 542, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='127.0.0.1', port=22376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1d88c41580>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/builder/torizoncore-builder", line 215, in <module>
    mainargs.func(mainargs)
  File "/builder/tcbuilder/cli/bundle.py", line 94, in do_bundle
    bundle(bundle_dir=args.bundle_directory,
  File "/builder/tcbuilder/cli/bundle.py", line 50, in bundle
    bundle_be.download_containers_by_compose_file(
  File "/builder/tcbuilder/backend/bundle.py", line 505, in download_containers_by_compose_file
    dind_client = manager.get_client()
  File "/builder/tcbuilder/backend/bundle.py", line 300, in get_client
    dind_client = docker.DockerClient(base_url=self.docker_host, tls=tls_config)
  File "/usr/local/lib/python3.9/dist-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: HTTPSConnectionPool(host='127.0.0.1', port=22376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1d88c41580>: Failed to establish a new connection: [Errno 111] Connection refused'))

I’ve made sure I have the latest version of Docker Desktop and that the WSL integration is enabled. I have also tried running in Visual Studio under the terminal, in powershell on a WSL terminal, both with and without using SUDO. The result is the same in all cases.

Am I missing some critical step?

10

Hi @toebes,

This seems to be a recent new issue. I was able to reproduce this same error myself on my own Windows/WSL setup. There is no issue with the same commands on a pure Linux machine, so the issue seems to be just with Windows/WSL. I’m not quite sure of the root cause yet but I’ll report this and see if we can get some further investigations by the team.

Thank you for reporting the issue.

Best Regards,
Jeremias

11

Thank you. It feels a bit better to know that someone else is encountering the same problem. I’ll use the time to tackle a different problem in the meantime while awaiting your solution!

13

I’m checking in to see if you have any insight or a workaround. It would be nice to be able to build my image.

14

Well the most immediate workaround would be to try and do the bundle on a Linux machine if available. As I said this strange issue seems to only affect Windows/WSL environments.

Other than that there’s not much else that could be done. Technically I suppose you could try and do what bundle does manually. But this would be very error-prone/complex since the bundle command does quite a bit under the hood.

Best Regards,
Jeremias

15

So I have at my disposal two mac minis (one Intel and one M1 based), a QNAP server with virtualization technology, and a high end window 10 box. I was under the impression that the tools don’t work on the mac platform, has that changed - particularly given that the new Mac Mini is ARM64 based? Any suggestions for which is the most promising? I also have an Azulle Quantum Byte (Atom based) which runs Linux but the memory and disk space is very limited.

What type of Linux box do you typically recommend?

16

Ok I’ve spent a day to build a new Linux box with Debian 11/Bullseye running on an Intel processor and went through all the steps again to do the build, but it is now generating a different error that it says to provide to the support team.

root@Goji-Build:~/tcbworkdir# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye
root@Goji-Build:~/tcbworkdir# ls
docker-compose.yml  tcb-env-setup.sh
root@Goji-Build:~/tcbworkdir# torizoncore-builder bundle --platform=linux/arm64 docker-compose.yml --bundle-directory bundle
An unexpected Exception occured. Please provide the following stack trace to
the Toradex TorizonCore support team:


Traceback (most recent call last):
  File "/builder/torizoncore-builder", line 215, in <module>
    mainargs.func(mainargs)
  File "/builder/tcbuilder/cli/bundle.py", line 94, in do_bundle
    bundle(bundle_dir=args.bundle_directory,
  File "/builder/tcbuilder/cli/bundle.py", line 46, in bundle
    host_workdir = common.get_host_workdir()
  File "/builder/tcbuilder/backend/common.py", line 426, in get_host_workdir
    container = docker_client.containers.get(container_id)
  File "/usr/local/lib/python3.9/dist-packages/docker/models/containers.py", line 889, in get
    resp = self.client.api.inspect_container(container_id)
  File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 16, in wrapped
    raise errors.NullResource(
docker.errors.NullResource: Resource ID was not provided

What do I need to do in order to build an image?

Thank you very much

17

That’s strange that’s an older bug that should’ve been fixed already. Perhaps the team hasn’t pushed the fix to Dockerhub yet.

Try this, this is the Github repository for TorizonCore Builder: GitHub - toradex/torizoncore-builder: TorizonCore Builder is a tool that allows the customization of TorizonCore images.

It should have the fix for the issue you just encountered. Clone this to your system and follow the instructions in the Building and running the Docker image of the tool section of the README. Then try the bundle command again it should be fixed. Meanwhile I’ll see if I can get the team to actually push this fix to Dockerhub publicly.

Best Regards,
Jeremias

18

Building it from the Github fixes it. Thanks! Let me know when the Dockerhub is fixed so I can switch back to the main deployment.

Thank you!

19

Glad to hear the Github version fixes it. The new Dockerhub version is waiting on some other in-progroess changes. But once those are ready a new version will be published.

Best Regards,
Jeremias

20

Hi @toebes,

I believe we have found out the original issue with the bundle command in WSL/Windows. Turns out the issue is with the tcb-env-setup.sh script. A recent change caused TorizonCore Builder to set up differently in a WSL/Windows environment. This different setup then causes issue with the bundle command.

It appears using an older version of the setup script seems to alleviate the issues with bundle. While we fix the setup script you could try to use an older version of the setup script like this: tcb-env-setup/tcb-env-setup.sh at ea116a08ae9a6ad3893b6c6ea988cbb7ec871d96 · toradex/tcb-env-setup · GitHub

Or you can continue to use your Linux box as well since there’s no issues with that.

Best Regards,
Jeremias

1 Like