TorizonCore Builder in Azure Pipeline

Casper · August 26, 2024, 11:25am

Dear Toradex,
In our Azure DevOps pipeline we want to pre-provision our docker containers into our image. To do that we want to integrate the TorizonCore builder tool in the pipeline, it seems to work most of the time, however sometimes I run into the following error:

2024-08-26T10:36:13.2074689Z Starting DIND container
2024-08-26T10:36:13.2075125Z Using Docker host "tcp://127.0.0.1:22376"
2024-08-26T10:36:16.4930349Z Connecting to Docker Daemon at "tcp://127.0.0.1:22376"
2024-08-26T10:36:16.4942490Z Stopping DIND container
2024-08-26T10:36:17.6076582Z An unexpected Exception occurred. Please provide the following stack trace to
2024-08-26T10:36:17.6076932Z the Toradex TorizonCore support team:
2024-08-26T10:36:17.6076995Z 
2024-08-26T10:36:17.6077034Z 
2024-08-26T10:36:17.6193184Z Traceback (most recent call last):
2024-08-26T10:36:17.6193922Z   File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 198, in _new_conn
2024-08-26T10:36:17.6194501Z     sock = connection.create_connection(
2024-08-26T10:36:17.6194850Z   File "/usr/local/lib/python3.9/dist-packages/urllib3/util/connection.py", line 85, in create_connection
2024-08-26T10:36:17.6195050Z     raise err
2024-08-26T10:36:17.6195297Z   File "/usr/local/lib/python3.9/dist-packages/urllib3/util/connection.py", line 73, in create_connection
2024-08-26T10:36:17.6195461Z     sock.connect(sa)
2024-08-26T10:36:17.6195604Z ConnectionRefusedError: [Errno 111] Connection refused
2024-08-26T10:36:17.6195681Z 
2024-08-26T10:36:17.6195862Z The above exception was the direct cause of the following exception:
2024-08-26T10:36:17.6195946Z 
2024-08-26T10:36:17.6202527Z Traceback (most recent call last):
2024-08-26T10:36:17.6203246Z   File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-08-26T10:36:17.6203570Z     response = self._make_request(
2024-08-26T10:36:17.6203999Z   File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-08-26T10:36:17.6204267Z     raise new_e
2024-08-26T10:36:17.6204663Z   File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-08-26T10:36:17.6205002Z     self._validate_conn(conn)
2024-08-26T10:36:17.6205462Z   File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-08-26T10:36:17.6205718Z     conn.connect()
2024-08-26T10:36:17.6206148Z   File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 616, in connect
2024-08-26T10:36:17.6206429Z     self.sock = sock = self._new_conn()
2024-08-26T10:36:17.6206794Z   File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 213, in _new_conn
2024-08-26T10:36:17.6206944Z     raise NewConnectionError(
2024-08-26T10:36:17.6207159Z urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fe9cc910f10>: Failed to establish a new connection: [Errno 111] Connection refused
2024-08-26T10:36:17.6207299Z 
2024-08-26T10:36:17.6207423Z The above exception was the direct cause of the following exception:
2024-08-26T10:36:17.6207489Z 
2024-08-26T10:36:17.6207579Z Traceback (most recent call last):
2024-08-26T10:36:17.6207800Z   File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 486, in send
2024-08-26T10:36:17.6207921Z     resp = conn.urlopen(
2024-08-26T10:36:17.6208108Z   File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 847, in urlopen
2024-08-26T10:36:17.6208231Z     retries = retries.increment(
2024-08-26T10:36:17.6208425Z   File "/usr/local/lib/python3.9/dist-packages/urllib3/util/retry.py", line 515, in increment
2024-08-26T10:36:17.6208627Z     raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
2024-08-26T10:36:17.6209034Z urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='127.0.0.1', port=22376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fe9cc910f10>: Failed to establish a new connection: [Errno 111] Connection refused'))
2024-08-26T10:36:17.6209211Z 
2024-08-26T10:36:17.6209356Z During handling of the above exception, another exception occurred:
2024-08-26T10:36:17.6209466Z 
2024-08-26T10:36:17.6209605Z Traceback (most recent call last):
2024-08-26T10:36:17.6209979Z   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
2024-08-26T10:36:17.6210229Z     return self.version(api_version=False)["ApiVersion"]
2024-08-26T10:36:17.6210533Z   File "/usr/local/lib/python3.9/dist-packages/docker/api/daemon.py", line 181, in version
2024-08-26T10:36:17.6210672Z     return self._result(self._get(url), json=True)
2024-08-26T10:36:17.6210866Z   File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 46, in inner
2024-08-26T10:36:17.6210994Z     return f(self, *args, **kwargs)
2024-08-26T10:36:17.6211175Z   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 237, in _get
2024-08-26T10:36:17.6211304Z     return self.get(url, **self._set_request_timeout(kwargs))
2024-08-26T10:36:17.6211509Z   File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 602, in get
2024-08-26T10:36:17.6211630Z     return self.request("GET", url, **kwargs)
2024-08-26T10:36:17.6211827Z   File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 589, in request
2024-08-26T10:36:17.6211949Z     resp = self.send(prep, **send_kwargs)
2024-08-26T10:36:17.6212146Z   File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 703, in send
2024-08-26T10:36:17.6212263Z     r = adapter.send(request, **kwargs)
2024-08-26T10:36:17.6212446Z   File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 519, in send
2024-08-26T10:36:17.6212576Z     raise ConnectionError(e, request=request)
2024-08-26T10:36:17.6212952Z requests.exceptions.ConnectionError: HTTPSConnectionPool(host='127.0.0.1', port=22376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fe9cc910f10>: Failed to establish a new connection: [Errno 111] Connection refused'))
2024-08-26T10:36:17.6213161Z 
2024-08-26T10:36:17.6213258Z During handling of the above exception, another exception occurred:
2024-08-26T10:36:17.6213316Z 
2024-08-26T10:36:17.6213459Z Traceback (most recent call last):
2024-08-26T10:36:17.6213864Z   File "/builder/torizoncore-builder", line 222, in <module>
2024-08-26T10:36:17.6214059Z     mainargs.func(mainargs)
2024-08-26T10:36:17.6214251Z   File "/builder/tcbuilder/cli/bundle.py", line 97, in do_bundle
2024-08-26T10:36:17.6214461Z     bundle(bundle_dir=args.bundle_directory,
2024-08-26T10:36:17.6214617Z   File "/builder/tcbuilder/cli/bundle.py", line 49, in bundle
2024-08-26T10:36:17.6214729Z     bundle_be.download_containers_by_compose_file(
2024-08-26T10:36:17.6214864Z   File "/builder/tcbuilder/backend/bundle.py", line 556, in download_containers_by_compose_file
2024-08-26T10:36:17.6214984Z     dind_client = manager.get_client()
2024-08-26T10:36:17.6215102Z   File "/builder/tcbuilder/backend/bundle.py", line 321, in get_client
2024-08-26T10:36:17.6215256Z     dind_client = docker.DockerClient(base_url=self.docker_host, tls=tls_config)
2024-08-26T10:36:17.6215469Z   File "/usr/local/lib/python3.9/dist-packages/docker/client.py", line 45, in __init__
2024-08-26T10:36:17.6215592Z     self.api = APIClient(*args, **kwargs)
2024-08-26T10:36:17.6215779Z   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 197, in __init__
2024-08-26T10:36:17.6215916Z     self._version = self._retrieve_server_version()
2024-08-26T10:36:17.6216123Z   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
2024-08-26T10:36:17.6216251Z     raise DockerException(
2024-08-26T10:36:17.6216661Z docker.errors.DockerException: Error while fetching server API version: HTTPSConnectionPool(host='127.0.0.1', port=22376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fe9cc910f10>: Failed to establish a new connection: [Errno 111] Connection refused'))

The pipeline script looks like this:

 shopt -s expand_aliases # enable aliasses
 source tcb-env-setup.sh -a remote
 eval $(alias torizoncore-builder | sed "s/-it /-i --privileged /") # remove interactive mode tag in alias add privileged mode
 torizoncore-builder bundle --platform=linux/arm/v7 docker-compose.yml --bundle-directory bundle --dind-param="--privileged" --login-to docker.io * *

ToradexAI · August 26, 2024, 11:26am

Hi, I’m the Toradex AI Assistant. I’m in early beta, and I would appreciate your feedback.
The Toradex Support team will follow up with you shortly!

Did you like my answer?

The error you’re encountering seems to be related to the Docker-in-Docker (DIND) setup within your Azure DevOps pipeline. The Connection refused error typically indicates that the DIND container is not starting correctly or that the Docker daemon inside the DIND container is not accessible.

Here are a few steps to troubleshoot and potentially resolve this issue:

Ensure DIND Container Starts Properly:
Make sure that the DIND container is starting correctly and that the Docker daemon inside it is running. You can add some debug statements to check the status of the DIND container and the Docker daemon.
Increase Timeout:
Sometimes, the Docker daemon inside the DIND container might take a bit longer to start. You can try increasing the timeout for the connection attempts.
Check Docker Daemon Logs:
Inspect the logs of the DIND container to see if there are any errors or issues with the Docker daemon startup. This can provide more insight into why the connection is being refused.
Modify Pipeline Script:
Ensure that the DIND container is given enough privileges and resources. Your current script already includes --privileged, but you might want to ensure that other necessary parameters are also included.

Here is an updated version of your pipeline script with additional debug steps:

shopt -s expand_aliases # enable aliases
source tcb-env-setup.sh -a remote

# Start DIND container with additional debug and timeout
docker run --privileged --name dind-container -d docker:19.03.12-dind
sleep 10 # Wait for the Docker daemon to start

# Check if the Docker daemon is running
docker exec dind-container docker info
if [ $? -ne 0 ]; then
    echo "Docker daemon inside DIND container is not running."
    exit 1
fi

# Modify alias to remove interactive mode and add privileged mode
eval $(alias torizoncore-builder | sed "s/-it /-i --privileged /")

# Run the TorizonCore Builder command
torizoncore-builder bundle --platform=linux/arm/v7 docker-compose.yml --bundle-directory bundle --dind-param="--privileged" --login-to docker.io

# Stop and remove the DIND container
docker stop dind-container
docker rm dind-container

This script includes a check to ensure that the Docker daemon inside the DIND container is running before proceeding with the `torizon

Casper · August 26, 2024, 2:36pm

I got it working again by removing the --privileged tags, no idea why

bruno.tx · August 26, 2024, 5:07pm

Hello @Casper,

Thanks for the update.

Is there any reason the container needs to be run as privileged?

Best Regards,
Bruno

Casper · August 27, 2024, 7:26am

Hello Bruno,

I added those after I got the same error before, adding them seemed to fix the issue, however it suddenly stopped working again after an unrelated change to pipeline. Is there any way I can see some more debug info from torizoncore builder?

Casper · August 27, 2024, 7:54am

Don’t see anything obvious from --log-level DEBUG, it seems to work fine now that I removed --dind-param=“–privileged” so for now it is solved.

vix · August 27, 2024, 10:59am

Hi @Casper
your usage of TorizonCore Builder is Azure pipeline is interesting for me too.
Can I ask you how you setup the Azure DevOps .yaml file for the pipeline?

Thanks

Casper · August 27, 2024, 11:18am

this is basically it:

    steps:
    - script: |
        cd $TORIZON_CORE_BUILDER_DIR
        wget https://raw.githubusercontent.com/toradex/tcb-env-setup/master/tcb-env-setup.sh
        shopt -s expand_aliases # enable aliasses
        source tcb-env-setup.sh -a remote -n
        eval $(alias torizoncore-builder | sed "s/-it /-i /") # remove interactive mode tag in alias
        torizoncore-builder -v

download the script, use shopt to enable expending aliasses as it used in the script, source the script, edit the alias to work without interactive shell

vix · August 27, 2024, 12:44pm

but did you download the tcb container in Ubuntu in the pipeline in advance?

Casper · August 27, 2024, 1:41pm

sourcing tcb-env-setup.sh sets everything up

bruno.tx · August 27, 2024, 2:20pm

Hello @Casper,

As you did, using the --log-level flag will give more information on what TorizonCore Builder is doing.

It is good to know that just removing the --privileged was able to resolve the problem.
If you run into other issues, please feel free to create another topic on the Community.

Best Regards,
Bruno

Casper · October 10, 2024, 11:14am

Unfortunately the issue still remains on my VM, most of the time it works, but sometimes it doesn’t.

First time it doesn’t work:

c.pennings@terberg.com@aztamyocto:~/torizon-core-builder$ source tcb-env-setup.sh 
You may have an outdated version installed. Would you like to check for updates online? [y/n] n
Setting up TorizonCore Builder with version 3.

Setup complete! TorizonCore Builder is now ready to use.
TorizonCore Builder internal status and image customizations will be stored in Docker volume named 'storage'.
********************
Important: When you run TorizonCore Builder, the tool can only access the files inside the current working directory. Files and directories outside of the current working directory, or links to files and directories outside of the current working directory, won't be visible to TorizonCore Builder. So please make sure that, when running TorizonCore Builder, all files and directories passed as parameters are within the current working directory.
Your current working directory is: /home/c.pennings/torizon-core-builder
********************
For more information, run 'torizoncore-builder -h' or go to https://developer.toradex.com/knowledge-base/torizoncore-builder-tool

c.pennings@terberg.com@aztamyocto:~/torizon-core-builder$ torizoncore-builder --log-level DEBUG bundle --platform=linux/arm/v7 docker-compose.yml --bundle-directory bundle --login-to docker
.io casperterberg *****
2024-10-10 10:21:48,398 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:21:48,398 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:21:48,398 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:21:48,398 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:21:48,408 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /version HTTP/11" 200 826
2024-10-10 10:21:48,410 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22name%22%3A+%5B%22tcb_1728555707%22%5D%7D HTTP/11" 200 1741
2024-10-10 10:21:48,411 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/containers/0f02b04e3d1360348b7ebcee9d4e68cbf03147e65ddf202eb32c100d85826445/json HTTP/11" 200 None
2024-10-10 10:21:48,411 - torizon.tcbuilder.backend.common - DEBUG - Current container ID (found by container name): 0f02b04e3d1360348b7ebcee9d4e68cbf03147e65ddf202eb32c100d85826445
2024-10-10 10:21:48,412 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/containers/0f02b04e3d1360348b7ebcee9d4e68cbf03147e65ddf202eb32c100d85826445/json HTTP/11" 200 None
2024-10-10 10:21:48,413 - torizon.tcbuilder.cli.bundle - INFO - Creating Docker Container bundle...
2024-10-10 10:21:48,413 - compose.config.config - DEBUG - Using configuration files: /workdir/docker-compose.yml
2024-10-10 10:21:48,427 - torizon.tcbuilder.backend.bundle - DEBUG - Using DindManager
2024-10-10 10:21:48,427 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:21:48,427 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:21:48,427 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:21:48,427 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:21:48,432 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /version HTTP/11" 200 826
2024-10-10 10:21:48,438 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/info HTTP/11" 200 None
2024-10-10 10:21:48,438 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:21:48,438 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:21:48,438 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:21:48,438 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:21:48,442 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /version HTTP/11" 200 826
2024-10-10 10:21:48,444 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22name%22%3A+%5B%22tcb_1728555707%22%5D%7D HTTP/11" 200 1741
2024-10-10 10:21:48,445 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/containers/0f02b04e3d1360348b7ebcee9d4e68cbf03147e65ddf202eb32c100d85826445/json HTTP/11" 200 None
2024-10-10 10:21:48,445 - torizon.tcbuilder.backend.common - DEBUG - Current container ID (found by container name): 0f02b04e3d1360348b7ebcee9d4e68cbf03147e65ddf202eb32c100d85826445
2024-10-10 10:21:48,446 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/containers/0f02b04e3d1360348b7ebcee9d4e68cbf03147e65ddf202eb32c100d85826445/json HTTP/11" 200 None
2024-10-10 10:21:48,446 - torizon.tcbuilder.backend.bundle - INFO - 
Starting DIND container
2024-10-10 10:21:48,446 - torizon.tcbuilder.backend.bundle - INFO - Using Docker host "tcp://127.0.0.1:22376"
2024-10-10 10:21:48,451 - urllib3.connectionpool - DEBUG - http://localhost:None "POST /v1.46/volumes/create HTTP/11" 201 182
2024-10-10 10:21:48,451 - torizon.tcbuilder.backend.bundle - DEBUG - Default platform: linux/arm/v7
2024-10-10 10:21:48,451 - torizon.tcbuilder.backend.bundle - DEBUG - Volume mapping for DinD: [{'Target': '/workdir/', 'Source': '/home/c.pennings/torizon-core-builder', 'Type': 'bind', 'ReadOnly': False}, {'Target': '/var/lib/docker/', 'Source': 'dind-volume', 'Type': 'volume', 'ReadOnly': False}]
2024-10-10 10:21:48,451 - torizon.tcbuilder.backend.bundle - DEBUG - Running DinD container: ports=None, network=host
2024-10-10 10:21:48,564 - urllib3.connectionpool - DEBUG - http://localhost:None "POST /v1.46/containers/create?name=tcb-fetch-dind HTTP/11" 201 88
2024-10-10 10:21:48,565 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/containers/570bc0a78c48632d1ce9afd798226ca8b4d2b376dbd8cec98a505e24ffa4a9ed/json HTTP/11" 200 None
2024-10-10 10:21:48,682 - urllib3.connectionpool - DEBUG - http://localhost:None "POST /v1.46/containers/570bc0a78c48632d1ce9afd798226ca8b4d2b376dbd8cec98a505e24ffa4a9ed/start HTTP/11" 204 0
2024-10-10 10:21:50,685 - torizon.tcbuilder.backend.bundle - INFO - Connecting to Docker Daemon at "tcp://127.0.0.1:22376"
2024-10-10 10:21:50,685 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:21:50,685 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:21:50,685 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:21:50,685 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:21:50,686 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): 127.0.0.1:22376
2024-10-10 10:21:50,686 - torizon.tcbuilder.backend.bundle - INFO - Stopping DIND container
2024-10-10 10:21:50,762 - urllib3.connectionpool - DEBUG - http://localhost:None "POST /v1.46/containers/570bc0a78c48632d1ce9afd798226ca8b4d2b376dbd8cec98a505e24ffa4a9ed/stop HTTP/11" 204 0
2024-10-10 10:21:51,769 - urllib3.connectionpool - DEBUG - http://localhost:None "DELETE /v1.46/volumes/dind-volume HTTP/11" 204 0
2024-10-10 10:21:51,769 - root - CRITICAL - An unexpected Exception occurred. Please provide the following stack trace to
the Toradex TorizonCore support team:


2024-10-10 10:21:51,787 - root - ERROR - Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 196, in _new_conn
    sock = connection.create_connection(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 490, in _make_request
    raise new_e
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 466, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 1095, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 615, in connect
    self.sock = sock = self._new_conn()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 211, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f9d282c3a60>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 589, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='127.0.0.1', port=22376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9d282c3a60>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 223, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/usr/local/lib/python3.9/dist-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 44, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 246, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 622, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='127.0.0.1', port=22376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9d282c3a60>: Failed to establish a new connection: [Errno 111] Connection refused'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/builder/torizoncore-builder", line 222, in <module>
    mainargs.func(mainargs)
  File "/builder/tcbuilder/cli/bundle.py", line 97, in do_bundle
    bundle(bundle_dir=args.bundle_directory,
  File "/builder/tcbuilder/cli/bundle.py", line 49, in bundle
    bundle_be.download_containers_by_compose_file(
  File "/builder/tcbuilder/backend/bundle.py", line 555, in download_containers_by_compose_file
    dind_client = manager.get_client()
  File "/builder/tcbuilder/backend/bundle.py", line 320, in get_client
    dind_client = docker.DockerClient(base_url=self.docker_host, tls=tls_config)
  File "/usr/local/lib/python3.9/dist-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 207, in __init__
    self._version = self._retrieve_server_version()
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 230, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: HTTPSConnectionPool(host='127.0.0.1', port=22376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9d282c3a60>: Failed to establish a new connection: [Errno 111] Connection refused'))

But running the same command again afterwards, it does work:

c.pennings@terberg.com@aztamyocto:~/torizon-core-builder$ torizoncore-builder --log-level DEBUG bundle --platform=linux/arm/v7 docker-compose.yml --bundle-directory bundle --login-to docker.io casperterberg ***
2024-10-10 10:30:24,470 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:30:24,470 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:30:24,470 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:30:24,470 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:30:24,476 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /version HTTP/11" 200 826
2024-10-10 10:30:24,477 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22name%22%3A+%5B%22tcb_1728556223%22%5D%7D HTTP/11" 200 1741
2024-10-10 10:30:24,478 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/containers/e062d06acc82769a0393a9cd496bcdc8c22399a76a09079cf18cb3c53de5dd89/json HTTP/11" 200 None
2024-10-10 10:30:24,479 - torizon.tcbuilder.backend.common - DEBUG - Current container ID (found by container name): e062d06acc82769a0393a9cd496bcdc8c22399a76a09079cf18cb3c53de5dd89
2024-10-10 10:30:24,479 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/containers/e062d06acc82769a0393a9cd496bcdc8c22399a76a09079cf18cb3c53de5dd89/json HTTP/11" 200 None
2024-10-10 10:30:24,480 - torizon.tcbuilder.cli.bundle - INFO - Creating Docker Container bundle...
2024-10-10 10:30:24,480 - compose.config.config - DEBUG - Using configuration files: /workdir/docker-compose.yml
2024-10-10 10:30:24,485 - torizon.tcbuilder.backend.bundle - DEBUG - Using DindManager
2024-10-10 10:30:24,485 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:30:24,486 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:30:24,486 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:30:24,486 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:30:24,491 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /version HTTP/11" 200 826
2024-10-10 10:30:24,496 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/info HTTP/11" 200 None
2024-10-10 10:30:24,496 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:30:24,496 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:30:24,496 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:30:24,496 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:30:24,501 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /version HTTP/11" 200 826
2024-10-10 10:30:24,502 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22name%22%3A+%5B%22tcb_1728556223%22%5D%7D HTTP/11" 200 1741
2024-10-10 10:30:24,503 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/containers/e062d06acc82769a0393a9cd496bcdc8c22399a76a09079cf18cb3c53de5dd89/json HTTP/11" 200 None
2024-10-10 10:30:24,503 - torizon.tcbuilder.backend.common - DEBUG - Current container ID (found by container name): e062d06acc82769a0393a9cd496bcdc8c22399a76a09079cf18cb3c53de5dd89
2024-10-10 10:30:24,504 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/containers/e062d06acc82769a0393a9cd496bcdc8c22399a76a09079cf18cb3c53de5dd89/json HTTP/11" 200 None
2024-10-10 10:30:24,504 - torizon.tcbuilder.backend.bundle - INFO - 
Starting DIND container
2024-10-10 10:30:24,504 - torizon.tcbuilder.backend.bundle - INFO - Using Docker host "tcp://127.0.0.1:22376"
2024-10-10 10:30:24,510 - urllib3.connectionpool - DEBUG - http://localhost:None "POST /v1.46/volumes/create HTTP/11" 201 182
2024-10-10 10:30:24,510 - torizon.tcbuilder.backend.bundle - DEBUG - Default platform: linux/arm/v7
2024-10-10 10:30:24,510 - torizon.tcbuilder.backend.bundle - DEBUG - Volume mapping for DinD: [{'Target': '/workdir/', 'Source': '/home/c.pennings/torizon-core-builder', 'Type': 'bind', 'ReadOnly': False}, {'Target': '/var/lib/docker/', 'Source': 'dind-volume', 'Type': 'volume', 'ReadOnly': False}]
2024-10-10 10:30:24,510 - torizon.tcbuilder.backend.bundle - DEBUG - Running DinD container: ports=None, network=host
2024-10-10 10:30:24,596 - urllib3.connectionpool - DEBUG - http://localhost:None "POST /v1.46/containers/create?name=tcb-fetch-dind HTTP/11" 201 88
2024-10-10 10:30:24,597 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/containers/29163f294f150b6debfa7eba4aa383c9913b318bf5f002fca7c06c67b2b72b4d/json HTTP/11" 200 None
2024-10-10 10:30:24,700 - urllib3.connectionpool - DEBUG - http://localhost:None "POST /v1.46/containers/29163f294f150b6debfa7eba4aa383c9913b318bf5f002fca7c06c67b2b72b4d/start HTTP/11" 204 0
2024-10-10 10:30:27,703 - torizon.tcbuilder.backend.bundle - INFO - Connecting to Docker Daemon at "tcp://127.0.0.1:22376"
2024-10-10 10:30:27,703 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:30:27,703 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:30:27,703 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:30:27,703 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:30:27,704 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): 127.0.0.1:22376
2024-10-10 10:30:27,769 - urllib3.connectionpool - DEBUG - https://127.0.0.1:22376 "GET /version HTTP/11" 200 861
2024-10-10 10:30:27,769 - torizon.tcbuilder.backend.bundle - INFO - Attempting to log in to registry 'docker.io' with username=casperterberg
2024-10-10 10:30:27,769 - docker.utils.config - DEBUG - Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
2024-10-10 10:30:27,769 - docker.utils.config - DEBUG - No config file found
2024-10-10 10:30:27,769 - docker.auth - DEBUG - Looking for auth entry for 'docker.io'
2024-10-10 10:30:27,769 - docker.auth - DEBUG - No entry found
2024-10-10 10:30:29,135 - urllib3.connectionpool - DEBUG - https://127.0.0.1:22376 "POST /v1.40/auth HTTP/11" 200 48
2024-10-10 10:30:29,135 - torizon.tcbuilder.backend.bundle - INFO - Fetching container image casperterberg/hello-world:latest
2024-10-10 10:30:29,135 - docker.auth - DEBUG - Looking for auth config
2024-10-10 10:30:29,135 - docker.auth - DEBUG - Looking for auth entry for 'docker.io'
2024-10-10 10:30:29,135 - docker.auth - DEBUG - Found 'docker.io'
2024-10-10 10:30:29,135 - docker.auth - DEBUG - Found auth config
2024-10-10 10:30:30,255 - urllib3.connectionpool - DEBUG - https://127.0.0.1:22376 "POST /v1.40/images/create?tag=latest&fromImage=casperterberg%2Fhello-world&platform=linux%2Farm%2Fv7 HTTP/11" 200 None
  Pulling from casperterberg/hello-world
  2d3d56bba6ab: Pull complete 
  Digest: sha256:d4d5c6c7728f837602dfefe67da7aae5e318b75ac89b045ea38611e9a998d1b3
  Status: Downloaded newer image for casperterberg/hello-world:latest
2024-10-10 10:30:30,722 - urllib3.connectionpool - DEBUG - https://127.0.0.1:22376 "GET /v1.40/images/casperterberg/hello-world:latest/json HTTP/11" 200 1630
2024-10-10 10:30:30,722 - torizon.tcbuilder.backend.bundle - INFO - Saving Docker Compose file
2024-10-10 10:30:30,723 - torizon.tcbuilder.backend.bundle - INFO - Exporting storage
2024-10-10 10:30:30,723 - torizon.tcbuilder.backend.bundle - INFO - Storing container bundle into "bundle"
2024-10-10 10:30:30,723 - torizon.tcbuilder.backend.bundle - DEBUG - Volume mapping for tar container: [{'Target': '/mnt', 'Source': '/home/c.pennings/torizon-core-builder', 'Type': 'bind', 'ReadOnly': False}, {'Target': '/var/lib/docker/', 'Source': 'dind-volume', 'Type': 'volume', 'ReadOnly': True}]
2024-10-10 10:30:30,723 - torizon.tcbuilder.backend.bundle - DEBUG - tar command: ['tar', '--numeric-owner', '--preserve-permissions', '--directory=/var/lib/docker', "--xattrs-include='*'", '--create', '--file', '/mnt/bundle/docker-storage.tar', 'overlay2/', 'image/']
2024-10-10 10:30:30,809 - urllib3.connectionpool - DEBUG - http://localhost:None "POST /v1.46/containers/create?name=tcb-build-tar HTTP/11" 201 88
2024-10-10 10:30:30,811 - urllib3.connectionpool - DEBUG - http://localhost:None "GET /v1.46/containers/a3442572220f14d341b60e3059544790ed7527160150ccd77e95e6f123e28c46/json HTTP/11" 200 None
2024-10-10 10:30:31,170 - urllib3.connectionpool - DEBUG - http://localhost:None "POST /v1.46/containers/a3442572220f14d341b60e3059544790ed7527160150ccd77e95e6f123e28c46/start HTTP/11" 204 0
2024-10-10 10:30:31,171 - urllib3.connectionpool - DEBUG - http://localhost:None "POST /v1.46/containers/a3442572220f14d341b60e3059544790ed7527160150ccd77e95e6f123e28c46/wait HTTP/11" 200 None
2024-10-10 10:30:31,467 - urllib3.connectionpool - DEBUG - http://localhost:None "POST /v1.46/containers/a3442572220f14d341b60e3059544790ed7527160150ccd77e95e6f123e28c46/stop HTTP/11" 304 0
2024-10-10 10:30:31,479 - urllib3.connectionpool - DEBUG - http://localhost:None "DELETE /v1.46/containers/a3442572220f14d341b60e3059544790ed7527160150ccd77e95e6f123e28c46?v=False&link=False&force=False HTTP/11" 204 0
2024-10-10 10:30:31,479 - torizon.tcbuilder.backend.bundle - DEBUG - compression_command: ['xz', '-3', '-z', 'docker-storage.tar']
2024-10-10 10:30:31,494 - torizon.tcbuilder.backend.bundle - INFO - Stopping DIND container
2024-10-10 10:30:32,570 - urllib3.connectionpool - DEBUG - http://localhost:None "POST /v1.46/containers/29163f294f150b6debfa7eba4aa383c9913b318bf5f002fca7c06c67b2b72b4d/stop HTTP/11" 204 0
2024-10-10 10:30:33,581 - urllib3.connectionpool - DEBUG - http://localhost:None "DELETE /v1.46/volumes/dind-volume HTTP/11" 204 0
2024-10-10 10:30:33,582 - torizon.tcbuilder.cli.bundle - INFO - Successfully created Docker Container bundle in "bundle"!

Casper · October 17, 2024, 9:47am

I’ll close this issue and open a new issue as the scope has changed. This is a VM wide issue and not just an azure pipeline issue.