Torizoncore build fails DIND step with HTTP 409 Client Error

Technical Support
, , ,
1

Going back to build some test images, and torizoncore-builder no longer works. This setup worked great about 2 months ago.

Poking around here it looks like it’s possibly an error with our private registry, but I’ve confirmed that the username and password in tcbuild.yml are correct and that my native docker can login just fine.

Updating TorizonCore image in place.
Bundling images to directory bundle_20230202162053_097191.tmp
Starting DIND container
Stopping DIND container
Removing output directory 'displayTest' due to build errors
An unexpected Exception occured. Please provide the following stack trace to
the Toradex TorizonCore support team:


Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.41/containers/create?name=tcb-fetch-dind

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/builder/tcbuilder/backend/bundle.py", line 510, in download_containers_by_compose_file
    manager.start(network, default_platform=platform, dind_params=dind_params)
  File "/builder/tcbuilder/backend/bundle.py", line 252, in start
    self.dind_container = self.host_client.containers.run(
  File "/usr/local/lib/python3.9/dist-packages/docker/models/containers.py", line 811, in run
    container = self.create(image=image, command=command,
  File "/usr/local/lib/python3.9/dist-packages/docker/models/containers.py", line 870, in create
    resp = self.client.api.create_container(**create_kwargs)
  File "/usr/local/lib/python3.9/dist-packages/docker/api/container.py", line 430, in create_container
    return self.create_container_from_config(config, name)
  File "/usr/local/lib/python3.9/dist-packages/docker/api/container.py", line 441, in create_container_from_config
    return self._result(res, True)
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.41/containers/create?name=tcb-fetch-dind: Conflict ("Conflict. The container name "/tcb-fetch-dind" is already in 
use by container "39f41a7d2ec808a6832084c5af7ad22911e23ef5a826e251577b1db8cd654308". You have to remove (or rename) that container to be able to reuse that name.")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/builder/tcbuilder/backend/bundle.py", line 550, in download_containers_by_compose_file
    raise OperationFailureError(
tcbuilder.errors.OperationFailureError: Error: container images download failed: 409 Client Error for http+docker://localhost/v1.41/containers/create?name=tcb-fetch-dind: Conflict ("Conflict. The container name "/tcb-fetch-dind" is already in use by container "39f41a7d2ec808a6832084c5af7ad22911e23ef5a826e251577b1db8cd654308". You have to remove (or rename) that container to be able to reuse that name.")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.41/volumes/dind-volume

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/builder/torizoncore-builder", line 217, in <module>
    mainargs.func(mainargs)
  File "/builder/tcbuilder/cli/build.py", line 471, in do_build
    build(args.config_fname, args.storage_directory,
  File "/builder/tcbuilder/cli/build.py", line 457, in build
    raise exc
  File "/builder/tcbuilder/cli/build.py", line 446, in build
    handle_output_section(
  File "/builder/tcbuilder/cli/build.py", line 302, in handle_output_section
    handle_bundle_output(
  File "/builder/tcbuilder/cli/build.py", line 356, in handle_bundle_output
    download_containers_by_compose_file(**download_params)
  File "/builder/tcbuilder/backend/bundle.py", line 555, in download_containers_by_compose_file
    manager.stop()
  File "/builder/tcbuilder/backend/bundle.py", line 286, in stop
    self.dind_volume.remove()
  File "/usr/local/lib/python3.9/dist-packages/docker/models/volumes.py", line 25, in remove
    return self.client.api.remove_volume(self.id, force=force)
  File "/usr/local/lib/python3.9/dist-packages/docker/api/volume.py", line 161, in remove_volume
    self._raise_for_status(resp)
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.41/volumes/dind-volume: Conflict ("remove dind-volume: volume is in use - [39f41a7d2ec808a6832084c5af7ad22911e23ef5a826e251577b1db8cd654308]")
2

Greetings @ed-bear,

Looking at the error log these two lines stick out to me in particular:

Conflict ("Conflict. The container name "/tcb-fetch-dind" is already in 
use by container "39f41a7d2ec808a6832084c5af7ad22911e23ef5a826e251577b1db8cd654308". You have to remove (or rename) that container to be able to reuse that name.")

And,

Conflict ("remove dind-volume: volume is in use - [39f41a7d2ec808a6832084c5af7ad22911e23ef5a826e251577b1db8cd654308]")

For context the bundle process uses an intermediary container to handle the container images that are to be bundled. This intermediary container is named tcb-fetch-dind and it also uses a docker volume named dind-volume. If the errors here are correct it seems like there’s already a container and volume going by these names that are in use. Therefore, TorizonCore Builder can’t use these names for it’s own purpose and just fails.

I would suggest then trying to remove any containers and docker volumes with these names so that TorizonCore Builder can use these names for it’s own purposes.

Best Regards,
Jeremias

3

Deleting the old docker container fixed it, but odd that it was running at all when i just booted up the machine.

4

but odd that it was running at all when i just booted up the machine.

That is certainly strange. Normally once TorizonCore Builder is done with the intermediary container, it should stop it and clean it up. Perhaps a previous process was somehow interrupted leaving the intermediary container running but not cleaned up.

Well in any case it seems like your issue has been resolved. If you come upon this again let us know as there could still be a possible bug surrounding this.

Best Regards,
Jeremias