After updating project to use Torizon SDK 3.6.0 configurations, pull-container-torizon-release failing at tunnel-check

Torizon 7

spin-up-tunnel appears to work (although it always seems to say “is already running” even after performing spin-down-tunnel)

Executing task: /home/tim/.local/bin/xonsh /home/tim/normoplatform_GUI/.conf/spin-up-down-registry.xsh up not_my_password torizon 192.168.1.109 /home/tim/normoplatform_GUI 

Container Runtime: docker
Run Arguments: --rm -d --network host -e SSHPASS torizonextras/ide-port-tunnel:0.0.1 sshpass -e ssh -vv -N -R 5002:localhost:5002 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PubkeyAuthentication=no torizon@192.168.1.109
Container Name: torizon-ide-port-tunnel-f213e628
Container torizon-ide-port-tunnel-f213e628 already created
Checking if container torizon-ide-port-tunnel-f213e628 is running...
Container torizon-ide-port-tunnel-f213e628 is already running.

but then tunnel-check fails

Executing task: xonsh /home/tim/normoplatform_GUI/.conf/service-check.xsh registry 'not_my_password' 22 torizon 192.168.1.109 'curl --silent --max-time 5 http://localhost:5002/v2/_catalog' 

Exception occurred: Command '['sshpass', '-p', 'not_my_password', 'ssh', '-p', '22', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'PubkeyAuthentication=no', 'torizon@192.168.1.109', 'curl --silent --max-time 5 http://localhost:5002/v2/_catalog']' returned non-zero exit status 7.
Attempt 1/15: waiting for registry...

and also pull-from-target-release

Executing task: sleep 1 && DOCKER_HOST=192.168.1.109:2375 docker pull localhost:5002/normoplatform:v0.0.0 

Error response from daemon: Get "http://localhost:5002/v2/": dial tcp [::1]:5002: connect: connection refused

 *  The terminal process "/usr/bin/bash '-c', 'sleep 1 && DOCKER_HOST=192.168.1.109:2375 docker pull localhost:5002/normoplatform:v0.0.0'" terminated with exit code: 1. 

iiuc both the tunnel-check and the pull and trying to access the docker repository hosted on my development system from the host imx8 device (192.168.1.109). So I am suspicious of the use of localhost…

I suspect a problem with my config files, having updated them from an earlier version of the Torizon IDE template.

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.


Looking at the issue you’re experiencing with the tunnel-check and pull-container-torizon-release tasks failing after updating to Torizon SDK 3.6.0, this appears to be related to the local registry configuration and port forwarding setup. (1)

Understanding the Issue

The error indicates that the SSH tunnel for the local registry is not properly established, even though the tunnel container appears to be running. (1) The localhost:5002 reference is correct - it’s accessing the local registry through the SSH tunnel from the target device perspective. (2)

Configuration Check

First, verify your workspace settings in .vscode/settings.json. The key properties to check are: (1)

  • host_ip: Should contain the IP address of your development machine (1)
  • torizon_ip: Should contain the IP address of your target device (192.168.1.109 in your case) (1)
  • torizon_debug_ssh_port: Should be set to the correct SSH port (1)

Local Registry Configuration

The Torizon IDE Extension stores container images in local registries by default. (2) The tunnel setup creates an SSH port forward that allows the target device to access your development machine’s local registry through localhost:5002. (2)

Troubleshooting Steps

  1. Verify SSH connectivity: Test direct SSH access to your device:

    ssh torizon@192.168.1.109
    
  2. Check local registry: Ensure your local registry is running on port 5000:

    docker ps | grep registry
    
  3. Test tunnel manually: Try establishing the tunnel manually to verify the connection:

    ssh -N -R 5002:localhost:5000 torizon@192.168.1.109
    
  4. Verify registry access: From the target device, test if the registry is accessible:

    curl --silent --max-time 5 http://localhost:5002/v2/_catalog
    

Configuration File Updates

When updating from earlier Torizon IDE template versions, ensure your configuration files match the current template structure. (1) The host_ip property should be automatically set by the extension, but you can override it using torizon.overwriteHostIp if the automatic detection is selecting the wrong network interface. (1)

If the issue persists, try recreating the project configuration by running the Torizon IDE Extension setup again or comparing your configuration with a fresh template of the same type.

Best regards, Toradex AI Assistant.


Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

host_ip matches the ip for my windows system from the output of ifconfig
torizon_ip is correct
I have torizon_ssh_port = 22, which I think is the relevant port, since I am trying to deploy the release container - I’m not trying to debug.
I have "torizon_debug_ssh_port": "2231" if that helps

  1. ssh to the device works
  2. docker ps gives
    ´´´
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    69f3ffaa5158 torizon/weston-imx8:4 “/usr/bin/entry.sh” 7 hours ago Up 2 hours (healthy) torizon-weston-1
    ´´´
    This is the container running on the device.
    There is no entry for “registry”.
  3. the tunnel failed due to the fingerprint not being in the knownhosts. Is this a different knownhosts ? I have connected many times from my account. e.g. step 1. Is it connecting as torizon ?! Nevertheless, the password doesn’t reply with anything…

I need to clarify which specific OS version and hardware you’re using to provide the most accurate guidance.

Could you let me know:

  • Which version of Torizon OS are you running? (e.g., Torizon OS 6.8.2, Torizon OS 7.2.0-build.13)
  • Which hardware platform are you using? (e.g., Verdin i.MX8M Plus, Verdin AM62, Apalis i.MX8QP)

You can get this information by running the tdx-info command on your device. For more details, see: Getting Device Information with Tdx-Info | Toradex Developer Center

Key Issues Identified

Based on your description, there are several configuration issues:

1. Missing Local Registry

The fact that docker ps shows no registry container on your development machine is the primary issue (1). The Torizon IDE Extension expects a local Docker registry running on your development machine to store and share container images (1).

2. SSH Configuration Issues

The tunnel setup uses SSH with specific parameters including UserKnownHostsFile=/dev/null and StrictHostKeyChecking=no (1). However, you’re encountering fingerprint issues, which suggests the SSH configuration isn’t working as expected.

3. Port Configuration

You mentioned torizon_ssh_port = 22 and torizon_debug_ssh_port = 2231. The correct property for SSH port in workspace settings should be torizon_ssh_port (1).

Troubleshooting Steps

  1. Verify your local registry is running: Check if you have a Docker registry container running on your development machine on port 5000
  2. Check SSH key setup: The extension should automatically set up SSH keys when you add the device (2)
  3. Verify network connectivity: Ensure your development machine and target device can communicate properly

Once you provide the OS version and hardware details, I can give you more specific configuration guidance for your setup.

Best regards, Toradex AI Assistant.


Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

@ToradexAI Torizon 7; colibri imx8

It’s not running :-

curl https://localhost:5000/v2/
curl: (7) Failed to connect to localhost port 5000 after 0 ms: Couldn't connect to server

The Missing Local Registry

The issue is clear now - your local Docker registry isn’t running on your development machine. The Torizon IDE Extension requires a local registry to store and share container images between your development environment and the target device (1).

Starting the Local Registry

The Torizon IDE Extension should automatically start the local registry, but you can manually start it using the workspace tasks. In VS Code, you should have a task called run-docker-registry that handles this (2).

You can run this task by:

  1. Opening the Command Palette (Ctrl+Shift+P)
  2. Typing “Tasks: Run Task”
  3. Selecting run-docker-registry

Alternatively, you can check if the task exists in your .vscode/tasks.json file (2).

Manual Registry Setup

If the task isn’t working, you can manually start a local Docker registry:

docker run -d -p 5000:5000 --restart=always --name registry registry:2

Workspace Configuration

For Torizon 7 with Colibri iMX8, ensure your .vscode/settings.json has the correct configuration (1):

  • host_ip: Your Windows development machine IP
  • torizon_ip: Your Colibri iMX8 device IP (192.168.1.109)
  • torizon_ssh_port: Should be “22” for standard SSH (1)

Verifying the Setup

Once the registry is running, verify it’s accessible:

curl http://localhost:5000/v2/_catalog

This should return a JSON response showing available repositories.

The tunnel setup creates an SSH port forward from the target device’s localhost:5002 to your development machine’s localhost:5000, allowing the device to access your local registry (1).

Best regards, Toradex AI Assistant.


Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

@ToradexAI

run-docker-registry appears to run without error, but curl http://localhost:5000/v2/_catalog gives Failed to connect to localhost port 5000 after 0 ms

docker run -d -p 5000:5000 --restart=always --name registry registry:2 appears to work, it downloads and starts a container which I can see under “Individual Containers” in the docker view. However curl http://localhost:5000/v2/_catalog still gives Failed to connect to localhost port 5000 after 0 ms.

Troubleshooting the Local Registry Connection

Even though the Docker registry container is running, the connection issue suggests there might be a network or configuration problem. Let’s diagnose this step by step.

Verify Container Status

First, check if the registry container is actually running and listening on the correct port:

docker ps | grep registry

You should see output showing the registry container with port mapping 0.0.0.0:5000->5000/tcp.

Check Port Binding

Verify that port 5000 is actually bound and listening:

netstat -tulpn | grep :5000

Or on Windows:

netstat -an | findstr :5000

Test Registry Access

Try accessing the registry using different methods:

  1. Test with Docker directly:
docker logs registry
  1. Test with the container IP instead of localhost:
docker inspect registry | grep IPAddress

Then try curling that IP address on port 5000.

Windows-Specific Considerations

Since you’re on Windows, there might be networking issues between WSL2 and Windows. If you’re using Docker Desktop with WSL2:

  1. Try using 127.0.0.1 instead of localhost:
curl http://127.0.0.1:5000/v2/_catalog
  1. Check if Windows Defender or firewall is blocking port 5000

Local Registry Configuration

The Torizon IDE Extension expects the local registry to be accessible (1). The registry should respond to requests at /v2/_catalog endpoint (1).

Alternative Registry Setup

If the manual registry still doesn’t work, try using the docker-compose approach that the IDE Extension might be expecting. Check your workspace for a docker-compose.yml file that might include registry configuration (1).

The pull-through cache configuration shows how registries should be set up (1):

version: "3"
services:
  docker-registry:
    container_name: docker-registry
    image: registry.hub.docker.com/library/registry:2
    ports:
      - 5000:5000/tcp
    volumes:
      - ./config.yml:/etc/docker/registry/config.yml
      - ${HOME}/registry:/var/lib/registry
    restart: unless-stopped

(1)

Next Steps

Once the registry is accessible at http://localhost:5000/v2/_catalog, the tunnel setup should work properly, allowing your Colibri iMX8 device to access the registry through the SSH tunnel on localhost:5002 (2).

Best regards, Toradex AI Assistant.


Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

@ToradexAI
~# docker run -d -p 5000:5000 --restart=always --name registry registry:2

~# docker ps | grep registry
c52524eb869b registry:2 "/entrypoint.sh /etc…" 51 seconds ago Up 50 seconds 0.0.0.0:5000->5000/tcp, [::]:5000->5000/tcp registry

but

~# sudo netstat -tulpn

tcp        0      0 0.0.0.0:5002            0.0.0.0:*               LISTEN      519/docker-proxy    
tcp6       0      0 :::5002                 :::*                    LISTEN      526/docker-proxy

I see the issue now! The registry container is running but it’s bound to port 5002 instead of port 5000. This explains why your curl to localhost:5000 fails but the container appears to be running.

Port Conflict Issue

Your netstat output shows:

  • Port 5002 is bound by docker-proxy (your registry)
  • Port 5000 is not listening

But your docker ps shows the registry should be on port 5000. This suggests there’s a port conflict or the container started with different port mapping than expected.

Resolving the Port Issue

  1. Stop and remove the existing registry container:
docker stop registry
docker rm registry
  1. Check if port 5000 is already in use:
sudo netstat -tulpn | grep :5000
  1. Start the registry on the correct port:
docker run -d -p 5000:5000 --restart=always --name registry registry:2

Verify the Fix

After starting the registry correctly, verify:

  1. Check the port binding:
docker ps | grep registry

You should see 0.0.0.0:5000->5000/tcp

  1. Verify port 5000 is listening:
sudo netstat -tulpn | grep :5000
  1. Test registry access:
curl http://localhost:5000/v2/_catalog

Torizon IDE Extension Registry Requirements

The Torizon IDE Extension stores container images in local registries by default (1). The local registry should run on port 5000 to work properly with the SSH tunnel setup that forwards from the target device’s localhost:5002 to your development machine’s localhost:5000 (1).

Once the registry is accessible on port 5000, the tunnel-check and pull-container-torizon-release tasks should work correctly.

Best regards, Toradex AI Assistant.


Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

@ToradexAI

After a cold reboot, I have left VSCode and the Torizon extension to run normally. I did not start the docker registry manually.

~#sudo netstat -tulpn

gives

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:5002            0.0.0.0:*               LISTEN      438/docker-proxy    
tcp6       0      0 :::5002                 :::*                    LISTEN      446/docker-proxy

however

curl https://localhost:5000/v2/_catalog
curl: (7) Failed to connect to localhost port 5000 after 0 ms: Couldn't connect to server

there is no docker container called registry.

So, why are there entries in netstat for port 5002 ? I noticed earlier somewhere a command to start the registry was using 2 ports; 5002:5000.

I guess that the torizon extension is trying to start the registry, but something is failing.
How does the torizon extension start the registry ? Where is the code that does this, and where are the settings that it uses ?

@ToradexAI

okay - I found this in share-wsl-ports.xsh

# FIXME:    this presumes that the xonsh is installed in the default location
#           that is the user path .local/bin
super_script += f" wsl -e /{home}/.local/bin/xonsh {workspace}/.vscode/tasks.xsh run run-docker-registry-wsl ; "

which seems to be called from the task

            "label": "run-share-wsl-ports",
            "detail": "",
            "hide": true,
            "command": "xonsh",
            "type": "shell",
            "args": [
                "${workspaceFolder}/.conf/share-wsl-ports.xsh",
                "${workspaceFolder}"
            ],

@ToradexAI
On the other hand I note that when I run the release container from a new template project, everything works as expected, except that the checks suggested above for the registry also fail the same way as for my project. So I think we’ll have to reset our search for the problem.

@ToradexAI

So, after the cold boot, I think that cured the problem, since pull-container and tunnel-check were working.

Quite what the problem was, I have no clue.

But the AI suggestions were misleading :frowning:

Hi @Timo!

I’m glad you could solve the issue.

I’m sorry the AI suggestions were misleading. We’re currently working to improve its responses.

Best regards.

Just to note that this problem occurred again today. I fixed it by rebooting windows. But on shutdown, I noticed that 2 powershell sessions were still running. I’m guessing that the powershell sessions are those that are responsible for the repository, and are still occupying the 5002:5000 ports (there were 2 since I made 2 attempts at running the application, reloading the “folder”).

Maybe something could be improved in the extension to force quit the powershell instance (though it sounds a bit hard to me !) ?

Hi @Timo!

PowerShell is no longer used by our IDE extension. I recommend updating the IDE extension before continuing with the development.

Best regards.