Error when remote debugging Python app with Torizon VSCode extension

CristianM · December 11, 2020, 5:51pm

Hi. I am trying to develop/debug Python apps using the Torizon VSCode extension. I have followed this article. When running the hello world example, I get the following error:

Preparing debug environment for Python application...
No preLaunchTask configured.
Selecting device...
Device 06650382 selected.
Updating app configuration...
Image is not up to date, building it (this may take some time)...
Error (500) - (2, 'WaitNamedPipe', 'The system cannot find the file specified.')
Internal server error

I can confirm that the module is accessible from the VSCode extension (it appears at the Devices tab).

What is the problem?

jeremias.tx · December 11, 2020, 10:16pm

Greetings @CristianM,

I wasn’t able to reproduce this on my setup. Are you running VSCode and the extension on a Windows or Linux desktop? Also are you using the stable version of the extension or the early access version?

Also could you provide the output of “Torizon IDE backend process”, available via the terminal pane in VSCode. With just the logs you’ve given it’s unclear to what file it is the system cannot find.

Best Regards,
Jeremias

CristianM · December 12, 2020, 10:26am

Hi. I am using VSCode and the extension on a Windows 10 desktop. This is the output of the Torizon IDE backend process:

Serving on http://Cristi-PC:5000
INFO:root:REST -> /api/version
INFO:root:REST <- /api/version - 200
INFO:root:REST -> /api/devices
INFO:root:REST <- /api/devices - 200
INFO:root:REST -> /api/applications/load
INFO:root:REST <- /api/applications/load - 200
INFO:root:REST -> /api/eulas
INFO:root:REST <- /api/eulas - 200
INFO:root:REST -> /api/devices/06650382/images
INFO:root:SSH - Creating tunnel to 06650382
2020-12-12 12:21:21,490| ERROR   | Could not establish connection from ('127.0.0.1', 52190) to remote side of the tunnel
ERROR:sshtunnel.SSHTunnelForwarder:Could not establish connection from ('127.0.0.1', 52190) to remote side of the tunnel
INFO:root:SSH - Creating tunnel to 06650382
2020-12-12 12:21:22,790| ERROR   | Could not establish connection from ('127.0.0.1', 52196) to remote side of the tunnel
ERROR:sshtunnel.SSHTunnelForwarder:Could not establish connection from ('127.0.0.1', 52196) to remote side of the tunnel
INFO:root:SSH - Creating tunnel to 06650382
2020-12-12 12:21:24,094| ERROR   | Could not establish connection from ('127.0.0.1', 52201) to remote side of the tunnel
ERROR:sshtunnel.SSHTunnelForwarder:Could not establish connection from ('127.0.0.1', 52201) to remote side of the tunnel
INFO:root:SSH - Creating tunnel to 06650382
WARNING:paramiko.transport:Success for unrequested channel! [??]
WARNING:paramiko.transport:Success for unrequested channel! [??]
WARNING:paramiko.transport:Success for unrequested channel! [??]
INFO:root:SSH - Tunnel to 06650382 activated
INFO:root:REST <- /api/devices/06650382/images - 200
INFO:root:REST -> /api/devices/06650382/containers
INFO:root:REST <- /api/devices/06650382/containers - 200
INFO:root:REST -> /api/devices/06650382/processes
INFO:root:SSH - Connecting to device 06650382
INFO:root:SSH - Connected to device 06650382
INFO:root:REST <- /api/devices/06650382/processes - 200
INFO:root:REST -> /api/devices/06650382/storage
INFO:root:REST <- /api/devices/06650382/storage - 200
INFO:root:REST -> /api/devices/06650382/memory
INFO:root:REST <- /api/devices/06650382/memory - 200
INFO:root:REST -> /api/platforms/arm32v7-debian-python3_buster/compatibledevicesINFO:root:REST <- /api/platforms/arm32v7-debian-python3_buster/compatibledevices - 200
INFO:root:REST -> /api/applications/cd1bbcfe-e7bc-4ef7-a0f0-4c4dd9c2da1c/updatedINFO:root:Image has never been built.
INFO:root:REST <- /api/applications/cd1bbcfe-e7bc-4ef7-a0f0-4c4dd9c2da1c/updated - 200
INFO:root:REST -> /api/applications/cd1bbcfe-e7bc-4ef7-a0f0-4c4dd9c2da1c/build
ERROR:root:(2, 'WaitNamedPipe', 'The system cannot find the file specified.')
Traceback (most recent call last):
  File "flask\app.py", line 1950, in full_dispatch_request
  File "flask\app.py", line 1936, in dispatch_request
  File "connexion\decorators\decorator.py", line 48, in wrapper
  File "connexion\decorators\uri_parsing.py", line 173, in wrapper
  File "connexion\decorators\validation.py", line 388, in wrapper
  File "connexion\decorators\parameter.py", line 126, in wrapper
  File "api.py", line 750, in applications_application_build_get
  File "applicationconfig.py", line 349, in build_image
  File "docker\models\images.py", line 279, in build
  File "docker\api\build.py", line 263, in build
  File "docker\utils\decorators.py", line 46, in inner
  File "docker\api\client.py", line 226, in _post
  File "requests\sessions.py", line 578, in post
  File "requests\sessions.py", line 530, in request
  File "requests\sessions.py", line 643, in send
  File "requests\adapters.py", line 439, in send
  File "urllib3\connectionpool.py", line 665, in urlopen
  File "urllib3\connectionpool.py", line 387, in _make_request
  File "http\client.py", line 1230, in request
  File "urllib3\connectionpool.py", line 387, in _make_request
  File "http\client.py", line 1230, in request
  File "http\client.py", line 1276, in _send_request
  File "http\client.py", line 1225, in endheaders
  File "http\client.py", line 1004, in _send_output
  File "http\client.py", line 944, in send
  File "docker\transport\npipeconn.py", line 32, in connect
  File "docker\transport\npipesocket.py", line 22, in wrapped
  File "docker\transport\npipesocket.py", line 50, in connect
pywintypes.error: (2, 'WaitNamedPipe', 'The system cannot find the file specified.')
INFO:root:REST <- /api/applications/cd1bbcfe-e7bc-4ef7-a0f0-4c4dd9c2da1c/build - 500

jeremias.tx · December 14, 2020, 7:14pm

Thank you, the full log gives a bit more information. I think I have an idea what’s wrong here. Based on the python trace, it seems the extension is unable to access/communicate with the Docker installation on your Windows machine.

Can you confirm whether Docker for Windows/Docker Desktop is up and running on your development host?

Best Regards,
Jeremias

CristianM · December 15, 2020, 12:06pm

Hi. I confirm that Windows Docker Desktop is running. I don’t know what happened, but now the Torizon Extension does not start. I get the following output:

[12-15 11:52:55.984] Initializing Torizon Extension
[12-15 11:52:56.024] ARM emulation for docker container does not seem to be enabled
This may lead to issues running Torizon tools.
Please check https://www.docker.com/blog/getting-started-with-docker-for-arm-on-linux/
or run the "Torizon:Enable ARM emulation for docker containers." command.
[12-15 11:52:56.039] Checking Moses ...
[12-15 11:52:56.040] Backend local instance running on port 5000

And the following notifications:

The terminal process "/home/torizon/.vscode-server/extensions/toradex.torizon-early-access-1.2.84/moses-linux/moses '--port', '5000'" terminated with exit code: 126.

ARM emulation for docker container does not seem to be enabled This may lead to issues running Torizon tools. Please check https://www.docker.com/blog/getting-started-with-docker-for-arm-on-linux/ or run the "Torizon:Enable ARM emulation for docker containers." command.

Cannot connect to Torizon IDE backend process.

If I run the enable ARM emulation command, I get the following message (although I can confirm that I’ve followed the instructions in Configure Build Environment for Torizon Containers).

Command 'Torizon: Enable ARM emulation for docker containers' resulted in an error (command 'torizon.enableARMemulation' not found)

If it’s of any use, when I executed the test command docker run --rm -it arm32v7/debian arch, I got the following message:

WARNING: The requested image's platform (linux/arm) does not match the detected host platform (linux/amd64) and no specific platform was requested
armv7l

What do you think is the problem here?

jeremias.tx · December 15, 2020, 7:59pm

Now the situation gets odder, this is strange.

Alright let’s take a step back and makes sure all the extension prerequisites are working properly. So you already confirmed that Docker Desktop is installed and running properly. Do you also have WSL2 with an appropriate Linux distro (we recommend Ubuntu 18.04) installed and working properly? As specified here: Visual Studio Code Extension for Torizon

Best Regards,
Jeremias

CristianM · December 15, 2020, 8:58pm

Yes. This is the output from Windows console:

wsl -l -v
  NAME                   STATE           VERSION
* docker-desktop-data    Running         2
  Ubuntu-18.04           Running         2
  docker-desktop         Running         2

This is the output from Ubuntu 18.04 LTS:

 uname -a
Linux Laptop3 4.19.128-microsoft-standard #1 SMP Tue Jun 23 12:58:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

However, entering the wsl command in the console doesn’t do nothing (in your tutorial it opens the Linux console). In order to access the Linux console, I use the Windows Terminal application. Also, if it’s of any use, my OS is Windows 10 Home Edition 20H2 (OS build 19042.685).

CristianM · December 16, 2020, 10:06am

I’ve changed the WSL backend. I can confirm that the problem was caused because I had enabled at the same time the Remote SSH extension. I disabled it and the Torizon extension connected.

But when it connects I get a SSH tunnel error and no containers are visible in the Devices tab, as you can see in the picture (you can find attached the logs).

Logs

What do you think is the problem? Also, I suspect the Python app will execute in the Torizon OS, not in the container. If I am right, can you please tell me if it’s possible to select the container where to execute/deploy the Python app?

jeremias.tx · December 16, 2020, 7:35pm

Interesting news, to be specific do you mean this remote ssh extension: Remote - SSH - Visual Studio Marketplace

As for the device SSH problem. Can you SSH/access the device from the PC normally outside of the extension?

Have you reflashed the device at any point? Doing so would have messed with the SSH keys used to access the device.

Also is this a new device entry or a pre-existing one? What happens when you try and recreate the device entry?

Also the extension is set to build and execute applications on Torizon OS in a container. It should never execute applications directly on the Torizon OS itself. What makes you suspect otherwise?

Best Regards,
Jeremias

CristianM · December 16, 2020, 9:09pm

Yes, that is the extension. Maybe you can try to recreate the problem yourself by installing the extension and relaunching VSCode.

Yes, SSH outside the extension is fine. SSH with the Remote SSH extension is also fine. The problem is that when using the Remote SSH extension, the Torizon extension does not work.

Yes, I have reflashed the module a few times. I don’t know what SSH keys you are referring to. If you are referring to the keys stored after Windows console or VSCode SSH sessions, in C:\Users\Username\.ssh, I have deleted these a few times, but they have not caused problems.

I thought it executed in the Torizon OS based on what I read and the videos I viewed, but I wasn’t sure. So, how can you select which container to deploy the Python app?

jeremias.tx · December 16, 2020, 11:47pm

After initial connection to a new device the extension should set up it’s own keys to access the device in the future. If you reflashed the device then I would assume the keys on the device side were also wiped with the reflash.

Can you try (without the Remote SSH extension) to delete the device entry in the extension then create a new one?

As a side note I can’t seem to reproduce any SSH connection issues with or without the Remote SSH extension.

When you create a new project with the extension you select the base container that the project uses. When your application is built a container is auto-generated based on the project’s configurations. The Dockerfile for this container should be present in your project’s “appconfig*” folder.

Once the container is built on the development host it’s then transferred and executed on the target device. So your application should always be in a container.

Best Regards,
Jeremias

CristianM · December 17, 2020, 5:50pm

Your suggestion worked. It connected after adding a new device. I’ve tried the Hello World Python console example. Unfortunately it didn’t work, because of certificate errors (I have attached the logs). Can you suggest a solution?

OK, I understand how it works and thank you. Is it possible to deploy the application to one of the containers already existing on the module (that were configured by me before creating the application using the extension)? If so, then how?

Logs

jeremias.tx · December 18, 2020, 4:22am

Oh that error you ran into is something we noticed very recently on the latest version of Docker Desktop.

The only known workaround at the moment is as follows. Open vs code, create a new, or open an existing project, wait for the window reload, open a windows terminal, type wsl --shutdown, then finally restart docker desktop. This has worked for some of our developers so far. It’s a bit tedious as you have to rerun this workaround every time a window reloads or you reopen VSCode.

The team is still investigating an actual fix.

As for your second question, in short no. The extension is designed to create a new container and deploy that to the device. If you already have an existing container you just need to configure the project to recreate it.

Best Regards,
Jeremias

CristianM · December 18, 2020, 9:53am

OK. I hope you find a fix. After the window reloads (including the Torizon extension), I executed the wsl --shutdown command and got this error when running the Hello World console example:

[12-18 12:14:08.191] Preparing debug environment for Python application...
[12-18 12:14:08.192] No preLaunchTask configured.
[12-18 12:14:08.192] Selecting device...
[12-18 12:14:08.544] Device 06650382 selected.
[12-18 12:14:08.544] Updating app configuration...
[12-18 12:14:08.908] Error (530) - Docker exception: Error while fetching server API version: (2, 'CreateFile', 'The system cannot find the file specified.')
[12-18 12:14:08.908] Local docker exception.

So, in order to recreate the container I need to modify only the Dockerfile.debug in appconfig_0/work, right?

jeremias.tx · December 18, 2020, 8:20pm

Apologies I forgot a step during the work around after the wsl --shutdown you then need to restart docker desktop. I amended my original comment with this step.

As for modifying your container. The Dockerfile(s) in appconfig are auto-generated by the extension based on the settings done in the configuration window of the extension. For example in the hello-world example you were referencing the guide had you add “python3-pexpect” to the “extrapackages” field this is equivalent to running apt install for that package in the generated container. This is how you configure and build up the Dockerfile.

Best Regards,
Jeremias

CristianM · December 18, 2020, 8:59pm

Thank you. It worked, finally.

jeremias.tx · December 15, 2020, 11:58pm

Ahh I think I see the issue now. Notice the * next to docker-desktop-data?

This means WSL2 is using docker-desktop-data as it’s WSL backend. For the extension to work properly WSL needs to be using Ubuntu or whatever chosen Linux distro as the WSL backend.

Best Regards,
Jeremias