IDE extensions connect to device but fail to get device information(images, containers, processes)

We are evaluating the Torizon platform with a Colibri iMX6DL/S on top of a Colibri Evaluation Board V3. We managed to install the TorizonCore image with no containers pre-provisioned on the Colibri. We were able to connect to device through SSH, downloaded and then executed some container images from the Torizon dockerhub channel for testing purposes.

Moving on to the developing environment, we followed instructions to set up both Visual Studio Code and Visual Studio 2019 IDE extensions. On both IDEs we were able to detect the device but failed to get device information of images, containers, processes.

The extensions would seem hang for long time and afterwords print an error 500: Internal server error. Further error details mention an error 403 (Client Error Forbidden) while trying to access “http://127.0.0.1:someport/version”.

Analyzing the requests and responses between the IDE extension and the moses backend server we observed that the extension would first request the docker version, then the eulas, then all configured devices, then the device details with the device ID, and finally it would request docker images in the device. At this point is when the error 500 would be returned.

Inspecting the error details it looks like a firewall response. We do have a corporate firewall, but as far as we know it only blocks upstream requests not local requests.

"Error while fetching server API version: 403 Client Error for http://127.0.0.1:1037/version: Forbidden (\"b'<!DOCTYPE html \\n     PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"\\n     \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\\n\\n<HTML><HEAD><TITLE>Endian Firewall - Access denied</TITLE>\\n<meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\">\\n<style type=\"text/css\" media=\"screen\">\\n/* Stylesheet for endian firewall */\\n/* Author: Raphael Vallazza */\\n\\n/* body */\\n\\nbody {\\n\\tmargin: 0;\\n\\tpadding: 0;\\n\\tpadding-top: 3px;\\n\\tbackground-color: #FFFFFF;\\n\\tfont-family: tahoma,verdana,arial,sans-serif;\\n\\tbackground-repeat: repeat-x;\\n\\tfont-size: 11px;\\n\\tcolor: #444;\\n\\tmin-width: 760px;\\n}\\n\\ntd {\\n\\tfont-family: tahoma,verdana,arial,sans-serif;\\n\\tbackground-repeat: repeat-x;\\n\\tfont-size: 11px;\\n\\tcolor: #444;\\n\\tmin-width: 760px;\\n}\\n\\nh1, h2 {\\n\\tfont-size: 16px;\\n\\tfont-weight: bold;\\n\\tcolor: #ca232a;\\n\\tpadding: 0;\\n\\tmargin: 0;\\n\\tmargin-bottom: 10px;\\n}\\n\\nh3 {\\n\\twidth: inherit;\\n\\tbackground-repeat: no-repeat;\\n\\tborder-right: 1px solid #ccc;\\n\\tfont-size: 10px;\\n\\tfont-weight: bold;\\n\\tcolor: #656565;\\n\\tmargin: 0;\\n\\tpadding-left: 30px;\\n\\tline-height: 20px;\\n\\tdisplay: block;\\n}\\n\\nh4, h5 {\\n\\tfont-size: 12px;\\n\\tfont-weight: bold;\\n\\tcolor: #ca232a;\\n}\\n\\na {\\n    text-decoration: underline;\\n    color: #ca232b;\\n}\\n\\na:hover {\\n\\ttext-decoration: underline;\\n\\tcolor: #000000;\\n}\\n\\t    \\t\\t\\t\\t\\t\\ntable {\\n\\tborder: 1px solid #ccc;\\n}\\n\\n</style>\\n\\n\\n</HEAD>\\n<BODY>\\n\\n<br>\\n<br>\\n<CENTER>\\n\\n<table cellpadding=\"20\" bgcolor=\"#efefef\">\\n<tr>\\n<td align=\"center\">\\n<H1>ERROR</H1>\\n<H2>The requested URL could not be retrieved</H2>\\n<HR noshade size=\"1px\">\\n<P>\\nWhile trying to retrieve the URL:\\n<A HREF=\"http://127.0.0.1:1037/version\"><strong><font color=\"#000000\">http://127.0.0.1:1037/version</font></strong></A>\\n</p>\\n<br>\\nThe following error was encountered:\\n<br>\\n<STRONG>\\nAccess Denied.\\n</STRONG>\\n<br>\\n<P>\\nAccess control configuration prevents your request from\\nbeing allowed at this time.  Please contact your service provider if\\nyou feel this is incorrect.\\n</p>\\n<P>Your cache administrator is <A HREF=\"mailto:webmaster\">webmaster</A>. \\n</td>\\n</tr>\\n</table>\\n\\n\\n<P><font size=-3><a href=\"http://www.endian.it/\">Endian Firewall</a> - Powered by <a href=\"http://www.squid-cache.org/\" target=\"_blank\">Squid</a></font>\\n</center>\\n</BODY>\\n</HTML>'\")"

We suspect that the error is thrown when the moses backend tries execute an API request to the device’s docker instance, in particular a docker version request. We tried logging into the device through ssh and executing an API version request to the default docker API port 2375 from the command line and the request was successful.

curl -X GET http://127.0.0.1:2375/version

Response

{"Platform":{"Name":""},"Components":[{"Name":"Engine","Version":"19.03.14-ce","Details":{"ApiVersion":"1.40","Arch":"arm","BuildTime":"1970-01-01T00:00:00.000000000+00:00","Experimental":"false","GitCommit":"5eb3275d4006e4093807c35b4f7776ecd73b13a7","GoVersion":"go1.14.15","KernelVersion":"5.4.154-5.5.0+git.73d0438130d3","MinAPIVersion":"1.12","Os":"linux"}},{"Name":"containerd","Version":"v1.2.14-3-g3b3e9d5f6.m","Details":{"GitCommit":"3b3e9d5f62a114153829f9fbe2781d27b0a2ddac.m"}},{"Name":"runc","Version":"1.0.0-rc8","Details":{"GitCommit":"425e105d5a03fabd737a126ad93d62a9eeede87f-dirty"}},{"Name":"docker-init","Version":"0.18.0","Details":{"GitCommit":"fec3683-dirty"}}],"Version":"19.03.14-ce","ApiVersion":"1.40","MinAPIVersion":"1.12","GitCommit":"5eb3275d4006e4093807c35b4f7776ecd73b13a7","GoVersion":"go1.14.15","Os":"linux","Arch":"arm","KernelVersion":"5.4.154-5.5.0+git.73d0438130d3","BuildTime":"1970-01-01T00:00:00.000000000+00:00"}

Any help or suggestions would be greatly appreciated.

Colibri iMX6DL 512MB V1.1A
Colibri Evaluation Board V3
TorizonCore Upstream 5.5.0+build.11 (dunfell)

Greetings @mmarcos.sensor,

First of all are you running our VSCode extension on Windows or Linux?

As for the issue itself, well it definitely looks like some kind of firewall or network restriction that’s blocking the API calls being made by the extension backend.

At least it doesn’t seem to be an inherent issue with the extension itself since i can’t seem to reproduce this. Are you sure there isn’t anything on your network that could be causing this?

Also does the extension fail in any other way or is it only when it tries to reach out to the device like this?

Best Regards,
Jeremias

Jeremias,

Thank you for responding.

We are running both VSCode and Visual Studio on Windows.

We can’t see how the firewall would be blocking the API calls between the development machine and the device. Only outgoing traffic gets blocked, we never had internal traffic being blocked before. The device and the PC are connect to the same network switch. We are going to test connecting both PC and device on a isolated router and see if that fixes the issue, although that can’t be a long term solution.

We had problems in the past (most recent the Toradex Easy Installer) with systems trying to connect to connect to insecure sites. Our firewall will drop any outgoing connection on HTTP port 80, only connections to remote port 443 are allowed. This begs the questions: does the device at any moment during the pairing process with the backend try to access remote sites, and is that access over HTTP port 80?

It would really help to know what goes on during that pairing process with the backend. We’ve read the documentation. It mentions enabling the device’s docker TCP/IP interface on 127.0.0.1, we are guessing that this means Docker Engine API, correct? Afterwards an SSH tunnel is created between the remote Docker API and the backend, correct?

We are currently looking at the moses source code for clues as to what could be wrong with our setup. We will post any updates.

If you or anyone has any suggestions/ideas we are open to test them.

Best regards.
Martin.

We figured what the issue was.

First we tried manually setting up a local port forwarding SSH tunnel to the device’s remote docker instance API and sending request to 127.0.0.1:localport, it worked successfully. So that was ruled out as the problem.

Next we tried connecting both the development PC and the device on an isolated router and the IDE extension worked but as soon as we connected the router to the rest of the network it would stop working. This gave us a clue.

We have enforced proxy settings. We can disable/modify them temporally, but shortly after the system reverts the changes and the proxy settings are again enforced. We tried momentarily disabling the proxy settings and the IDE extensions started working. It came as surprise to us that windows would proxy connections directed to the 127.0.0.1 loopback interface. We tried adding 127.0.0.1 to the list of proxy address exceptions and the IDE extensions worked. Of course that fixed the problem for while, soon after proxy settings were reverted and it stopped working.

What seems odd is that when we tested the SSH tunnel manually, we directed requests to 127.0.0.1 from the console and those request were not sent to the proxy. It seems like the docker client python library used on the moses backend uses the system proxy settings.

Is there a way to disable or override proxy settings used by the moses backend?

Best regards,
Martin

Thank you for the feedback I’ll need to pass this along to our IDE extensions team. They’ll see if something can possibly be done here.

Best Regards,
Jeremias

According to our IDE extensions team it seems your intuition is correct. This proxy is something that the Docker Client Python library is doing. Unfortunately there doesn’t appear to be a way to trivially disable/change this behavior. At least not without breaking extension functionality in other ways.

Best Regards,
Jeremias