ApolloX - docker-compose build error in Node.js project

vix · December 22, 2022, 9:11am

I installed ApolloX on my Ubuntu 22.04 machine (Virtualbox Guest OS) and I followed the documentation here.
I created the project from template TypeScript / Node.js

When I try to debug the project, the task build-container-torizon-debug-arm64 fails with the following error:

*  Executing task: sshpass -p mypassword ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no torizon@192.168.1.99 LOCAL_REGISTRY=10.0.2.15 TAG=arm64 GPU= docker-compose down --remove-orphans 

Warning: Permanently added '192.168.1.99' (ED25519) to the list of known hosts.
time="2022-12-22T10:10:08Z" level=warning msg="The \"DOCKER_LOGIN\" variable is not set. Defaulting to a blank string."
Container torizon-weston-1  Stopping
Container torizon-weston-1  Stopping
Container torizon-kiosk-1  Stopping
Container torizon-kiosk-1  Stopping
Container torizon-nextjs-verdin-1  Stopping
Container torizon-nextjs-verdin-1  Stopping
Container torizon-nextjs-verdin-1  Stopped
Container torizon-nextjs-verdin-1  Removing
Container torizon-nextjs-verdin-1  Removed
Container torizon-weston-1  Stopped
Container torizon-weston-1  Removing
Container torizon-weston-1  Removed
Container torizon-kiosk-1  Stopped
Container torizon-kiosk-1  Removing
Container torizon-kiosk-1  Removed
Network torizon_default  Removing
Network torizon_default  Removed
*  Terminal will be reused by tasks, press any key to close it. 

*  Executing task: docker-compose build --build-arg SSHUSERNAME=torizon --build-arg IMAGE_ARCH=arm64 --build-arg SSH_DEBUG_PORT=2229 apollox-nextjs-debug 

execvp(3) failed.: Permission denied

*  The terminal process "docker-compose 'build', '--build-arg', 'SSHUSERNAME=torizon', '--build-arg', 'IMAGE_ARCH=arm64', '--build-arg', 'SSH_DEBUG_PORT=2229', 'apollox-nextjs-debug'" failed to launch (exit code: 1). 
*  Terminal will be reused by tasks, press any key to close it.

From the command line I can connect to the module in SSH

matheus.tx · December 27, 2022, 1:32am

Hey @vix

can you run docker without sudo? Try on a new terminal:

docker run --rm -it hello-world

The extension expects that your default user is in the Docker group and that it is possible to run docker without sudo. Please follow this Docker Engine post-installation steps | Docker Documentation . We will add a note about this on the documentation (it’s missing).

I had installed here the VBox to try the same environment and I was able to debug the application successfully. But I had to allow the ports 8090 and 5002 on the windows firewall (for inbound and outbound):

These are used on the device to get the Docker image from the local registry (inside the VM) (I’m using bridge network on my VM).

Let me know if this works on your side.

Best regards,

vix · January 2, 2023, 9:48am

I confirm that I can run docker without sudo.
Moreover I created the two rules to allow ports 8090 and 5002 on the windows firewall for my Host Windows 10 machine (Guest VM is Ubuntu 22.04), but I have the same error.

Executing task: docker-compose build --build-arg SSHUSERNAME=torizon --build-arg IMAGE_ARCH=arm64 --build-arg SSH_DEBUG_PORT=2229 apollox-nextjs-debug 

execvp(3) failed.: Permission denied

 *  The terminal process "docker-compose 'build', '--build-arg', 'SSHUSERNAME=torizon', '--build-arg', 'IMAGE_ARCH=arm64', '--build-arg', 'SSH_DEBUG_PORT=2229', 'apollox-nextjs-debug'" failed to launch (exit code: 1). 
 *  Terminal will be reused by tasks, press any key to close it.

It comes to my mind that I can connect to the Verdin in ssh; the username is torizon but the password is a custom one (I had to change at the first connection).
Is this the same for your test environment?

matheus.tx · January 5, 2023, 4:12am

Hi @vix ,

sorry for the delay. Yeah, the password change is mandatory on the first login. Could you also try to run on your VM:

docker-compose --version

Looks like there is something wrong with local docker-compose …

Best regards,

vix · January 5, 2023, 6:50am

Here is what I get with docker-compose --version

docker-compose version 1.29.2, build unknown

I found another issue related to different releases of docker-compose, discussed here. Not sure if this helps.

matheus.tx · January 12, 2023, 2:31am

Hi @vix

sorry for the delay. We have decided not to use docker-compose anymore ( that is now called Compose standalone Install the Compose standalone | Docker Documentation ), since it is no longer the way recommended by Docker. The standard now is the Compos plugin ( Install the Compose plugin | Docker Documentation ). We had changed it also in the templates this last week.

Could you create a new project from a fresh template after install the Compose plugin to test? Maybe that solves the issue, let me know …

Best regards,

vix · January 12, 2023, 2:10pm

Hi @matheus.tx

after I created a new project, now the step copy-docker-compose fails with this error

sshpass -p  scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no /home/ubuntu/newproject/docker-compose.yml torizon@192.168.1.99:~/

henrique.tx · January 17, 2023, 8:02pm

Hi @vix !

We can see that your sshpass command has no actual password after the -p option.

Adding "torizon_psswd": "xablau" to the project’s settings.json might help you (please replace xablau with your actual password).

I am just not sure that this is the “right” way of solving it.

You can of course try

Best regards,

matheus.tx · January 17, 2023, 11:08pm

Hey @vix

sorry for the delay. As @henrique.tx said adding torizon_psswd might do the trick. But that must have been a side effect of one of our break changes that changed the device storage mechanism. Could you please uninstall and reinstall the extension. If the previously registered devices are still listed, I also ask that you to remove and register them again. Once a device is set as default settings.json data is automatically set.

Sorry for the inconvenience.

Best regards,

kevin.tx · February 1, 2023, 1:49pm

Hi @vix ,

Did the suggested method from Matheus work for you?

Let us know.

Best Regards
Kevin

vix · February 1, 2023, 4:24pm

Hi @kevin.tx
sorry but I had to switch to an urget project in the past few days.
I hope I can work on this next week.

vix · February 15, 2023, 10:57am

Hi @henrique.tx , @matheus.tx and @kevin.tx
I can share an update on this topic.

After having uninstalled and reinstalled ApolloX, I remove and registered again my Verdin device.
I see that, while regikstering, the settings.json is automatically filled with host_ip, torizon_ip, torizon_login, torizon_passw, …

Now it’s the task pull-container-torizon-debug-arm64 that fails

Executing task: sshpass -p mypassword ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no torizon@10.1.3.66 LOCAL_REGISTRY=10.0.2.15 TAG=arm64 docker-compose pull node-to-del-debug 

Warning: Permanently added '10.1.3.66' (ED25519) to the list of known hosts.
time="2023-02-15T11:56:21Z" level=warning msg="The \"DOCKER_LOGIN\" variable is not set. Defaulting to a blank string."
node-to-del-debug Pulling 
node-to-del-debug Warning 
WARNING: Some service image(s) must be built from source by running:
    docker compose build %s node-to-del-debug
Error response from daemon: Get "https://10.0.2.15:5002/v2/": dial tcp 10.0.2.15:5002: connect: connection refused

I can say that

10.0.2.15 is the ip of my host
10.1.3.66 is the ip of Verdin.
node-to-del is the name of node.js project I’ve been working on

vix · February 15, 2023, 3:27pm

I think I have one interesting information on why this fails.
I see the error

Error response from daemon: Get "https://10.0.2.15:5002/v2/": dial tcp 10.0.2.15:5002: connect: connection refused

that refers to https protocol.
But if I enter http://10.0.2.15:5002/v2/ (without the secure s) into my browser, I get an answer.
If I enter https I get no answer.

Is it necessary that ApolloX configures the local registry to allow unsecure connections too?
Or is it necessary some other settings to have https working on local registry?

matheus.tx · February 16, 2023, 1:55am

Hey @vix

good catch, yeah, it should add the insecure registries (the development machine IP) on the target dev board /etc/docker/daemon.json, during the device discover and registry.

Could you please report to us the content of this file /etc/docker/daemon.json from your dev board?

Thanks in advance,

vix · February 16, 2023, 9:17am

Today the IPs are different, because my PC has several ethernet interfaces, as shown here

WARNING: Some service image(s) must be built from source by running:
    docker compose build %s node-backend-debug
Error response from daemon: Get "http://10.10.10.79:5002/v2/": dial tcp 10.10.10.79:5002: connect: connection refused

The content of /etc/docker/daemon.json is

{
   "insecure-registries" : ["10.10.10.79:5002"]
}

But the problem I can see is that 10.10.10.79 is one of the IPs of my development machine, but the corresponding ethernet adapter is NOT connected to Verdin.
Today Verdin is connected through DHCP-assigned IP (that today is 10.1.2.50) with mask 255.255.0.0

For this reason I did the folowing changes:

in setings.json of the node.js project on my development machine I changed “host_ip”: “10.10.10.79” to “host_ip”: “10.1.2.50”
on the Verdin I changed the content of /etc/docker/daemon.json to

{
   "insecure-registries" : ["10.1.2.50:5002"]
}

Now the error is a little bit different

Error response from daemon: Get "https://10.1.2.50:5002/v2/": http: server gave HTTP response to HTTPS client

 *  The terminal process "sshpass '-p', 'cemb', 'ssh', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', 'torizon@10.1.3.66', 'LOCAL_REGISTRY=10.1.2.50 TAG=arm64 docker-compose pull node-backend-debug'" terminated with exit code: 18.

I hope that this helps

matheus.tx · February 16, 2023, 3:51pm

ok, I see that you changed the address to 10.1.2.50:5002 in the /etc/docker/daemon.json, but to make effect you need to also run:

sudo systemctl restart docker

This should work, let me know.

vix · February 16, 2023, 4:06pm

It works.

What is the conclusion of this topic?
Is it necessary some change in ApolloX or should I always do some manual chnage to the files?

matheus.tx · February 16, 2023, 5:21pm

Hey @vix,

this can be improved, I will take it to the team, and we’ll think about solutions. We should add some mechanism to check these points before starting to build and deploy, and possible automatically update the data on the target device and settings.json.

I will let you know when we have something published, thanks.

Best Regards,

matheus.tx · August 17, 2023, 3:13pm

Hey @vix

we added a property for the global settings, the apollox.overwriteHostIp, that override the host_ip property from the workspace settings.

So, if for some unknown reason, the system gets the wrong interface, which is set on the host_ip from workspace settings automatically, it can override it with the right one set by the user on global settings. Unfortunately, this is something very specific and dependent on the network settings of the development environment, so we thought it would be better to follow this path instead of something automatic that could still fail.

BR,

henrique.tx · September 5, 2023, 2:03pm

Hi @vix !

Have you tested the new option @matheus.tx commented on above?

Do you have any feedback for us?

Best regards,