Docker gRPC Networking Problem

Hi,

I have two C#/.NET apps communicating using gRPC. In Portainer, I have created a user defined bridge network called “doms”. In VS Code I have used “extraparms” to configure the apps to use this network as follows:

In the gRPC client app:

image

And in the gRPC server app:

image

Note the “hostname” specification.

If I create and use the client gRPC channel using the server container’s IP address, it all works fine. But if I use the server’s hostname of “controller-executive” (as defined above), the channel creation fails with the following exception:

Exception has occurred: CLR/Grpc.Core.RpcException

An exception of type 'Grpc.Core.RpcException' occurred in System.Private.CoreLib.dll but was not handled in user code: 'Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: Name or service not known (controller-executive:6000) SocketException: Name or service not known"

I have also tried pinging the server container from the client container. Again, pinging the IP address works, but pinging the hostname does not.

I can’t rely on using an IP address that may change, so how can I address the server by name?

Thanks for your help,

Peter

Greetings @bcpberry,

But if I use the server’s hostname of “controller-executive” (as defined above), the channel creation fails with the following exception:

It appears setting the hostname for a container does not actually provide any network resolution to that hostname. Meaning in terms of providing a hostname to connect with, this option isn’t helpful. There’s various forum posts about this behavior with more detail like this: When to use --hostname in docker? - Stack Overflow

As for reliably connecting to another container on the same network. The docker documentation here: Use bridge networks | Docker Documentation

States the following:

On a user-defined bridge network, containers can resolve each other by name or alias.

So I did a test for this:

# Creating a network first
$ docker network create test
$ docker run --rm -it -d --network test --name test1 torizon/debian-shell:2 /bin/bash
$ docker run --rm -it -d --network test --name test2 torizon/debian-shell:2 /bin/bash
# Get a shell inside one of the containers
$ docker exec -it e /bin/bash
# Commands now are being run inside one of the containers
$ ping test1
PING test1 (172.19.0.2): 56 data bytes
64 bytes from 172.19.0.2: icmp_seq=0 ttl=64 time=0.593 ms
64 bytes from 172.19.0.2: icmp_seq=1 ttl=64 time=0.291 ms
64 bytes from 172.19.0.2: icmp_seq=2 ttl=64 time=0.206 ms
64 bytes from 172.19.0.2: icmp_seq=3 ttl=64 time=0.211 ms
^C--- test1 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.206/0.325/0.593/0.158 ms
$ ping test3
ping: unknown host

Therefore network contact via container name or ID seems to work as long as they’re on the same bridged network.

Best Regards,
Jeremias

Hi @jeremias.tx ,

Thank you for that comprehensive and detailed response – much appreciated.

I have read the links you provided. The Docker docs link I had read previously. The Stackoverflow link confirms similar things I had read elsewhere – there seems to much confusion about the use of “hostname”.

The test that you did makes total sense to me - however I have some problems with using that manual method, as I describe below.

Some follow-up points:

  • We are still in the early concept testing phase of this project which has many aspects to it. As such, we are in rapid code/build/debug cycles. This means that the one key press (F5) launch process is very important for both apps. As such, what ever configuration is required needs to be inherent in the automated build/deploy sequence.

  • I tried setting the name of the container in the VS Code extension as follows:

image

However, that did not work – the following line appears in the backend process log:

INFO:root:running startup script...

INFO:root:Starting new instance...

WARNING:root:Can't use a different name for container.

So, I cannot see a way of setting the name of the container, as you did in your test. Again, I really need to do this in an automated way in the build/deploy step in VS Code.

  • Once the container is running, I can see in Portainer that it has the standard very long name that Torizon uses (i.e. confirming that my attempt to rename it did not work). Although this is not a name that I think is useful in resolving this issue, I tried to ping it from the other app (as per your test). That did not work – here is what I got:
root@2eedf4e6208d:/# ping s36072.controllerexecutive_arm64v8-dotnet-60_bullseye_debug_1221dcee-3663-4326-8744-77d78904c16f_latest_instance

ping: s36072.controllerexecutive_arm64v8-dotnet-60_bullseye_debug_1221dcee-3663-4326-8744-77d78904c16f_latest_instance: Name or service not known
  • Your linked articles, and your reply, mentions an “ID”, which I see as a long hex number in Portainer. Again, I don’t think this is the way to go, but I tried pinging that, and it also did not work.

  • There is much talk in the articles (and what I have read elsewhere) of using an “alias”. This may be the way to go, but I cannot find any way of configuring this in the Torizon VS Code extension.

Just in case you were wondering, I have confirmed that both the containers are on the same network, and I can ping one from the other using the IP address.

Any other ideas would be very welcome.

Best regards,

Peter

This is strange I was able to reproduce the behavior from the extension as well.

After further testing there seems to be a weird limit on container name. For some reason if the name of a container is too long then you can’t ping it from another container on the same network.

I was able to reproduce this outside of the extension as well. I created a container with a long name (a long string of "a"s) and suddenly I couldn’t ping it by name from another container. Now this is a problem since as you noticed the container name generated from the extension can’t be changed.

I looked at alternative ways like setting a container’s IP with --ip or providing a hostname alias via --network-alias. These work if I’m manually starting containers from the command-line, but I can’t seem to find the equivalent way to invoke these options from the VSCode extension itself.

Let me ask our IDE extensions team if they have any ideas or alternatives.

Best Regards,
Jeremias

Hi @jeremias.tx

Thanks for verifying that. I’m glad it was not because I was doing something stupid.

I have to say that I am quite surprised by our experience with this. One of the main reasons we went for this Docker/Torizon platform was because it embraces the “microservices” paradigm – something we intend expanding on beyond these two initial applications. And yet, the simple thing of connecting two containers in a flexible deterministic way seems to be very difficult – both from our own experience and from reading multiple forum posts. Surely we are not the first people using Torizon that has tried to do this? How do you guys, or your other customers, do this?

I am looking forward to hearing what your IDE extensions team has to say about this. I really hope they come up with a workable solution, because at the moment I am not sure what to do about this.

Thanks again for your time,

Peter

After speaking with our IDE extensions team internally they suggested the following. They believe it’d be better if you tried our extensions V2. Our extensions V2 is still in an early phase but is available to try out. I’m told it should have greater flexibility and better support for developing multiple containers at a time which is what you’re doing.

There’s a page here to get you setup with our extensions V2: Torizon VS Code Extension v2 (ApolloX)

Once that’s setup you can start a multi-container project which creates a docker-compose file to define both containers. Which by default puts both containers on the same network. You are more easily able to customize the names of each container as well. Which should get around that issue we had earlier where the V1 extension was creating container names that were too long to ping.

Give this a try and let me know how it works out for you.

Best Regards,
Jeremias

Hi @jeremias.tx ,

Thanks for that feedback.

Some follow-up points:

  • I’ve managed to temporarily work around this issue by deploying the app and then renaming it using Portainer. Obviously, this is not a workable solution for production but for now it is enabling us to move ahead.

  • We recognise that ultimately a Docker Compose controlled deployment will be required (we also are using a custom device tree which needs to be managed). At the moment though we have not got into the details of this.

  • Thanks for pointing me to your new “V2” environment. However, we are about to go into a test phase with this project and so I am freezing development for a while – so at the moment I am reluctant to move to something new, especially as it is still “alpha”. However, it’s very good to know that things are progressing, and when we get back into the next development phase of the cycle, I will give this a close look. Can you give me some idea of the projected beta and final release times for this?

  • Lastly, a request. I have made this request before and was told that it was “unlikely”. However, I do believe it is valid, and have seen other users in your forum make the same request. Please can you enable this .NET development in Visual Studio (i.e., the full featured IDE, not VS Code). VS Code is great, but when it comes to C#/.NET development, it is way inferior to the classic Visual Studio. With this project we are working on now, all the work is done in Visual Studio, and we only swap to VS Code to do the debugging (because we have no other choice). In our Visual Studio “solution”, we have multiple cross-dependent “projects” (including these two Torizon executables, other web servers, a mobile executable, DLLs, and others to come). Managing these in Visual Studio is easy, in “Code” it is certainly not easy. I know that developing extensions for “code” is easier than for the “classic” version, but I really do think that the “classic” version offers a far superior professional experience. Anyway, I just wanted to throw that in as one lonely vote for this.

Thanks again for your time with this.

All the best,

Peter

Can you give me some idea of the projected beta and final release times for this?

Unfortunately I don’t have a more clear timeline to give you other than “soon”. Though it is our plan to eventually replace the current extension with this new one. I would keep an eye on the page I linked for updates to the release state of this V2 extension.

I have made this request before and was told that it was “unlikely”. However, I do believe it is valid, and have seen other users in your forum make the same request. Please can you enable this .NET development in Visual Studio (i.e., the full featured IDE, not VS Code)

If I were to be honest with you I’d still say this is unlikely given the resources of our extensions team. Also this V2 extensions that I linked will only be for VSCode and not Visual Studio as it currently stands. Though your request is still taken into consideration if we ever have the chance to re-evaluate things.

Best Regards,
Jeremias

Thanks Jeremias - all good.

We are starting with some preliminary tests today and I expect that will go on for a while. Once that is done we will be back into a hardware/software development phase and I will have a look at “V2” then.

All the best,
Peter