What is the best method for managing networks with TorizonOS and Docker containers?

dleberfinger · June 26, 2024, 1:00pm

I’m in the process of porting a Python application from a custom Yocto project built on a Colibri module to a Docker container to run on Torizon OS on an AM62 Verdin module. One of the items that I am struggling with is getting my networking set up properly. On the Colibri board we were able to access nmcli using subprocess.run to handle all of the networking. After reading over a lot of different articles and threads I haven’t been able to come up with what would be the best method here.

My use case provides users with a web interface through a flask server. From the web interface, the user needs to be able to adjust the ethernet IP of the board as well as scan for and connect to different wifi networks.

I’ve seen some threads that discuss using NMCLI or straight DBUS and others that suggest not doing this from the container, but creating a host application that would communicate to the containerized application to handle the changes.

Is there a best practice method here?

I’ve been trying to use the NMCLI route since that is what my code is already set up to work with, but I am struggling with this. I have NMCLI and network-manager in my container, but I cannot seem to get network-manager to start when the container is started, I have to manually start it.

This is my docker-compose service:

build:
      context: .
      dockerfile: Dockerfile.debug
    image: ${LOCAL_REGISTRY}:5002/myapp-debug:${TAG}
    network_mode: "host"
    ports:
      - 6502:6502
      - 6512:6512
      - 5000:5000
      - 502:502
    environment:
      - APP_ROOT=/home/torizon/app
    cap_add:
      - NET_ADMIN
    volumes:
       - appdata:${APP_ROOT}/src/data
       - /etc/NetworkManager:/etc/NetworkManager

When I start bash in my container I have run cmd NetworkManager to get nmcli to work.

root@verdin-am62-15207040:/# nmcli
Error: NetworkManager is not running.
root@verdin-am62-15207040:/# NetworkManager

I have tried starting NetworkManager through my entrypoint.sh using this:

if ! pgrep -x "NetworkManager"; then 
    NetworkManager
fi

Then I get this error:

root@verdin-am62-15207040:/# nmcli
Error: Could not create NMClient object: Could not connect: Connection refused.
root@verdin-am62-15207040:/# NetworkManager
root@verdin-am62-15207040:/# nmcli
Error: Could not create NMClient object: Could not connect: Connection refused.

Any help here would be greatly appreciated. If there is a best practice for handling this, please let me know.

ToradexAI · June 26, 2024, 1:00pm

Hi, I’m the Toradex AI Assistant. I’m in early beta, and I would appreciate your feedback.
The Toradex Support team will follow up with you shortly!

Did you like my answer?

Hello,

For managing networks within Docker containers on Torizon OS, especially when dealing with dynamic network configurations like changing Ethernet IP addresses or managing WiFi connections, the best practice is to interact with the host’s network stack indirectly rather than directly managing network interfaces from within containers. This approach ensures isolation and security while maintaining flexibility in network management.

Given your use case with a Python application and a Flask server, here are some recommendations:

Leverage the Host’s NetworkManager: Since Torizon OS uses NetworkManager for network configuration, it’s advisable to keep network management responsibilities on the host system rather than within the container. This approach simplifies network management and leverages the existing infrastructure provided by Torizon OS.
Create a Host Application for Network Management: Develop a small application or service that runs on the Torizon OS host, which exposes an API (e.g., REST API) that your Docker containerized application can call. This service would use nmcli or D-Bus to configure NetworkManager based on the requests received from the container.
Communicate Between Container and Host Application: From your containerized Flask application, make HTTP requests to the host application whenever you need to change network settings. This method abstracts the network management logic away from your application, allowing you to focus on the application logic itself.
Security and Authentication: Ensure that the communication between your containerized application and the host network management service is secure. Implement authentication mechanisms to prevent unauthorized network configuration changes.
Container Configuration: Your current Docker container configuration with network_mode: "host" and the necessary capabilities (cap_add: - NET_ADMIN) is generally correct for scenarios where the container needs to perform network operations that would typically require elevated privileges. However, since you’ll be moving network management tasks to a host application, you might not need these capabilities, depending on how your application and the host service communicate.
Troubleshooting NetworkManager in Containers: The issues you’re encountering with starting NetworkManager inside a container stem from the fact that containers are designed to run isolated processes. Running system services like NetworkManager inside containers is not recommended and can lead to various issues, including the ones you’ve experienced. By moving network management to the host, you avoid these complications.

By adopting this architecture, you not only adhere to best practices for containerized applications but also simplify the process of managing network configurations in a secure and scalable way.

Best regards, Toradex AI Assistant

Please, also check if these links can help you solve your problem:

Networking with Torizon OS | Toradex Developer Center.

dleberfinger · June 26, 2024, 1:03pm

If I follow the suggestion made by the ToradexAI, how do I build that application and create a production image using it? If I am going to go to the extent of building a special application on the host to manage the network settings, it feels like I’m better off forgoing the docker container all together and running my application directly on the host as I did when I was developing the on the colibri board without docker.

jeremias.tx · June 26, 2024, 7:55pm

Greetings @dleberfinger,

Managing network inside of a container has a lot of varied approaches from what I’ve seen. I don’t think there’s single best practice solution I can prescribe for every situation. It’s typically a use-case dependent kind of approach from what I’ve found with other customers.

Now if you want to try and use nmcli inside a container, this has worked for me in the past: Change the apalis imx6 static IP address from inside a container

The main thing I see here compared to your setup is needing to bind-mount /var/run/dbus into your container. This should allow the nmcli tool inside the container to communicate with the NetworkManager that is running outside of the container. This should get basic nmcli working.

You may also need to give your container extra privileges or permissions depending on the kind of network tasks you end up doing. As I said, it all depends.

Best Regards,
Jeremias

dleberfinger · June 27, 2024, 12:02pm

@jeremias.tx

That thread you referenced is one of the ones I had read through while trying to tackle this issue myself and I actually did have the bind-mount you mentioned in my docker compose already, guess I just didn’t copy down far enough.

I still am struggling to access NMCLI through the container. With docker-compose as shown below I am getting NetworkManager not running errors. If I manually start NetworkManager using the NetworkManager cmd I get NMCLI. I cannot seem to get NetworkManager to start by default.

build:
      context: .
      dockerfile: Dockerfile.debug
    image: ${LOCAL_REGISTRY}:5002/remotemonitor-debug:${TAG}
    network_mode: "host"
    ports:
      - 6502:6502
      - 6512:6512
      - 5000:5000
      - 502:502
    environment:
      - APP_ROOT=/home/torizon/app
    cap_add:
      - NET_ADMIN
    volumes:
       - appdata:${APP_ROOT}/src/data
       - /etc/NetworkManager:/etc/NetworkManager
       - /var/run/dbus:/var/run/dbus

root@verdin-am62-15207040:/# nmcli
Error: NetworkManager is not running.
root@verdin-am62-15207040:/# NetworkManager
root@verdin-am62-15207040:/# nmcli -t
ethernet0: connected to network0
        "ethernet0"
        ethernet (am65-cpsw-nuss), 00:14:2D:E8:0A:80, hw, mtu 1500
        ip4 default
        inet4 10.10.10.69/24
        route4 default via 10.10.10.2 metric 100
        route4 10.10.10.0/24 metric 100
        route4 default via 10.10.10.2 metric 100
        inet6 fe80::ea53:2ce0:3ba4:1a1d/64
        route6 fe80::/64 metric 1024

As I mentioned in my initial post, I have tried getting NetworkManager to start by default through my entrypoint.sh, but I am not having any luck with that. What am I missing here that is preventing my Docker container from seeing NetworkManager running on start?

I am able to get NetworkManager running by using this CMD at the end of the Dockerfile, however the Torizon IDE is no longer able to connect to the board for debugging without the sshd command and it appears as those my Python script is not launching with the new CMD line shown.

# CMD [ "/usr/sbin/sshd", "-D" ]
CMD ["/usr/sbin/NetworkManager", "--no-daemon"]

jeremias.tx · June 27, 2024, 8:26pm

Okay so I did a quick test here to demonstrate how to get this working. I created a simple minimal docker-compose.yml so that no other factors are involved in the test. Here’s what I have:

version: "3.9"
services:
  test:
    image: torizon/debian:3-bookworm
    command: tail -F anything
    # Required to get udev events from host udevd via netlink
    network_mode: host
    cap_add:
      - NET_ADMIN
    volumes:
       - /etc/NetworkManager:/etc/NetworkManager
       - /var/run/dbus:/var/run/dbus

This will start a simple Debian-based container with network mode set to host and some volumes and network related permissions applied. Please ignore the command: tail -F anything line. This line is just so my container runs indefinitely to make things easier to test, otherwise it’s not required for anything NetworkManager related.

With this configuration, I start the container and install network-manager inside this container. I run nmcli without starting NetworkManager inside the container. With just this I can see all the network interfaces on the system:

root@verdin-am62-15230292:/# nmcli
ethernet0: connected to network0
        "ethernet0"
        ethernet (am65-cpsw-nuss), 00:14:2D:E8:65:54, hw, mtu 1500
        ip4 default
        inet4 192.168.1.26/24
        route4 192.168.1.0/24 metric 100
        route4 default via 192.168.1.1 metric 100
        inet6 fe80::38bc:3c26:23e8:5e07/64
        route6 fe80::/64 metric 1024

mlan0: disconnected
        "mlan0"
        wifi (mwifiex_sdio), 66:3E:73:C0:91:89, hw, mtu 1500

uap0: disconnected
        "uap0"
        wifi (mwifiex_sdio), CE:BA:9E:A9:27:53, hw, mtu 1500

ethernet1: unavailable
        "ethernet1"
        ethernet (am65-cpsw-nuss), 00:14:2D:F8:65:54, hw, mtu 1500

br-51300ec79660: unmanaged
        "br-51300ec79660"
        bridge, 02:42:96:E1:F8:26, sw, mtu 1500

docker0: unmanaged
        "docker0"
        bridge, 02:42:FD:A2:20:0D, sw, mtu 1500

can0: unmanaged
        "can0"
        can (m_can_platform), hw, mtu 16

can1: unmanaged
        "can1"
        can (m_can_platform), hw, mtu 16

sit0: unmanaged
        "sit0"
        iptunnel (sit), 00:00:00:00, sw, mtu 1480

lo: unmanaged

So no you don’t need to start NetworkManager inside the container itself. Via the container volumes provided the nmcli tool inside the container should be able to access the instance of NetworkManager that is running outside of the container. With this you can keep your entrypoint/CMD as the usual and not need to invoke NetworkManager.

Best Regards,
Jeremias