Zeroconf issues with TEZI and larger images

Hi,

I’ve noticed some issues when using TEZI with both our custom hardware and the Verdin development board to flash a custom TorizonCore image with preprovisioned containers over ethernet. I’m running nginx in a VM with zeroconf setup. When I connect to my server by entering the IP address in the feeds, everything works fine.

But when I let zeroconf do its thing I run into issues. First of, I have to restart the avahi daemon on the VM before TEZI discovers it, every time. I’m guessing this is an issue with my setup but didn’t look into it as it wasn’t an issue for my current workflow. However, if I install an image this way it always hang on 100% downloading complete. TEZI is still responding but just doesn’t move on to the final steps beyond the download. It does without using avahi. When I then run the curl command that TEZI executes, manually, via the CLI exposed on UART, it tells me it cannot find the server. One theory I had was that after a longer download (this image is just over 2 GB), it ‘forgets’ what the server’s IP address was as it is still using the hostname and somehow mDNS stopped working. I did find some stuff online about disabling IPv6 with the avahi daemon but do not have time to investigate further.

Have you noticed issues when flashing (larger) images using zeroconf?

This was using TEZI 5.7.1 build 13. I also tried build 443 and TEZI 5.4.0 with no improvements.

Kind regards,

Ernest Van Hoecke

Hello @Ernest,

Thanks for reaching out.

I could suspect two possible causes for this issue (although I’m more biased towards the second one):

  1. Your custom image build is not proper
  2. Your zeroconf server setup is not proper

Have you tried installing your custom image already by other means such as from a USB stick or SD card or from the network without zero-conf being used? If the installation was successful with any of these other methods, that would confirm that the image is being built properly.

In order to dig into the cause for the second possibility, I would like to ask you to try out a couple of things:

  1. Please check the tezi log to check exactly where tezi is waiting. The file is located in /var/volatile/tezi.log . If tezi is waiting on a network request, you could run Wireshark or tcpdump to check what requests are failing/tîming out → this could be the case if the caching time is short and the download / install takes more time. This combined with the fact that the avahi seems to stop working after a while could be the cause for the issue.

  2. Avahi (mDNS) works by sending multicast UDP packets over the local network. Using this together with a VM could cause some unexpected interactions, depending on how the VM networking is configured. By default, the network interfaces are configured in “forwarding” mode and this will not work well for this kind of application. The best way to try to make this work would be to leave the VM running (without suspending it) and also configure the VM network interface in bridge mode. This way, the VM’s network interface will also become a part of the local network segment, and the possibility of mDNS working increases.

  3. Check if the TEZI services are announced and visible by running the command avahi-browse --t _tezi._tcp from the server side.

  4. Using python to serve images instead of ngix using the following steps:

sudo firewall-cmd --add-service=http 
cd <images directory> 
# For Python2: 
python2 -m SimpleHTTPServer 8000 
# For Python3 
python3 -m http.server 8000

Please let me know if this helps :slight_smile:

Hi @rudhi.tx,

The image is indeed fine as I’ve successfully flashed it over both USB and Ethernet (by entering the IP manually instead of using zeroconf)>

I don’t have time to repeat the process and retrieve the log right now but if we continue with this I will certainly try your suggestions, thank you!

Kind regards,

Ernest Van Hoecke

Hi @Ernest,

Thanks for the feedback.

Great! That confirms that your custom image is being built properly.

I have another update for you that could help you with quick testing:

We have the images serve feature in TorizonCore Builder that’s meant particularly for scenarios like this. It is designed to simplify the configuration of an HTTP server and the avahi-daemon, also this feature is tested regularly. This helps you serve (via HTTP) TorizonCore images from a directory so those images can be installed in a Toradex device using Toradex Easy Installer. Please find more details here: TorizonCore Builder Tool - Commands Manual | Toradex Developer Center

Please give it a try when you find time to work on this again. We would be looking forward to hearing your feedback.