Serial problem with RS-485

Hello, I’m facing a problem using the Colibri iMx7 serial bus. Everything is explained below. I hope it’s clear enough.

Modbus Issue

System configuration

### Kernel Version
4.9.220
git://git.toradex.com/linux-toradex.git
Colibri-iMX7_LXDE-Image_2.8b7.-20200610
### BSP Version
Kernel built out of the tree

Hardware

Colibir iMX7D 1GB V1.1B
Custom board

mcmon@pmm-300p-22-287 ~
 $ sudo ./tdx-info.sh
Password:

Software summary
------------------------------------------------------------
Bootloader:               U-Boot
Kernel version:           4.9.220 #1 SMP Thu Mar 21 14:40:00 CET 2024
Kernel command line:      clk_ignore_unused root=/dev/mmcblk0p2 ro rootfstype=ext4 rootwait rauc.slot=A
Distro name:              NAME=Mizar
Distro version:           VERSION_ID=V4.1.1
Distro variant:           -
Hostname:                 pmm-300p-22-287
------------------------------------------------------------

Hardware info
------------------------------------------------------------
HW model:                 MC-Monitoring PMM300 PLUS
Toradex version:          0039 V1.1B
Serial number:            14774805
Processor arch:           armv7l

Libraries

Problem

We have 4 devices with Colibri connected to a modbus RTU network managed by 1 modbus master. The master read some random registers on each slave every 5 ms.

Io_ABorGoCdFB-3oJImkIIrIqBLJ2Cv9B4irLB1I24fDJ54eIatF1CaKSq74GIeWQi4g2d8g5SepIpBWwXIIy_DAKlDAq0qqmc6W4Ra66kCmq9XS0qrm66X2ke48vUM9cIKAfWgvnLofUIa0

Suddenly after a certain amount of time, one the slave write some noise during 2 seconds then everything come back to normal.

hP5HIiOm48JVSugvWFpQlo2KqXleWvuWDaiCfjcmMQlUtaIBe52HuclcopS3UsEBak1YmdsqArrXm1rP1o4VMbBCzbw9ZMr4Zczu2VhunM7cf9JK_11sEyBkgF2y8pmV5VOxmlwRS9mvibJ6JRZbK0ZkMKBow52ecbqPJn_8P4v8h0HUIRR_0o9xtsYRi_1hMAo6bUh2wqj

Troubleshooting Attempts

First we analyzed deeply, to notice the library function calls. The noise packets was not correlated with the function calls. We assumed, the problem is not from our code written in go.

We rewrote a master and a slave in C with the libmodbus library. With one slave the problem didn’t happen. Once we connected additional slaves, it happened again.

Now we assume, we have to go deeper towards low-level functionning code.

Configuration

Configuration from the C code:

#ifdef __arm__
  mb = modbus_new_rtu("/dev/ttymxc3", 19200, 'N', 8, 2);
#else
  mb = modbus_new_rtu("/dev/ttyUSB0", 19200, 'N', 8, 2);
#endif
  if (mb == NULL) {
    fprintf(stderr, "modbus: modbus_new_rtu failed\n");
    return -1;
  }

Configuration from the device tree:

/* UART C */
&uart3 {
    pinctrl-names = "default";
    pinctrl-0 = <&pinctrl_uart3>;
    assigned-clocks = <&clks IMX7D_UART3_ROOT_SRC>;
    assigned-clock-parents = <&clks IMX7D_OSC_24M_CLK>;
    fsl,dte-mode;
    linux,rs485-enabled-at-boot-time;
    rs485-rx-during-tx;
    status = "okay";
};

Additional Context

Network topology

ZSk_3e8m4CVncJx5XTs1FqOJOi8pwAWEXJRcOdirlI8cXdUNFmm45xhrTzzFoK7wq5WZg5Ke7_0L0AH3w_mJa8Bs9CquHX5OoVrGz9zkKLcdgeR7PdXIAQ-PuGJn-RBBZ_kuc2yJXwlIN7jy1NGqLDCStminTspaMoT_NUqCLXu1xUTmep62f6TBbMvIwKfbcrIsKlacbK-

Master Node

ZSyzZiD028RXdbDGxfRvIMCNyLM8Z1qaXh409r9E7skK0qm7qlCLxnH9ddjHQ0V0KEhys0F598LKFfHIRGHuFqKP_tmt4rjnKLhl_u0uEDEClvNpAf7iWOlNWg7qOZoDNUpSnovzxDfOAFE6jn8ZiCqj0I2oRZ0T_v7z1G00

#include <errno.h>
#include <modbus.h>
#include <stdio.h>
#include <unistd.h>

int main(void) {
  int return_code;
  int i;
  modbus_t *mb;
  uint16_t tab_reg[32];
  uint8_t tab_bits[32];

  mb = modbus_new_rtu("/dev/ttyUSB0", 19200, 'N', 8, 2);
  if (mb == NULL) {
    fprintf(stderr, "modbus: modbus_new_rtu failed\n");
    return -1;
  }

  return_code = modbus_set_slave(mb, 11);
  if (return_code == -1) {
    fprintf(stderr, "modbus: modbus_set_slave failed: %s\n",
            modbus_strerror(errno));
    return -1;
  }

  return_code = modbus_connect(mb);
  if (return_code == -1) {
    fprintf(stderr, "modbus: modbus_connect failed: %s\n",
            modbus_strerror(errno));
    return -1;
  }

  while (1) {
    return_code = modbus_read_registers(mb, 0, 10, tab_reg);
    if (return_code == -1) {
      fprintf(stderr, "modbus: modbus_read_registers failed: %s\n",
              modbus_strerror(errno));
      continue;
    }

    for (i = 0; i < 10; i++) {
      printf("%d\n", tab_reg[i]);
    }

    usleep(50000);
  }

  modbus_close(mb);
  modbus_free(mb);
}

Slave Node

9OrB3i8m30NNVQVmihq2NT2hLE2q5iQ9yW7bzfXqPOyy6c-bxhcsjs8imFSKPPfpCnCxAEX-t1TCjymFbW-lc2Ju-IAAhuNgoKPpv-BQtqlQtM9xIbZXOnJX_PYmEVkp7m00

#include <errno.h>
#include <modbus.h>
#include <stdio.h>
#include <stdlib.h>

#define BITS_ADDRESS (0)
#define INPUT_BITS_ADDRESS (0)
#define REGISTERS_ADDRESS (0)
#define INPUT_REGISTERS_ADDRESS (0)

#define BITS_NUMBER (65536)
#define INPUT_BITS_NUMBER (65536)
#define REGISTERS_NUMBER (65536)
#define INPUT_REGISTERS_NUMBER (65536)

int running = 1;

void signal_handler(int sig_num) {
  running = 0;
  signal(sig_num, signal_handler);
}

int main(void) {
  // Inspiration from:
  // https://gist.github.com/JGrossholtz/d1d02a87d535d7772139
  int return_code;
  int i;
  uint8_t *request;
  modbus_t *mb;
  modbus_mapping_t *mb_mapping;

  signal(SIGINT, signal_handler);
  signal(SIGTERM, signal_handler);

#ifdef __arm__
  mb = modbus_new_rtu("/dev/ttymxc3", 19200, 'N', 8, 2);
#else
  mb = modbus_new_rtu("/dev/ttyUSB0", 19200, 'N', 8, 2);
#endif
  if (mb == NULL) {
    fprintf(stderr, "modbus: modbus_new_rtu failed\n");
    return -1;
  }

  return_code = modbus_set_debug(mb, FALSE);
  if (return_code == -1) {
    fprintf(stderr, "modbus: modbus_set_debug failed: %s\n",
            modbus_strerror(errno));
    return -1;
  }

  return_code = modbus_set_indication_timeout(mb, 1, 0);
  if (return_code == -1) {
    fprintf(stderr, "modbus: modbus_set_timeout failed: %s\n",
            modbus_strerror(errno));
    return -1;
  }

  request = malloc(MODBUS_RTU_MAX_ADU_LENGTH);
  if(request == NULL) {
    perror("modbus: request memory allocation failed");
    return -1;
  }

  return_code = modbus_set_slave(mb, 11);
  if (return_code == -1) {
    fprintf(stderr, "modbus: modbus_set_slave failed: %s\n",
            modbus_strerror(errno));
    return -1;
  }

  return_code = modbus_connect(mb);
  if (return_code == -1) {
    fprintf(stderr, "modbus: modbus_connect failed: %s\n",
            modbus_strerror(errno));
    return -1;
  }

  mb_mapping = modbus_mapping_new(
      BITS_ADDRESS + BITS_NUMBER, INPUT_BITS_ADDRESS + INPUT_BITS_NUMBER,
      REGISTERS_ADDRESS + REGISTERS_NUMBER,
      INPUT_REGISTERS_ADDRESS + INPUT_REGISTERS_NUMBER);
  if (mb_mapping == NULL) {
    perror("cannot allocate mb_mapping");
    return -1;
  }

  mb_mapping->tab_input_registers[0] = 4;
  mb_mapping->tab_input_registers[1] = 42;
  mb_mapping->tab_input_registers[2] = 421;
  mb_mapping->tab_input_registers[3] = 342;
  mb_mapping->tab_input_registers[4] = 412;
  mb_mapping->tab_input_registers[5] = 429;
  mb_mapping->tab_input_registers[6] = 842;
  mb_mapping->tab_input_registers[7] = 8842;

  while (running) {
    return_code = modbus_receive(mb, request);
    if (return_code == 0) {
      printf("modbus: modbus_receive return_code=0 : %s\n",
             modbus_strerror(errno));
      continue;
    } else if (return_code < 0) {
      if (errno == ETIMEDOUT) {
        fprintf(stderr, "modbus: debug: timeout, continue ...\n");
        continue;
      }
      fprintf(stderr, "modbus: modbus_receive error: %s\n",
              modbus_strerror(errno));
      return -1;
    }
    // printf("Request received return_code=%d\n", return_code);

    return_code = modbus_reply(mb, request, return_code, mb_mapping);
    if (return_code < 0) {
      fprintf(stderr, "modbus: modbus_reply error: %s\n",
              modbus_strerror(errno));
      return -1;
    }
    mb_mapping->tab_input_registers[0]++;
    mb_mapping->tab_input_registers[1]--;
    mb_mapping->tab_input_registers[2]++;
    mb_mapping->tab_input_registers[3]--;
    mb_mapping->tab_input_registers[4]++;
    mb_mapping->tab_input_registers[5]--;
    mb_mapping->tab_input_registers[6]++;
    mb_mapping->tab_input_registers[7]--;
  }

  printf("closing server\n");

  modbus_mapping_free(mb_mapping);

  free(request);

  modbus_close(mb);
  modbus_free(mb);
}

Questions

Is this problem known at Toradex ?
Would a kernel upgrade solve this problem ?
Do you have any other suggestions ?

Hi, I’m the Toradex AI Assistant. I’m in early beta, and I would appreciate your feedback.
The Toradex Support team will follow up with you shortly!

Did you like my answer?

:+1: :-1:


Handshake,

The issue you’re encountering with noise on the RS-485 network when using multiple slaves on a Colibri iMX7D module does not appear to be a widely reported or recognized problem specific to Toradex hardware. However, RS-485 communication can be sensitive to various factors including hardware design, software configuration, and electrical interference, which can cause the symptoms you’re experiencing.

Kernel Upgrade

A kernel upgrade could potentially resolve your issue if it is related to a bug or limitation in the RS-485 driver or related subsystems in the Linux kernel version you are currently using (4.9.220). Newer kernel versions often include bug fixes, improved drivers, and better support for hardware protocols including RS-485. However, upgrading the kernel should be approached with caution, as it may introduce compatibility issues with your current setup or other unforeseen problems.

Suggestions for Troubleshooting and Resolution

  1. Electrical Interference: Ensure that the RS-485 network is properly terminated and that cables are shielded and grounded correctly to minimize electrical noise. Incorrect termination or grounding can lead to the kind of intermittent communication issues you’ve described.

  2. Driver Configuration: Double-check the RS-485 mode configuration in both the device tree and at runtime (using ioctl calls as necessary). Ensure that the settings for linux,rs485-enabled-at-boot-time, rs485-rx-during-tx, and other relevant parameters match the requirements of your network and devices.

  3. Software Libraries: Although you’ve ruled out the high-level Modbus libraries as the source of the noise, it’s worth testing with different versions of these libraries or even different libraries altogether, if possible. There could be subtle bugs or incompatibilities in the library code that only manifest under specific conditions, such as when multiple slaves are present.

  4. Hardware Testing: If feasible, test your setup with a different Colibri iMX7D module or even a different Toradex module family to rule out a hardware-specific issue. Also, consider testing with a different carrier board or a simplified setup to isolate the problem.

  5. Serial Port Configuration: Verify the serial port configuration for baud rate, parity, stop bits, etc., both in your application code and in the system settings. Mismatches in these settings can cause communication errors.

  6. Kernel Debugging: If you have the capability, enabling kernel debugging for the serial and RS-485 drivers may provide additional insights into what’s happening at the lower levels of the

Please, also check if these links can help you solve your problem:

UART (Linux) | Toradex Developer Center.

Hello @arnaud_mc,

Apologies for the delay in answering you! Also, thank you for sending in this detailed information regarding your issue.

We do not have any known issues like yours on our software side regarding modbus. However, it is possible that something was fixed in the later versions of kernel. The kernel version that you are using is really outdated and there have been so many changes since then. You have a Colibri iMX7 version that supports our BSP 6 as well. It would be great if you could give it a try our BSP 6. If you see that the issue does not happen there, then I would recommend you migrate to it.

You can download our OS images here: Download Links | Toradex Developer Center
Otherwise, you will also find this image in the feeds of Toradex Easy Installer if you put your module into recovery.

Hello @rudhi.tx ,

Thank you for your reply.

I followed your advices to try with an official image. As we need to customize the kernel, I started the guide about creating a base image based on BSP version 6. So I followed this documentation: Build a Reference Image with Yocto Project/OpenEmbedded | Toradex Developer Center

I go stuck at the building chapter, when I have to write bitbake.


arnaud in  ivonne in Workspace/oe-core/build took 6s 
 bitbake  tdx-reference-minimal-image
 
WARNING: You have included the meta-tpm layer, but 'tpm or tpm2' has not been enabled in your DISTRO_FEATURES. Some bbappend files and preferred version setting may not take effect. See the meta-tpm README for details on enabling tpm support.
WARNING: Host distribution "debian-12" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
ERROR:  OE-core's config sanity checker detected a potential misconfiguration.
    Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
    Following is the list of potential problems / advisories:

    Your Python 3 is not a full install. Please install the module distutils (see the Getting Started guide for further information).


Summary: There were 2 WARNING messages.
Summary: There was 1 ERROR message, returning a non-zero exit code.


It didn’t work but I followed the requirement form written in this page: 1 System Requirements — The Yocto Project ® 5.0.1 documentation


arnaud in  ivonne in Workspace/oe-core/build                                                                                                                                                                                                
 sudo apt install gawk wget git diffstat unzip texinfo gcc build-essential chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping python3-git python3-jinja2 python3-subunit zstd liblz4-tool file locales
 libacl1                                                                                                                                                                                                                                     
                                                                                                                                                                                                                                             
Reading package lists... Done                                                                                                                                                                                                                
Building dependency tree... Done                                                                                                                                                                                                             
Reading state information... Done                                                                                                                                                                                                            
gawk is already the newest version (1:5.2.1-2).                                                                                                                                                                                              
gawk set to manually installed.                                                                                                                                                                                                              
wget is already the newest version (1.21.3-1+b2).                                                                                                                                                                                            
git is already the newest version (1:2.39.2-1.1).                                                                                                                                                                                            
diffstat is already the newest version (1.65-1).                                                                                                                                                                                             
unzip is already the newest version (6.0-28).                                                                                                                                                                                                
gcc is already the newest version (4:12.2.0-3).                                                                                                                                                                                              
gcc set to manually installed.                                                                                                                                                                                                               
build-essential is already the newest version (12.9).                                                                                                                                                                                        
chrpath is already the newest version (0.16-2+b1).                                                                                                                                                                                           
cpio is already the newest version (2.13+dfsg-7.1).                                                                                                                                                                                          
python3 is already the newest version (3.11.2-1+b1).                                                                                                                                                                                         
python3-pip is already the newest version (23.0.1+dfsg-1).                                                                                                                                                                                   
xz-utils is already the newest version (5.4.1-0.2).                                                                                                                                                                                          
debianutils is already the newest version (5.7-0.5~deb12u1).                                                                                                                                                                                 
iputils-ping is already the newest version (3:20221126-1).                                                                                                                                                                                   
python3-jinja2 is already the newest version (3.1.2-1).                                                                                                                                                                                      
python3-jinja2 set to manually installed.                                                                                                                                                                                                    
zstd is already the newest version (1.5.4+dfsg2-5).                                                                                                                                                                                          
file is already the newest version (1:5.44-3).                                                                                                                                                                                               
locales is already the newest version (2.36-9+deb12u7).                                                                                                                                                                                      
libacl1 is already the newest version (2.3.1-3).                                                                                                                                                                                             
The following additional packages will be installed:     

I tried a seccond time with the same result:


arnaud in  ivonne in Workspace/oe-core/build
 bitbake  tdx-reference-minimal-image
 
WARNING: You have included the meta-tpm layer, but 'tpm or tpm2' has not been enabled in your DISTRO_FEATURES. Some bbappend files and preferred version setting may not take effect. See the meta-tpm README for details on enabling tpm support.
WARNING: Host distribution "debian-12" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
ERROR:  OE-core's config sanity checker detected a potential misconfiguration.
    Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
    Following is the list of potential problems / advisories:

    Your Python 3 is not a full install. Please install the module distutils (see the Getting Started guide for further information).


Summary: There were 2 WARNING messages.
Summary: There was 1 ERROR message, returning a non-zero exit code.


I even checked the installation. Python3-full is installed as well as python3-distutils and python3-distutils-extra.


arnaud in  ivonne in Workspace/oe-core/build 
 apt search python3-full
Sorting... Done
Full Text Search... Done
python3-full/stable,now 3.11.2-1+b1 amd64 [installed]
  Default Python Interpreter with complete class library


arnaud in  ivonne in Workspace/oe-core/build 
 apt search python3-distutils
Sorting... Done
Full Text Search... Done
python3-distutils/stable,now 3.11.2-3 all [installed]
  distutils package for Python 3.x

python3-distutils-extra/stable,now 2.50 all [installed]
  enhancements to the Python3 build system


In addition Debian 12 is supported by Yocto: 1 System Requirements — The Yocto Project ® 5.0.1 documentation

What should I do to solve this ?

Best Rergards,
Arnaud Sautaux

Hello @arnaud_mc,

Could you please check if the python version used by the build system is correct? Sometimes multiple Python versions on the system can cause such issues.
Please check the outputs of:

python3 --version
which python3

Hello @rudhi.tx,

Thank you. I discovered that a python version from homebrew was used intead from the debian package. I just launched the build.

I will let you know as soon as I get a result.

Best regards,
Arnaud Sautaux

Finally, it works !

1 Like

Glad to hear that it works :slight_smile: