Dropped UART rx packets on Apalis imx8qm

Symptoms
● Occasionally see rx packets dropping on ttyLP0
● Bytes in the middle of the packet are dropped
● Communication recovers by itself for the next packet
● No pattern for reproducibility identified

Hardware Setup
● Ixora Carrier board - v1.3A
● Apalis imx8qm
● UART peripherals in use - default device tree config:
○ ttyLP0 - UART 3 (according to datasheet), “No” DMA enabled
○ ttyLP2 - UART 4 (according to datasheet), edma2 enabled
○ No CTS/RTS being used or configured
● Snapshots of dts file obtained from live setup attached

Software Setup
● Baud - 921600
● On ttyLP0, the application process is run that receives a large
chunk of data ~8000 bytes every 2.5s-3s
● SW is configured with a batched size of 100 bytes in a single
UART transfer

RCH/Solutions tried
● Tried running the application code on ttyLP2 and don’t see any
errors indicating DMA may be required
● Tried enabling DMA on ttyLP0 using overlays, but that doesn’t seem to show up in the system
● Below is the overlay that was attempted (used edma 2, ch 12/13 as that’s the default I saw after researching online)

Another observation is I see dma controller err on that channel (among others), not sure what that’s related to

Help Required
● If DMA needs to be enabled to resolve the issue (likely is), it would help to get feedback on the overlay and what am I doing wrong
● Is it advisable to compile the overlay on a system that doesn’t
have access to the full kernel source/without the torizon
builder
● If torizon builder needs to be used
○ Is it advisable to use an overlay?
○ Can the full device tree (with DMA enabled) be built instead of an overlay?
● Is there a specific reason DMA is disabled by default on ttyLP0
and if turning it ON may have any other side effects?

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.


Based on your UART packet dropping issue on ttyLP0 with the Apalis iMX8QM, let me address your questions about enabling DMA and using device tree overlays.

Device Tree Overlay Approach

For your DMA configuration issue, you can use device tree overlays to enable DMA on ttyLP0(1). The overlay approach is recommended for runtime modifications without rebuilding the entire device tree(2).

Creating Your Overlay

Based on the documentation, here’s the proper structure for a device tree overlay(1):

/dts-v1/;
/plugin/;

#include <soc-pinfunc.h>

/ {
    compatible = "toradex,apalis-imx8";
};

&node {
    pinctrl-names = "default";
    pinctrl-0 = <&pinctrl_my_pins>;
    status = "okay";
};

(1)

Common Overlay Issues

A critical point from the community experience is avoiding syntax errors in the &iomuxc node structure(3). Make sure your overlay doesn’t include unnecessary subnode wrappers that can cause conflicts with other peripherals(3).

Compilation and Deployment Options

Using TorizonCore Builder (Recommended)

For Torizon OS, you can use TorizonCore Builder for overlay compilation and deployment(1):

$ torizoncore-builder dto apply custom-overlay.dts

(1)

Or use the build command with a tcbuild.yaml file(1):

customization:
  device-tree:
    include-dirs:
      - device-trees/include/
    overlays:
      add:
        - your-dma-overlay.dts

(1)

Manual Compilation

If you need to compile manually, you can use the Device Tree Compiler (dtc)(4):

$ make DTC_FLAGS="-@" freescale/<custom-device-tree>.dtb

(4)

Deployment Strategies

You have two main deployment options(4):

  1. Deploy to Running Device: Copy the compiled overlay to /boot and apply it
  2. Deploy to Static Image: Integrate the overlay into your production image

For development and testing, deploying to a running device is more convenient(4).

Recommendations

  1. Use TorizonCore Builder: It provides integrated compilation and deployment in a single step(1)
  2. Overlay vs Full Device Tree: Overlays are preferable for isolated changes like enabling DMA, as they don’t require rebuilding the entire device tree(2)
  3. Test Incrementally: Start with a minimal overlay that only enables DMA, then expand if needed

The packet dropping at 921600 baud with 8KB transfers every 2.5-3 seconds suggests the UART buffer is being overwhelmed without DMA assistance. Enabling DMA should resolve this issue by offloading the data transfer from the CPU.

Best regards,
Toradex AI Assistant.


Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

The baud rate you’re using is quite high, and if the CPU or scheduler is under load, data may overrun before the UART FIFO can be serviced. Enabling DMA should help mitigate this.

In your device tree overlay, you’re using DMA request lines 13 and 12, which correspond to lpuart2. I assume lpuart0, should use request lines 9 and 8

dmas = <&edma2 9 0 0>,   /* TX */
       <&edma2 8 0 1>;   /* RX */
dma-names = "tx", "rx";

but you should verify this numbers against the i.MX8QM Reference Manual to be certain.

Thank you for the reply Alex!
According to the dtb I decompiled from my live system, I got the following for lpuart2

serial@5a080000 {
reg = <0x5a080000 0x1000>;
interrupts = <0x00 0x15b 0x04>;
clocks = <0x74 0x01 0x74 0x00>;
clock-names = “ipg\0baud”;
assigned-clocks = <0x08 0x3b 0x02>;
assigned-clock-rates = <0x4c4b400>;
power-domains = <0x29 0x3b>;
power-domain-names = “uart”;
dma-names = “tx\0rx”;
dmas = <0x69 0x11 0x00 0x00 0x69 0x10 0x00 0x01>;
status = “okay”;
compatible = “fsl,imx8qm-lpuart\0fsl,imx8qxp-lpuart”;
pinctrl-names = “default”;
pinctrl-0 = <0x75>;
phandle = <0x1e3>;
};

indicating channels 16 and 17 in use, which I could also verify from monitoring the traffic from /proc/interrupts

As per your suggestion, I also tried channels 8 and 9 on dma2 for lpuart0, I still see the same problem

Are you using the correct device tree? Both downstream and mainline device trees assign DMA channels 12 and 13 to LPUART2. Please double-check your device tree deployment process, as well as your compilation and decompilation procedures.

Request #43333 “Issue with communication dropout…” was closed and merged into this request. Last comment in request #43333:

This message originated from outside your organization


Hello Support,

We are currently facing an issue with packet dropouts on our Ixora v1.3A/apalis imx8qm setup with one of the UART peripherals. All the details about the problem symptoms, setup and attempted solutions have been mentioned in the attached slide deck

Looking forward to hearing back from you soon

Thanks and Regards,

Akshit

Attachment(s):
Toradex

Yes, you’re right — I missed the DMA channel assignments defined in imx8qm-ss-dma.dtsi. So your original overlay uses the correct channel numbers.

The overlay to enable DMA on LPUART0 should look like this:

/dts-v1/;
/plugin/;

/ {
    compatible = "toradex,apalis-imx8qm";

    fragment@0 {
        target = <&lpuart0>;
        __overlay__ {
            status = "okay";

            /* Enable DMA support */
            dma-names = "tx", "rx";
            dmas = <&edma2 13 0 0>,   /* TX */
                   <&edma2 12 0 1>;   /* RX */

            power-domain-names = "uart";
        };
    };
};

Hi Alex,

Thanks for the suggestion. We tried the overlay above but we’re not able to identify if overlay was applied successfully or not. Here is what we observe

torizon@apalis-imx8-15610250:/proc/device-tree/bus@5a000000/dma-controller@5a1f0000$ cat interrupt-names 
edma0-chan0-rxedma0-chan1-txedma0-chan2-rxedma0-chan3-txedma0-chan4-rxedma0-chan5-txedma0-chan6-rxedma0-chan7-txedma0-chan12-rxedma0-chan13-txedma0-chan14-rxedma0-chan15-txedma0-chan16-rxedma0-chan17-txedma0-chan18-rxedma0-chan19-txedma0-chan20-rxedma0-chan21-txedma0-err
torizon@apalis-imx8-15610250:/proc/device-tree/bus@5a000000/dma-controller@5a1f0000$ hexdump interrupts 
0000000 0000 0000 0000 a001 0000 0400 0000 0000
0000010 0000 a101 0000 0400 0000 0000 0000 a201
0000020 0000 0400 0000 0000 0000 a301 0000 0400
0000030 0000 0000 0000 a401 0000 0400 0000 0000
0000040 0000 a501 0000 0400 0000 0000 0000 a601
0000050 0000 0400 0000 0000 0000 a701 0000 0400
0000060 0000 0000 0000 b201 0000 0400 0000 0000
0000070 0000 b301 0000 0400 0000 0000 0000 b401
0000080 0000 0400 0000 0000 0000 b501 0000 0400
0000090 0000 0000 0000 b601 0000 0400 0000 0000
00000a0 0000 b701 0000 0400 0000 0000 0000 b801
00000b0 0000 0400 0000 0000 0000 b901 0000 0400
00000c0 0000 0000 0000 ba01 0000 0400 0000 0000
00000d0 0000 bb01 0000 0400 0000 0000 0000 4300
00000e0 0000 0400                              
00000e4
torizon@apalis-imx8-15610250:/proc/device-tree/bus@5a000000/serial@5a060000$ cat dma-names 
txrx
torizon@apalis-imx8-15610250:/proc/device-tree/bus@5a000000/serial@5a060000$ hexdump dmas
0000000 0000 6800 0000 0d00 0000 0000 0000 0000
0000010 0000 6800 0000 0c00 0000 0000 0000 0100
0000020

The above indicates the dma is enabled on channels 12 and 13 from both controllers and uarts pov

However when we look under /proc/interrupts it doesn’t seem to be enabled

torizon@apalis-imx8-15610250:/proc/device-tree/bus@5a000000/dma-controller@5a1f0000$ cat /proc/interrupts | grep edma
 22:          0          0          0          0          0          0     GICv3 489 Level     edma2-chan8-rx
 23:          0          0          0          0          0          0     GICv3 491 Level     edma2-chan9-tx
 24:          0          0          0          0          0          0     GICv3 493 Level     edma2-chan10-rx
 25:          0          0          0          0          0          0     GICv3 495 Level     edma2-chan11-tx
 27:          0          0          0          0          0          0     GICv3 349 Level     edma2-chan15-tx, edma2-chan14-rx
 31:          0          0          0          0          0          0     GICv3 360 Level     edma2-chan19-tx
 42:          0          0          0          0          0          0     GICv3 448 Level     edma0-chan0-rx
 43:          0          0          0          0          0          0     GICv3 449 Level     edma0-chan1-tx
 46:          0          0          0          0          0          0     GICv3 452 Level     edma0-chan4-rx
 47:          0          0          0          0          0          0     GICv3 453 Level     edma0-chan5-tx
 52:          0          0          0          0          0          0     GICv3 468 Level     edma0-chan14-rx
 53:          0          0          0          0          0          0     GICv3 469 Level     edma0-chan15-tx

We tried this using both

  1. Just dtc utility, compile dtbo, place it under /boot/overlays/, updated the overlays.txt with the name of the dtbo and reboot
  2. Torizon builder to build the full image with the overlay change

None seem to work

Thanks
Akshit

Please refer to this article.

Thanks Alex, from the article you shared, looks like its not enabled

torizon@apalis-imx8-15610250:~$ cat /proc/device-tree/chosen/overlays/lpuart0-dma-overlay.dtbo 
0

Please check if your overlay file name is included to the overlay.txt

Hi Alex,

Yes we verified our overlay file name is included in overlay.txt
Also verified that the overlay file is included under overlays/ folder along with other existing overlays

Could you please collect the full boot log from the debug UART and share it here as a plain text file for further analysis?

Not sure why I’m getting this error when trying to upload. Is it ok if grep for uart and dma from dmesg + journalctl logs and share here?

dmesg logs

[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x0000000090400000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node vdevbuffer@90400000, compatible id shared-dma-pool
[    0.000000]   DMA      [mem 0x0000000080200000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000] On node 0, zone DMA: 512 pages in unavailable ranges
[    0.062430] DMA: preallocated 1024 KiB GFP_KERNEL pool for atomic allocations
[    0.062793] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.063297] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.064336] Serial: AMBA PL011 UART driver
[    0.132704] iommu: DMA domain TLB invalidation policy: strict mode
[    0.170042] /system-controller/pinctrl: Fixed dependency cycle(s) with /system-controller/pinctrl/uart24forceoffgrp
[    0.170195] /system-controller/pinctrl: Fixed dependency cycle(s) with /system-controller/pinctrl/lpuart1ctrlgrp
[    1.106285] mxs-dma 5b810000.dma-apbh: initialized
[    1.157572] 5a060000.serial: ttyLP0 at MMIO 0x5a060010 (irq = 61, base_baud = 5000000) is a FSL_LPUART
[    1.158635] 5a070000.serial: ttyLP1 at MMIO 0x5a070010 (irq = 62, base_baud = 5000000) is a FSL_LPUART
[    1.158654] fsl-lpuart 5a070000.serial: Serial: Console lpuart rounded baud ratefrom 35971 to 115200
[    1.159900] 5a080000.serial: ttyLP2 at MMIO 0x5a080010 (irq = 63, base_baud = 5000000) is a FSL_LPUART
[    1.160895] 5a090000.serial: ttyLP3 at MMIO 0x5a090010 (irq = 64, base_baud = 5000000) is a FSL_LPUART
[    1.322087] mmc0: SDHCI controller on 5b010000.mmc [5b010000.mmc] using ADMA
[    1.435047] mmc1: SDHCI controller on 5b020000.mmc [5b020000.mmc] using ADMA
[    7.254263] ata1: SATA max UDMA/133 mmio [mem 0x5f020000-0x5f02ffff] port 0x100 irq 586
[    7.505274] imx6q-pcie 5f010000.pcie: eDMA: unroll F, 1 wr, 1 rd
[    7.575892] imx6q-pcie 5f000000.pcie: eDMA: unroll F, 1 wr, 1 rd

Device tree overlays are applied during the U-Boot stage of the boot process, well before the kernel is initialized. As such, overlay-related changes are not reflected in the output of dmesg. You can use our share server to upload your log,
Please remember to save and share the link generated after the upload completes

Got it, confirming UART 1 is the right one for default console logging?

Hi Alex, I’m trying to listen to the UART through my computer, but the content I’m seeing is gibberish, even though I matched my UART settings (115200 baud, 8 bit, no parity, 1 stop bit). here’s my serial terminal (several runs, consistently the same output):

Am I misconfiguring the UART settings?

The port settings appear to be correct. Could there be an issue with your carrier board, or perhaps with how it’s connected to your PC?
After power on you should see something like this: