Colibri imx6ULL: UART ports dropping random rx bytes on new BSP build

Technical Support
, , , ,
1

Hi,

I am currently working on making a new Kernel build for our boards that use a Colibri iMX6ULL module, upgrading from 4.9.166 to 6.1.76. I have a very strange issue with the new build where multiple uart ports will transmit data fine, but will always drop bytes when a large number come in at a time. This has been confirmed on UART6 which talks to a modem, and also UART1 which is used for the normal serial terminal (I haven’t tested any other UARTs). I can replicate it on the serial terminal easily by pasting some text into vi and always some bytes are dropped randomly:

vi_snip
vi_snip926×487 7.64 KB

Note on the above image the left section is from the new kernel build, and the right section is from the same test on the same Colibri module and same carrier board after re-flashing with the old kernel image.

Is there any setting in the device tree or kernel that could explain this issue?

Relevant section of device tree:

&uart6 {
	pinctrl-names = "default";
	status = "okay";
	pinctrl-0 = <&pinctrl_uart6>;
	fsl,uart-has-rtscts;
	/* not dte fsl,dte-mode; */
};

/* ... */

pinctrl_uart6: uart6grp {
		fsl,pins = <
			MX6UL_PAD_CSI_MCLK__UART6_DCE_TX	0x1b0b1		/* sodimm 75 */
			MX6UL_PAD_CSI_PIXCLK__UART6_DCE_RX	0x1b0b1		/* sodimm 96 */
			MX6UL_PAD_ENET1_TX_DATA1__UART6_DCE_CTS		0x1b0b1		/* CTS sodimm 73 */
			MX6UL_PAD_ENET1_TX_EN__UART6_DCE_RTS	0x1b0b1		/* RTS sodimm 30 */
		>;
	};

/* ... */

&uart1 {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_uart1>;
	fsl,dte-mode;
};

/* ... */

pinctrl_uart1: uart1grp {
	fsl,pins = <
		MX6UL_PAD_UART1_TX_DATA__UART1_DTE_RX	0x1b0b1	/* SODIMM 33 */
		MX6UL_PAD_UART1_RX_DATA__UART1_DTE_TX	0x1b0b1	/* SODIMM 35 */
	>;
};

Output of tdx-info:

Software summary
------------------------------------------------------------
Bootloader:               U-Boot
Kernel version:           6.1.76-6.6.0-devel+git.e5c3b988b827 #1 SMP Thu Feb  1 00:17:12 UTC 2024
Kernel command line:      user_debug=30 ubi.mtd=ubi root=ubi0:rootfs rw rootfstype=ubifs ubi.fm_autoconvert=1 console=tty1 console=ttymxc0,115200n8 consoleblank=0
Distro name:              NAME="TDX Wayland with XWayland Upstream"
Distro version:           VERSION_ID=6.6.0-devel-20240221165153-build.0
Distro variant:           -
Hostname:                 colibri-imx6
------------------------------------------------------------

Hardware info
------------------------------------------------------------
HW model:                 Toradex Colibri iMX6ULL 512MB on Colibri Evaluation Board V3
Toradex version:          0040 V1.1A
Serial number:            06822734
Processor arch:           armv7l
------------------------------------------------------------
2

Try if echo 1 > /sys/devices/system/cpu/cpu0/cpuidle/state2/disable helps.
I had some issues with timing critical latency on serial port on imx6ull, disabling sleep helped.

3

Hi kokkonenfi,

This solved my issue, thanks so much!

4

I believe it just masked the problem for a while. Make sure you have DMA enabled for your UART. To verify cat /proc/interrupts and check that interrupt events count is 0 for UART in question. It is not 0 when DMA is disabled

193: 0 GPC 27 Level 21e8000.serial

1 Like
5

It would appear that dma is disabled:

root@colibri-imx6:~# cat /proc/interrupts | grep serial
199:       1732       GPC  39 Level     2018000.serial
200:        482       GPC  26 Level     2020000.serial
201:         26       GPC  27 Level     21e8000.serial
202:          0       GPC  30 Level     21f4000.serial
203:        101       GPC  17 Level     21fc000.serial
204:          0       GPC  40 Level     2288000.serial

How is DMA enabled / disabled for uart? Looking at the kernel config file and comparing to my old kernel there are not any obvious DMA setting differences.

6

Perhaps DMA is disabled in DT. You may further check it looking for serial ports in /sys/firmware,

# find /sys/firmware/ -name "*serial@*"
/sys/firmware/devicetree/base/soc/bus@2200000/serial@2288000
/sys/firmware/devicetree/base/soc/bus@2100000/serial@21ec000
/sys/firmware/devicetree/base/soc/bus@2100000/serial@21f4000
/sys/firmware/devicetree/base/soc/bus@2100000/serial@21fc000
/sys/firmware/devicetree/base/soc/bus@2100000/serial@21f0000
/sys/firmware/devicetree/base/soc/bus@2100000/serial@21e8000
/sys/firmware/devicetree/base/soc/bus@2000000/spba-bus@2000000/serial@2020000
/sys/firmware/devicetree/base/soc/bus@2000000/spba-bus@2000000/serial@2018000

then checking if dma-names and dmas aren’t empty like

# hexdump /sys/firmware/devicetree/base/soc/bus@2200000/serial@2288000/dmas
0000000 0000 0a00 0000 2d00 0000 0400 0000 0000
0000010 0000 0a00 0000 2e00 0000 0400 0000 0000
0000020
#  hexdump -C /sys/firmware/devicetree/base/soc/bus@2200000/serial@2288000/dma-names
00000000  72 78 00 74 78 00                                 |rx.tx.|
00000006

If these are OK, then perhaps something wrong with SDMA

# dmesg | grep dma
[    0.344622] imx-sdma 20ec000.dma-controller: firmware found.
[    0.344929] imx-sdma 20ec000.dma-controller: loaded firmware 3.5
[    0.348321] mxs-dma 1804000.dma-apbh: initialized
7

My serial port list in /sys/firmware is the same but none of the folders have the subfolders “dma” and “dma-names”.

In the device tree itself the only relevant section seems to be

&sdma {
   status = "okay";
}

I used the provided device tree files from the latest BSP and only edited to change pin muxing as required and a few other minor things that shouldn’t relate to serial or DMA. Is there anything else that controls if the uarts want to use DMA?

The dma controller starts fine:

[    0.676253] mxs-dma 1804000.dma-apbh: initialized
[   14.079838] imx-sdma 20ec000.dma-controller: loaded firmware 3.5

– but only if I compile the kernel with CONFIG_IMX_SDMA=m . If I compile with CONFIG_IMX_SDMA=y the firmware won’t load with error “imx-sdma 20ec000.dma-controller: Direct firmware load for imx/sdma/sdma-imx6q.bin failed with error -2”, even though the bin file listed is present in the correct location. Presumably it should work fine as a module though because this is the default config for the BSP I believe.

8

dmas and dma-names are required. I’m using downstream kernel from NXP. Missing dmas and dma-names could be taken for example here

imx6ul.dtsi « dts « boot « arm « arch - linux-toradex.git - Linux kernel for Apalis, Colibri and Verdin modules

Don’t confuse with 6ul instead of 6ull. imx6ul.dtsi is included by some 6ull file. Settings for uart8, which is different on 6ull, are specified in imx6ull.dtsi. You may miss dmas settings for uart1, which someone at NXP thinks doesn’t deserve DMA because it is console port by default, but you still can find settings for it in imx6ull reference manual.

– but only if I compile the kernel with CONFIG_IMX_SDMA=m . If I compile with CONFIG_IMX_SDMA=y the firmware won’t load with error “imx-sdma 20ec000.dma-controller: Direct firmware load for imx/sdma/sdma-imx6q.bin failed with error -2”, even though the bin file listed is present in the correct

To use =y you need to incorporate FW in kernel with CONFIG_EXTRA_FIRMWARE.
CONFIG_EXTRA_FIRMWARE=“imx/sdma/sdma-imx6q.bin”
,where imx/sdma/ is sub-path from /lib/firmware/ on target to FW binary. Other FWs may be specified separating them with space.

9

Hi Edward,

Thanks for the detailed explanation and info. I used the downstream example to update the device tree to add the dmas to uart6 (which is the main one I am concerned about), and now the dma files under that serial@… are present, and the port works properly without having to disable the cpuidle/state2 . Thanks for the help.