Colibri iMX6dl - Kernel crash on Linux Image

Hi, I’m trying to compile a Linux Image for iMX6dl modules. I need to apply
some customization to devicetree to activate a serial port and some gpio
signals.

My starting point is tag Colibri-iMX6_LXDE-Image_2.8b3.111-20180627. My image
in an Iris carrier systems seems unstable: when I do simple operations (eg.
cut/paste) on terminal using serial cable my login session crash with errors
like this:

[  253.253651] Unhandled fault: imprecise external abort (0x1406) at 0x76d660f2
[  253.260933] pgd = 8cf68000
[  253.263655] [76d660f2] *pgd=1cf5c831, *pte=00000000, *ppte=00000000
[  253.270002] Internal error: : 1406 [#1] SMP ARM
[  253.274545] Modules linked in: bluetooth usb_f_rndis u_ether libcomposite configfs
[  253.282253] CPU: 0 PID: 911 Comm: sh Not tainted 4.9.87-2.8.3+g07d40f6f #5
[  253.289141] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[  253.295682] task: 8c4f1b80 task.stack: 8c93e000
[  253.300244] PC is at _raw_spin_unlock_irqrestore+0x28/0x2c
[  253.305752] LR is at uart_write+0x118/0x180
[  253.309956] pc : [<80837764>]    lr : [<804ab1c8>]    psr: 60030013
[  253.309956] sp : 8c93fdf0  ip : 8c93fe00  fp : 8c93fdfc
[  253.321453] r10: 8c395c10  r9 : 8c92a003  r8 : 00000fff
[  253.326693] r7 : 00000003  r6 : 00000000  r5 : 8c370800  r4 : 00000000
[  253.333237] r3 : 00002094  r2 : 00000730  r1 : a0030013  r0 : 8c395c10
[  253.339781] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[  253.346930] Control: 10c5387d  Table: 1cf6804a  DAC: 00000051
[  253.352692] Process sh (pid: 911, stack limit = 0x8c93e210)
[  253.358278] Stack: (0x8c93fdf0 to 0x8c940000)
[  253.362659] fde0:                                     8c93fe34 8c93fe00 804ab1c8 80837748
[  253.370863] fe00: a0030013 8ca09e00 8ca09e00 8ca09e00 00000003 8c92a000 80932af0 8c5fce40
[  253.379065] fe20: a0d34270 00000000 8c93fe8c 8c93fe38 80493b30 804ab0bc 8ca09f8c 8c93e000
[  253.387266] fe40: 8c92a000 8ca09e80 a0d34258 00000002 8c4f1b80 80163a0c 8ca09f90 8ca09f90
[  253.395468] fe60: 8c93ff78 8ca09e00 00000003 01b0fc28 00000003 00000000 00000051 00000003
[  253.403669] fe80: 8c93fed4 8c93fe90 8048edbc 8049397c 8c93feac 8caa9b40 8c5fce40 80493970
[  253.411869] fea0: ffffe000 8c92a000 8048ec04 8048ec30 8c5fce40 8c93ff78 8c93ff78 00000000
[  253.420071] fec0: 00000003 00000000 8c93ff44 8c93fed8 802092f0 8048ec3c 802091dc 8048eb60
[  253.428272] fee0: 8c93ff0c 8c93fef0 8016f1b0 80171d18 8c553310 00000004 8c0aff00 8c0aff60
[  253.436471] ff00: 00000001 8c5fce48 00000001 8083760c 8c253290 00000001 80172160 00000000
[  253.444672] ff20: 00000003 8c5fce40 01b0fc28 8c93ff78 00000000 00000003 8c93ff74 8c93ff48
[  253.452873] ff40: 8020a134 802092cc 76e52da8 00000004 8c5fce40 8c5fce40 00000000 00000000
[  253.461073] ff60: 01b0fc28 00000003 8c93ffa4 8c93ff78 8020af98 8020a094 00000000 00000000
[  253.469274] ff80: 00000003 01b0fc28 76e52da8 00000004 80108c24 8c93e000 00000000 8c93ffa8
[  253.477473] ffa0: 80108a60 8020af60 00000003 01b0fc28 00000001 01b0fc28 00000003 00000000
[  253.485673] ffc0: 00000003 01b0fc28 76e52da8 00000004 00000003 00000003 76e507a0 76e52000
[  253.493874] ffe0: 0006c234 7eabb678 76d7f670 76dd718c 60030010 00000001 1ff7e861 1ff7ec61
[  253.502058] Backtrace: 
[  253.504551] [<8083773c>] (_raw_spin_unlock_irqrestore) from [<804ab1c8>] (uart_write+0x118/0x180)
[  253.513458] [<804ab0b0>] (uart_write) from [<80493b30>] (n_tty_write+0x1c0/0x44c)
[  253.520969]  r10:00000000 r9:a0d34270 r8:8c5fce40 r7:80932af0 r6:8c92a000 r5:00000003
[  253.528810]  r4:8ca09e00
[  253.531373] [<80493970>] (n_tty_write) from [<8048edbc>] (tty_write+0x18c/0x36c)
[  253.538793]  r10:00000003 r9:00000051 r8:00000000 r7:00000003 r6:01b0fc28 r5:00000003
[  253.546635]  r4:8ca09e00
[  253.549206] [<8048ec30>] (tty_write) from [<802092f0>] (__vfs_write+0x30/0x118)
[  253.556540]  r10:00000000 r9:00000003 r8:00000000 r7:8c93ff78 r6:8c93ff78 r5:8c5fce40
[  253.564381]  r4:8048ec30
[  253.566947] [<802092c0>] (__vfs_write) from [<8020a134>] (vfs_write+0xac/0x170)
[  253.574279]  r9:00000003 r8:00000000 r7:8c93ff78 r6:01b0fc28 r5:8c5fce40 r4:00000003
[  253.582049] [<8020a088>] (vfs_write) from [<8020af98>] (SyS_write+0x44/0x98)
[  253.589121]  r9:00000003 r8:01b0fc28 r7:00000000 r6:00000000 r5:8c5fce40 r4:8c5fce40
[  253.596898] [<8020af54>] (SyS_write) from [<80108a60>] (ret_fast_syscall+0x0/0x48)
[  253.604492]  r9:8c93e000 r8:80108c24 r7:00000004 r6:76e52da8 r5:01b0fc28 r4:00000003
[  253.612259] Code: e1c030b0 f57ff04a e320f004 e121f001 (e89da800) 
[  253.618371] ---[ end trace 35c880184f21edb0 ]---

This is my device tree overlay:

 /dts-v1/;
 
 #include <dt-bindings/input/input.h>
 #include <dt-bindings/interrupt-controller/irq.h>
 #include "imx6dl.dtsi"
 #include "imx6qdl-colibri.dtsi"
 
 
 / {
 	model = "Toradex Colibri iMX6DL/S on custom board";
         version = "002";
 	compatible = "toradex,colibri_imx6dl-eval", "toradex,colibri_imx6dl", "fsl,imx6dl";
 
 };
 
 
 &iomuxc {
 	pinctrl-names = "default";
 	pinctrl-0 = <&pinctrl_csi_gpio_1
 	             &pinctrl_gpio_1
 	             &pinctrl_gpio_2
 	             &pinctrl_usbh_oc_1 &pinctrl_usbc_id_1>;
 
 	uart1 {
 		pinctrl_uart1_dte: uart1-dtegrp { /* DTE mode */
 			fsl,pins = <
 				MX6QDL_PAD_CSI0_DAT10__UART1_RX_DATA 0x1b0b1
 				MX6QDL_PAD_CSI0_DAT11__UART1_TX_DATA 0x1b0b1
 				MX6QDL_PAD_EIM_D19__UART1_CTS_B 0x1b0b1
 				MX6QDL_PAD_EIM_D20__UART1_RTS_B 0x1b0b1
 			>;
 		};
 		pinctrl_uart1_ctrl: uart1-ctrlgrp { /* Additional DTR, DSR, DCD */
 			fsl,pins = <
 				MX6QDL_PAD_EIM_D23__UART1_DCD_B 0x1b0b0
 				MX6QDL_PAD_EIM_D24__UART1_DTR_B 0x1b0b0
 				MX6QDL_PAD_EIM_D25__UART1_DSR_B 0x1b0b0
 			>;
 		};
 	};
 
 	uart2 {
 		pinctrl_uart2_dte: uart2grp-dte { /* DTE mode */
 			fsl,pins = <
 				MX6QDL_PAD_SD4_DAT7__UART2_RX_DATA   0x1b0b1
 				MX6QDL_PAD_SD4_DAT4__UART2_TX_DATA   0x1b0b1
 				MX6QDL_PAD_SD4_DAT5__UART2_RTS_B    0x1b0b1
 			>;
 		};
 	};
 
 	uart3 {
 		pinctrl_uart3_dte: uart3grp-dte { /* DTE mode */
 			fsl,pins = <
 				MX6QDL_PAD_SD4_CLK__UART3_TX_DATA 0x1b0b1
 				MX6QDL_PAD_SD4_CMD__UART3_RX_DATA 0x1b0b1
 			>;
 		};
 	};
 
 	uart4 {
 		pinctrl_uart4_dte: uart4grp-dte { /* DTE mode */
 			fsl,pins = <
 				MX6QDL_PAD_CSI0_DAT13__UART4_TX_DATA 0x1b0b1
 				MX6QDL_PAD_CSI0_DAT12__UART4_RX_DATA 0x1b0b1
 			>;
 		};
 	};
 
        gpio {
                 pinctrl_gpioreset: gpioreset {
                         fsl,pins = <
                                 MX6QDL_PAD_GPIO_9__GPIO1_IO09 0x80 /* SoC PAD 32 */
                         >;
                 };
 
                 pinctrl_gpioled: gpioled {
                         fsl,pins = <
                                 MX6QDL_PAD_EIM_D25__GPIO3_IO25 0x80 /* SoC 28 - Blue */
                                 MX6QDL_PAD_SD4_DAT2__GPIO2_IO10 0x80 /* SoC 29 - Green*/
                                 MX6QDL_PAD_SD4_DAT6__GPIO2_IO14 0x80 /* SoC 30 - Red*/
                         >;
                 };
        };
 };
 
 &uart1 {
 	status = "okay";
 	pinctrl-0 = <&pinctrl_uart1_dte>;
 	linux,rs485-enabled-at-boot-time;
 	rs485-rts-delay = <0 0>;
 };
 
 &uart2 {
 	status = "okay";
 	pinctrl-0 = <&pinctrl_uart2_dte>;
 	linux,rs485-enabled-at-boot-time;
 	rs485-rts-delay = <0 0>;
 };
 
 &uart3 {
 	status = "okay";
 };
 
 &uart4 {
         status = "okay";
 	pinctrl-0 = <&pinctrl_uart4_dte>;
 	pinctrl-names = "default";
 	fsl,dte-mode;
 };
 
 &weim {
 	status = "disabled";
 };
 
 &pwm1 {
        status = "disabled";
 };
 
 &pwm2 {
        status = "disabled";
 };
 
 &pwm3 {
        status = "disabled";
 };
 
 &pwm4 {
        status = "disabled";
 };

Another difference between the vanilla image and my image is the root
partition: I cannot explain why, using my image, root filesystem is in
/dev/mmcblk2p2 (in vanilla is /dev/mmcblk0p2). This breaks also the auto
resize script.

Hi @albeus

Which commands did you apply exactly? Did you try doing the operations without any changes of the regular Bsp?

Another difference between the vanilla image and my image is the root partition: I cannot explain why, using my image, root filesystem is in /dev/mmcblk2p2 (in vanilla is /dev/mmcblk0p2). This breaks also the auto resize script.

How did you create the image?

Hi @jaski.tx

Which commands did you apply exactly? Did you try doing the operations
without any changes of the regular Bsp?

Using regular Bsp (tag: Colibri-iMX6_LXDE-Image_2.8b3.111-20180627) all works fine.

How did you create the image?

I’m using openembedded as described here: https://developer.toradex.com/knowledge-base/board-support-package/openembedded-(core)#Building

I use the target “console-tdx-image” (command: bitbake console-tdx-image).

Thanks for this Information. How did you flash the image? Using Toradex Esay Installer or doing legacy update?

I use uboot commands as described here: https://developer.toradex.com/knowledge-base/flashing-linux-on-imx6-modules#Colibri_iMX6.

I solved the problem related to different root device. There was an error in my
device tree, I didn’t included in my code Iris configurations to fix error in
aliases assignments:

    aliases {
            /* the following, together with kernel patches, forces a fixed assignment
               between device id and usdhc controller */
            /* i.e. the eMMC on usdhc3 will be /dev/mmcblk0 */
            mmc0 = &usdhc3; /* eMMC */
            mmc1 = &usdhc1; /* MMC 4bit slot */
    };

Now my overlay includes all configurations of Iris board (imx6dl-colibri-eval-v3.dts).

However, main problem is not solved: kernel crashes during serial connection
still happens.

Perfect that your root device problem is solved.

when I do simple operations (eg. cut/paste) on terminal using serial cable my login session crash with errors like this.

I did not understand what simple operations are you doing? Could you provide some information for that.
Have you done other changes to the software except the device tree files? Could you provide the output git diff in a file?

Thanks.

I did some tests and for delineating the problem. The only change that cause
the instability is the device tree overlay (see my original request). I
previously created another image for old kernel version (4.1.2) using an almost
identical devicetree and they worked without problems.

Sorry for the ambiguous description of the circumstance that triggers the
crash, I still have no clear ideas. I only noticed it happens after some time I
use the serial login session (edit files, scrolling, cut/paste commands…)

I attach here my device tree compiled blob converted using dtc.

Thanks for the file. I did see anything suspicious. Could you send also your dts/dtsi files?
The Issue is linked to UART communication. Could you try to disable the Uart Interfaces and just enable an Interface 1 by 1 to see which Uart port is causing the crash. To get the connection to the module, you can use SSH Connection.

Thanks for your support.

I did some test as you suggested. It seems that only the uart1 configuration is
resposible of the unstability. I commented uarts in turn and only with uart1
defined I had the problem.

I attach here:

  • my dts file and the dtb decompiled
  • my pin designer configuration

Other files are the default one for tag
Colibri-iMX6_LXDE-Image_2.8b3.111-20180627. So you can find them here:

hi @albeus

Thanks for the files. There is a problem with uart1 pin multiplexing, which should be as following in dte mode.

uart2 {
		pinctrl_uart2_dte: uart2grp-dte { /* DTE mode */			fsl,pins = <
				MX6QDL_PAD_SD4_DAT4__UART2_TX_DATA   0x1b0b1
				MX6QDL_PAD_SD4_DAT7__UART2_RX_DATA   0x1b0b1
				MX6QDL_PAD_SD4_DAT6__UART2_RTS_B 0x1b0b1
				MX6QDL_PAD_SD4_DAT5__UART2_CTS_B 0x1b0b1
			>;
		};
};

Actually you can use our regular Uart Configuration. Please read also the following to get the difference between DTE and DCE.

Thank you @jaski.tx!

I’ve only a doubt about your example: it’s related to uart2 but my problem related to uart1. I tryed anyway to update my uart2 configuration as you told but nothing changes.

However your links to documentation seem interesting, I’ll go in deep to the DTE/DCE issue.

Hi @albeus
Sorry about the mistake. Here is the same example with uart1

uart1 {
		pinctrl_uart1_dte: uart1-dtegrp { /* DTE mode */
			fsl,pins = <
				MX6QDL_PAD_CSI0_DAT10__UART1_RX_DATA 0x1b0b1
				MX6QDL_PAD_CSI0_DAT11__UART1_TX_DATA 0x1b0b1
				MX6QDL_PAD_EIM_D19__UART1_RTS_B 0x1b0b1
				MX6QDL_PAD_EIM_D20__UART1_CTS_B 0x1b0b1
			>;
		};
}

@jaski.tx:

I reflected about this issue and I reached the conclusion that this
“unstability” is not a real problem but the consequence of using and rs485
channel as an rs232. If you look at my device tree overlay you can see that I
redefine the uart1 as an rs485 port (because I use it in my board for reading a
specific data stream), but the console dialog assume using an rs232.

I will change my setup procedure to use networking as dialog channel to
finalize system setup.

Do you agree?

Hmm, this is strange. You just enabled the RS485 support. If you disable this support, do you still see kernel crashes or not? By the way, you should not use uart1 for something else, if you are using it for the console.

I will change my setup procedure to use networking as dialog channel to finalize system setup.

Yeah, you can make a ssh connection to the module.