Colibri imx7 rpmsg kernel crash

Technical Support
, , ,
1

Hi,

I can run the two RPMsg tty and ping pong examples on the imx7 fine but when there is little free memory the kernel module will crash.

I’m using the Colibri_iMX7_LinuxImageV2.6_20160630 release on a Colibri iMX7 Solo 256MB and following the example given at FreeRTOS on the Cortex-M4 of a Colibri iMX7 | Toradex Developer Center

A simple, reproducible way to show the crash is to fill /tmp since it’s mounted as a tmpfs. In the serial console log below i run up the RPMsg TTY example fine, use dd to fill /tmp and then the next message i send causes the imx_rpmsg_tty module to crash.

root@colibri-imx7:~# modprobe imx_rpmsg_tty
[   22.826041] imx_rpmsg_tty rpmsg0: new channel: 0x400 -> 0x0!
[   22.838912] Install rpmsg tty driver!
root@colibri-imx7:~# stty -F /dev/ttyRPMSG -echo
root@colibri-imx7:~# exec 3<> /dev/ttyRPMSG
root@colibri-imx7:~# echo Test >&3 
root@colibri-imx7:~# cat <&3
Test

^C
root@colibri-imx7:~# free
              total        used        free      shared  buff/cache   available
Mem:         250164       65116      142976         704       42072      167448
Swap:             0           0           0
root@colibri-imx7:~# df 
Filesystem           1K-blocks      Used Available Use% Mounted on
ubi0:rootfs             444980    145832    299148  33% /
devtmpfs                 59388         4     59384   0% /dev
tmpfs                   125080         0    125080   0% /dev/shm
tmpfs                   125080       312    124768   0% /run
tmpfs                   125080         0    125080   0% /sys/fs/cgroup
tmpfs                   125080         4    125076   0% /tmp
tmpfs                   125080         0    125080   0% /var/volatile
/dev/mmcblk0p1          524008    103696    420312  20% /media/mmcblk0p1
tmpfs                    25020         0     25020   0% /run/user/0
root@colibri-imx7:~# dd if=/dev/urandom of=/tmp/random.bin bs=1M
dd: writing '/tmp/random.bin': No space left on device
124+0 records in
122+1 records out
root@colibri-imx7:~# free
              total        used        free      shared  buff/cache   available
Mem:         250164       65336       17392      125876      167436       41988
Swap:             0           0           0
root@colibri-imx7:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
ubi0:rootfs             444980    146132    298848  33% /
devtmpfs                 59388         4     59384   0% /dev
tmpfs                   125080         0    125080   0% /dev/shm
tmpfs                   125080       316    124764   0% /run
tmpfs                   125080         0    125080   0% /sys/fs/cgroup
tmpfs                   125080    125080         0 100% /tmp
tmpfs                   125080         0    125080   0% /var/volatile
/dev/mmcblk0p1          524008    103696    420312  20% /media/mmcblk0p1
tmpfs                    25020         0     25020   0% /run/user/0
root@colibri-imx7:~# echo Test >&3
[  435.266193] Unable to handle kernel paging request at virtual address 9083f4ac
[  435.273422] pgd = 86cb0000
[  435.276181] [9083f4ac] *pgd=86000811, *pte=00000000, *ppte=00000000
[  435.282498] Internal error: Oops: 807 [#1] SMP ARM
[  435.287290] Modules linked in: imx_rpmsg_tty usb_f_rndis u_ether usb_f_acm u_serial mcp251x can_dev libcomposite configfs
[  435.298385] CPU: 0 PID: 593 Comm: sh Not tainted 4.1.15-v2.6b2+g0ff849d #1
[  435.305261] Hardware name: Freescale i.MX7 Dual (Device Tree)
[  435.311010] task: 86c3e4c0 ti: 86c96000 task.ti: 86c96000
[  435.316421] PC is at virtqueue_add_outbuf+0xbc/0x250
[  435.321392] LR is at rpmsg_send_offchannel_raw+0x270/0x32c
[  435.326883] pc : [<8032c914>]    lr : [<804a2858>]    psr: a00e0013
[  435.326883] sp : 86c97da8  ip : 000000d0  fp : 86185000
[  435.338363] r10: 808b97c4  r9 : 00000001  r8 : 809441b4
[  435.343591] r7 : 9083c000  r6 : 0000034a  r5 : 9083f4a0  r4 : 86c97e0c
[  435.350121] r3 : 87dba000  r2 : 000000fb  r1 : 00080000  r0 : 86c97e0c
[  435.356652] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  435.363791] Control: 10c5387d  Table: 86cb006a  DAC: 00000015
[  435.369539] Process sh (pid: 593, stack limit = 0x86c96210)
[  435.375114] Stack: (0x86c97da8 to 0x86c98000)
[  435.379479] 7da0:                   600d0013 0000034a 88060800 00000000 0000000a 00000000
[  435.387663] 7dc0: 00000001 88060800 8615fc00 00000012 8615fc20 8658c600 0000000a 804a2858
[  435.395847] 7de0: 000000d0 88060800 00000012 00000001 00000014 00000001 86190608 00000400
[  435.404032] 7e00: 00000000 807bea38 00000000 87ebac02 00000800 00000012 00000000 00000014
[  435.412217] 7e20: 00000000 00000002 86190600 807bec28 00000001 00000002 806502c4 8658c600
[  435.420401] 7e40: 0000000a 7f04b1a0 00000002 00000001 00000004 86655000 00000001 865a5004
[  435.428586] 7e60: 90c6e274 00002274 806502c4 803394c8 86655000 8033a60c 865a5000 86655080
[  435.436770] 7e80: 86c96000 8665518c 000a3a04 00000000 86c3e4c0 8005e5c0 86655190 86655190
[  435.444955] 7ea0: 8785a380 00000005 86655000 76f76000 00000005 86c96000 8658c600 00000000
[  435.453139] 7ec0: 8033a404 80336ccc 8782db40 00000005 000a3a04 8658c600 80336b5c 76f76000
[  435.461324] 7ee0: 86c97f88 8000f444 86c96000 00000000 000a3a10 800e7d58 76f76000 86c97fb0
[  435.469508] 7f00: fffffffc 00000000 7ed0d99c 80009268 00000002 00000000 808b4084 7ed0d1f4
[  435.477693] 7f20: 87c3e1a0 8657b480 00000000 7ed0d1f4 86c96000 800f8f90 00000000 8657b480
[  435.485877] 7f40: ffffe000 8658c600 00000005 76f76000 86c97f88 8000f444 86c96000 800e85f0
[  435.494061] 7f60: 8657b480 00000000 00000001 8658c600 8658c600 76f76000 00000005 8000f444
[  435.502246] 7f80: 86c96000 800e8eb8 00000000 00000000 00d49640 00000005 76f76000 76f17d60
[  435.510431] 7fa0: 00000004 8000f2c0 00000005 76f76000 00000001 76f76000 00000005 00000000
[  435.518615] 7fc0: 00000005 76f76000 76f17d60 00000004 00000005 000a6414 00086598 000a3a10
[  435.526799] 7fe0: 00000000 7ed0d94c 76e43818 76e9ce20 600e0010 00000001 412ca838 21ad5129
[  435.534994] [<8032c914>] (virtqueue_add_outbuf) from [<804a2858>] (rpmsg_send_offchannel_raw+0x270/0x32c)
[  435.544578] [<804a2858>] (rpmsg_send_offchannel_raw) from [<7f04b1a0>] (rpmsgtty_write+0x5c/0xb4 [imx_rpmsg_tty])
[  435.554858] [<7f04b1a0>] (rpmsgtty_write [imx_rpmsg_tty]) from [<803394c8>] (do_output_char+0x15c/0x1f4)
[  435.564348] [<803394c8>] (do_output_char) from [<8033a60c>] (n_tty_write+0x208/0x4bc)
[  435.572186] [<8033a60c>] (n_tty_write) from [<80336ccc>] (tty_write+0x170/0x294)
[  435.579593] [<80336ccc>] (tty_write) from [<800e7d58>] (__vfs_write+0x1c/0xd8)
[  435.586824] [<800e7d58>] (__vfs_write) from [<800e85f0>] (vfs_write+0x90/0x19c)
[  435.594141] [<800e85f0>] (vfs_write) from [<800e8eb8>] (SyS_write+0x44/0x9c)
[  435.601201] [<800e8eb8>] (SyS_write) from [<8000f2c0>] (ret_fast_syscall+0x0/0x3c)
[  435.608778] Code: e0875206 e5983000 e59a1000 e1a00004 (e1c590bc) 
[  435.614959] ---[ end trace 64724adfa2ce3c6e ]---

Has anyone else seen this issue?

Thanks

Martin

2

I am seeing the same issue. Has anybody from Toradex had an opportunity to look at this?

Thanks

Jonathan

3

@jonathan @martin Sorry for the delay in reply on this issue. Can you check if setting the following from u-boot resolves the issue.

setenv defargs "mem=255M"

Testing this here as per instructions by @martin does not reproduce the issue with the above set in u-boot. Please note that setting this, will now make 1MB of area not accessible to Linux.

On iMX7 we use a part of DRAM between the A7 and M4 core for communication using rpmsg. This memory is not reserved and managed by Linux explicitly at the moment and is only mapped by the concerned driver. To avoid Linux using the same memory location which is shared between A7 and M4, the above needs to be set explicitly.

4

Using linux,usable-memory would also be a correct solution. This is done already for another imx7d module. We will fix and deploy this to our kernel for Colibri iMX7 or update the documentation for including it in u-boot.

5

I just found a similar solution but i fixed it in the device tree. I created a new device tree file and set it’s contents to

#include "imx7s-colibri-eval-v3.dts"
/ {
	memory {
		linux,usable-memory = <0x80000000 0xff00000>;
	};
};

Is this not a bug therefore in imx7d-colibri-eval-v3.dts ? Should this not be set by default?

Thanks

6

@martin, thanks for the update, yeah that is definitely a bug, I will create a ticket for it.

As for the solution: The manual device tree entry is certainly a better solution than we have today. But I’d rather prefer U-Boot to setup the memory node, including reserved space for the M4 core. We use U-Boot’s memory size detection already today, and if we build a new SKU with more or less RAM in the future, it would just work.

7

@stefan.tx, i agree uboot seems the correct place to neatly solve this.

Note that freescale on the sabre board has a separate device tree for when you use the the M4. See imx7d-sdb-m4.dts and imx7d-sdb-m4.dtsi. This is where i got the clue to fix this.