Kernel Panics due to hung tasks when USB 3.0 USB stick is connected - colibri_vf50

Hello community!

We are facing a problem with below stack trace when we connect USB 3.0 stick to our custom board running linux on colibri_vf50.

[   20.112345] INFO: task usb-storage:52 blocked for more than 10 seconds.
[   20.119121]       Not tainted 4.1.15-00002-g59d7825e-dirty #204
[   20.125160] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   20.133127] usb-storage     D 80513138     0    52      2 0x00000000
[   20.139742] Backtrace: 
[   20.142398] [<80512f7c>] (__schedule) from [<8051347c>] (schedule+0x44/0x9c)
[   20.149551]  r9:87ae3390 r8:00000000 r7:00000000 r6:00000002 r5:87b5c000 r4:87b5c000
[   20.157797] [<80513438>] (schedule) from [<805155cc>] (schedule_timeout+0x110/0x158)
[   20.165642]  r5:87b5c000 r4:7fffffff
[   20.169466] [<805154bc>] (schedule_timeout) from [<80513e54>] (wait_for_common+0x90/0x13c)
[   20.177825]  r9:87ae3390 r8:00000000 r7:00000000 r6:00000002 r5:87b5c000 r4:7fffffff
[   20.186058] [<80513dc4>] (wait_for_common) from [<80513f18>] (wait_for_completion+0x18/0x1c)
[   20.194631]  r9:8605700f r8:0000001f r7:00000001 r6:00000000 r5:87ae3390 r4:87ae3374
[   20.202911] [<80513f00>] (wait_for_completion) from [<80365fa8>] (usb_sg_wait+0xec/0x144)
[   20.211263] [<80365ebc>] (usb_sg_wait) from [<8037bc30>] (usb_stor_bulk_transfer_sglist.part.2+0xac/0)
[   20.221021]  r7:00000024 r6:c0008280 r5:87ae3374 r4:87ae32f0
[   20.227052] [<8037bb84>] (usb_stor_bulk_transfer_sglist.part.2) from [<8037bcdc>] (usb_stor_bulk_srb+)
[   20.237273]  r7:00000024 r6:87be1200 r5:87ae32f0 r4:87be1200
[   20.243354] [<8037bc8c>] (usb_stor_bulk_srb) from [<8037be00>] (usb_stor_Bulk_transport+0x11c/0x378)
[   20.252581]  r4:86057000
[   20.255321] [<8037bce4>] (usb_stor_Bulk_transport) from [<8037c674>] (usb_stor_invoke_transport+0x28/)
[   20.265161]  r9:87ae32f0 r8:ffffff80 r7:00000000 r6:00000000 r5:87be1200 r4:87ae32f0
[   20.273437] [<8037c64c>] (usb_stor_invoke_transport) from [<8037b40c>] (usb_stor_transparent_scsi_com)
[   20.284052]  r10:00000000 r9:87ae32f0 r8:ffffff80 r7:00000000 r6:00000000 r5:87ae33a8
[   20.292366]  r4:00000000
[   20.295039] [<8037b3fc>] (usb_stor_transparent_scsi_command) from [<8037d9b4>] (usb_stor_control_thre)
[   20.305716] [<8037d86c>] (usb_stor_control_thread) from [<8003a2bc>] (kthread+0xdc/0xf4)
[   20.313911]  r9:00000000 r8:00000000 r7:8037d86c r6:87ae32f0 r5:87b51780 r4:00000000
[   20.322172] [<8003a1e0>] (kthread) from [<8000f6a8>] (ret_from_fork+0x14/0x2c)
[   20.329541]  r7:00000000 r6:00000000 r5:8003a1e0 r4:87b51780
[   20.335538] Kernel panic - not syncing: hung_task: blocked tasks
[   20.341578] Rebooting in 1 seconds..

I also tried updating the kernel to 4.4 and it gave the same results. USB 2.0 sticks work perfectly. The port we are dealing with is the OTG port. If I limit the speed to full-speed by updating the device tree property, kernel panic only happens if I try to write large files e.g. 20MB to the USB stick.

Any support here is very much appreciated. Thanks.

Hi @nikKamath and Welcome to the Toradex Community!

Could you provide the version of the hardware and software of your module? Which carrier board are you using? Have you done any changes (Kernel, devicetree, …) to the Software?

I tried connecting a USB 3.0 to the OTG Port of VF50 with the Bsp 2.8b6 and it works fine.
Could you check this also on your side?

Best regards,
Jaski

@jaski.tx Thanks for the reply

  1. SOM - >Toradex Colibri VF50 128MB IT V1.2A
  2. Carrier Board → custom
  3. Image → Colibri_VF_LinuxImageV2.5_20151215.tar.bz2
  4. Kernel → 4.1.15-00002

I also tried linux-toradex branch toradex_vf_4.4, also resulted in same issue → git.toradex.com/linux-toradex.git

Additional notes:
The USB 3.0 stick which causes panic on the target , works perfectly when I connect the same to Desktop.
Also tried complete formatting with different schemes - > ext4, FAT, NTFS.

@jaski.tx Can you please try dd command on USB stick e.g. dd if=/dev/zero of=test.img bs=1M count=100 conv=sync

I tried the kernel from Bsp 2.8b6 along with the device tree. It panics!

 140.880878] INFO: task usb-storage:181 blocked for more than 10 seconds.
[  140.887619]       Not tainted 4.4.177-2.8.6+g6f01eb5bf8e8 #1
[  140.893443] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  140.901445] usb-storage     D 805fde1c     0   181      2 0x00000000
[  140.907862] Backtrace: 
[  140.910387] [<805fdc5c>] (__schedule) from [<805fe18c>] (schedule+0x44/0xa4)
[  140.917583]  r10:00000000 r9:00000000 r8:00000000 r7:00000002 r6:ffffe000 r5:7fffffff
[  140.925620]  r4:ffffe000
[  140.928212] [<805fe148>] (schedule) from [<80600618>] (schedule_timeout+0xd8/0x118)
[  140.936002]  r5:7fffffff r4:7fffffff
[  140.939660] [<80600540>] (schedule_timeout) from [<805fed00>] (wait_for_common+0xa8/0x154)
[  140.948053]  r6:ffffe000 r5:8516a4a0 r4:7fffffff
[  140.952858] [<805fec58>] (wait_for_common) from [<805fedc4>] (wait_for_completion+0x18/0x1c)
[  140.961426]  r9:0000001f r8:02080020 r7:00000001 r6:00000000 r5:8516a4a0 r4:8516a484
[  140.969313] [<805fedac>] (wait_for_completion) from [<803dba0c>] (usb_sg_wait+0x12c/0x138)
[  140.977746] [<803db8e0>] (usb_sg_wait) from [<803f47f4>] (usb_stor_bulk_transfer_sglist.part.2+0x80/0)
[  140.987442]  r9:0000001f r8:000000c0 r7:000000c0 r6:c0008380 r5:8516a484 r4:8516a400
[  140.995404] [<803f4774>] (usb_stor_bulk_transfer_sglist.part.2) from [<803f48a4>] (usb_stor_bulk_srb+)
[  141.005644]  r7:00000000 r6:8519b700 r5:86062000 r4:8519b700
[  141.011535] [<803f4868>] (usb_stor_bulk_srb) from [<803f49cc>] (usb_stor_Bulk_transport+0x110/0x36c)
[  141.020792]  r4:8516a400
[  141.023389] [<803f48bc>] (usb_stor_Bulk_transport) from [<803f5250>] (usb_stor_invoke_transport+0x28/)
[  141.033253]  r9:00000000 r8:00000000 r7:8516a400 r6:80862700 r5:8519b700 r4:8516a400
[  141.041230] [<803f5228>] (usb_stor_invoke_transport) from [<803f4014>] (usb_stor_transparent_scsi_com)
[  141.051876]  r10:00000000 r9:00000000 r8:00000000 r7:8516a400 r6:80862700 r5:8516a4b8
[  141.059799]  r4:8516a400
[  141.062505] [<803f4004>] (usb_stor_transparent_scsi_command) from [<803f65e0>] (usb_stor_control_thre)
[  141.073103] [<803f6518>] (usb_stor_control_thread) from [<8003b7f0>] (kthread+0x108/0x110)
[  141.081491]  r9:00000000 r8:803f6518 r6:851ae000 r5:85d98c80 r4:00000000
[  141.088335] [<8003b6e8>] (kthread) from [<8000fad0>] (ret_from_fork+0x14/0x24)
[  141.095691]  r8:00000000 r7:00000000 r6:00000000 r5:8003b6e8 r4:85d98c80

Anyone here knows if we need 15K pull down on USBC_P and USBC_N ? or it’s internally done?
Thanks!

hi @nikKamath: The Soc of Colibri VF50 contains switch-able Pull-up and Pull-down as described in section 11.2.4.5.4 and 11.2.4.5.5 of the Reference Manual.