Kernel getting crash randomly

Hi all,

In VF61, kernel is getting cr̥ash randomly. Below are the logs which shows the crash issue.

[175329.355177] Unable to handle kernel NULL pointer dereference at virtual address 00000004
[175329.363413] pgd = 80004000
[175329.366220] [00000004] *pgd=00000000
[175329.369931] Internal error: Oops: 17 [#1] ARM
[175329.374388] Modules linked in: virtio_rpmsg_bus vf610_rpmsg virtio virtio_ring flexcan can_dev libcomposite configfs
[175329.385179] CPU: 0 PID: 66 Comm: kworker/u2:2 Not tainted 4.4.88-2.7.4+ge0f2806 #3
[175329.392889] Hardware name: Freescale Vybrid VF5xx/VF6xx (Device Tree)
[175329.399478] task: 8f68a140 ti: 8f75e000 task.ti: 8f75e000
[175329.405028] PC is at process_one_work+0x2c/0x3d8
[175329.409785] LR is at worker_thread+0x78/0x564
[175329.414280] pc : [<8003553c>] lr : [<80036560>] psr: 00010093
[175329.414280] sp : 8f75fee0 ip : 8f75ff20 fp : 8f75ff1c
[175329.426012] r10: 8f74a800 r9 : 8f402000 r8 : 8f402000
[175329.431365] r7 : 8f74a818 r6 : 00000000 r5 : 8f74a800 r4 : 8f40200c
[175329.438024] r3 : 00000004 r2 : 8f402010 r1 : 8f40200c r0 : 8f74a800
[175329.444692] Flags: nzcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none
[175329.452076] Control: 10c5387d Table: 8db60059 DAC: 00000051
[175329.457957] Process kworker/u2:2 (pid: 66, stack limit = 0x8f75e208)
[175329.464440] Stack: (0x8f75fee0 to 0x8f760000)
[175329.468937] fee0: ffffe000 8f402014 8f402000 00000088 8f402000 8f402000 ffffe000 8f402014
[175329.477289] ff00: 8f74a818 00000088 8f402000 8f74a800 8f75ff5c 8f75ff20 80036560 8003551c
[175329.485638] ff20: 807bcc28 8f402160 808bf040 808eb49b 00000000 00000000 8f7497c0 8f75e000
[175329.493986] ff40: 8f74a800 800364e8 00000000 00000000 8f75ffac 8f75ff60 8003b1c4 800364f4
[175329.502334] ff60: 00000000 00000000 8003b0bc 8f74a800 00000000 8f75ff74 8f75ff74 00000000
[175329.510683] ff80: 8f75ff80 8f75ff80 80040a14 8f7497c0 8003b0bc 00000000 00000000 00000000
[175329.519031] ffa0: 00000000 8f75ffb0 8000fa78 8003b0c8 00000000 00000000 00000000 00000000
[175329.527378] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[175329.535727] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[175329.544060] Backtrace:
[175329.546663] [<80035510>] (process_one_work) from [<80036560>] (worker_thread+0x78/0x564)
[175329.554913] r10:8f74a800 r9:8f402000 r8:00000088 r7:8f74a818 r6:8f402014 r5:ffffe000
[175329.562975] r4:8f402000
[175329.565660] [<800364e8>] (worker_thread) from [<8003b1c4>] (kthread+0x108/0x110)
[175329.573211] r10:00000000 r9:00000000 r8:800364e8 r7:8f74a800 r6:8f75e000 r5:8f7497c0
[175329.581269] r4:00000000
[175329.583957] [<8003b0bc>] (kthread) from [<8000fa78>] (ret_from_fork+0x14/0x3c)
[175329.591332] r8:00000000 r7:00000000 r6:00000000 r5:8003b0bc r4:8f7497c0
[175329.598247] Code: e2136004 e1a05000 13c360ff e598300c (e5962004)
[175329.604478] —[ end trace 644d96da60d7174f ]—
[175329.609241] ------------[ cut here ]------------
[175329.614016] WARNING: CPU: 0 PID: 66 at /home/sysadmin/dhawal/bh_bpl_gcm/yocto/build_6700d/tmp-vfxx-nlg-glibc/work-shared/colibri-vf/kernel-source/kernel/softirq.c:150 __local_bh_enable_ip+0x98/0xcc()
[175329.631939] Modules linked in: virtio_rpmsg_bus vf610_rpmsg virtio virtio_ring flexcan can_dev libcomposite configfs
[175329.642772] CPU: 0 PID: 66 Comm: kworker/u2:2 Tainted: G D 4.4.88-2.7.4+ge0f2806 #3
[175329.651719] Hardware name: Freescale Vybrid VF5xx/VF6xx (Device Tree)
[175329.658301] Backtrace:
[175329.660909] [<800136a0>] (dump_backtrace) from [<800138b8>] (show_stack+0x18/0x1c)
[175329.668632] r7:80024e68 r6:00000096 r5:00000009 r4:00000000
[175329.674497] [<800138a0>] (show_stack) from [<802acac4>] (dump_stack+0x24/0x28)
[175329.681892] [<802acaa0>] (dump_stack) from [<80021e74>] (warn_slowpath_common+0x88/0xb4)
[175329.690154] [<80021dec>] (warn_slowpath_common) from [<80021f44>] (warn_slowpath_null+0x24/0x2c)
[175329.699094] r8:00000000 r7:8f75fcc8 r6:00000000 r5:808eb48e r4:00000200
[175329.706012] [<80021f20>] (warn_slowpath_null) from [<80024e68>] (__local_bh_enable_ip+0x98/0xcc)
[175329.714976] [<80024dd0>] (__local_bh_enable_ip) from [<800769e0>] (cgroup_exit+0x50/0x74)
[175329.723314] r5:0000000b r4:8f68a140
[175329.727053] [<80076990>] (cgroup_exit) from [<80022bcc>] (do_exit+0x250/0x904)
[175329.734446] [<8002297c>] (do_exit) from [<80013c28>] (die+0x36c/0x3cc)
[175329.741102] r7:00000001
[175329.743788] [<800138bc>] (die) from [<8001e89c>] (__do_kernel_fault.part.0+0x6c/0x1f0)
[175329.751865] r10:8f74a800 r9:8f402000 r8:00000004 r7:00000000 r6:00000017 r5:8f75fe90
[175329.759921] r4:00000004
[175329.762609] [<8001e830>] (__do_kernel_fault.part.0) from [<8001a610>] (do_bad_area+0x0/0x90)
[175329.771202] r7:00000017 r4:8f75fe90
[175329.774942] [<8001a268>] (do_page_fault) from [<80009290>] (do_DataAbort+0x44/0xc0)
[175329.782751] r10:8f74a800 r9:8f402000 r8:8f75fe90 r7:00000004 r6:8001a268 r5:00000017
[175329.790816] r4:808bb894
[175329.793491] [<8000924c>] (do_DataAbort) from [<800142e0>] (__dabt_svc+0x40/0x60)
[175329.801047] Exception stack(0x8f75fe90 to 0x8f75fed8)
[175329.806231] fe80: 8f74a800 8f40200c 8f402010 00000004
[175329.814581] fea0: 8f40200c 8f74a800 00000000 8f74a818 8f402000 8f402000 8f74a800 8f75ff1c
[175329.822926] fec0: 8f75ff20 8f75fee0 80036560 8003553c 00010093 ffffffff
[175329.829674] r8:8f402000 r7:8f75fec4 r6:ffffffff r5:00010093 r4:8003553c
[175329.836593] [<80035510>] (process_one_work) from [<80036560>] (worker_thread+0x78/0x564)
[175329.844837] r10:8f74a800 r9:8f402000 r8:00000088 r7:8f74a818 r6:8f402014 r5:ffffe000
[175329.852901] r4:8f402000
[175329.855582] [<800364e8>] (worker_thread) from [<8003b1c4>] (kthread+0x108/0x110)
[175329.863138] r10:00000000 r9:00000000 r8:800364e8 r7:8f74a800 r6:8f75e000 r5:8f7497c0
[175329.871195] r4:00000000
[175329.873875] [<8003b0bc>] (kthread) from [<8000fa78>] (ret_from_fork+0x14/0x3c)
[175329.881249] r8:00000000 r7:00000000 r6:00000000 r5:8003b0bc r4:8f7497c0
[175329.888153] —[ end trace 644d96da60d71750 ]—
[175329.893094] Unable to handle kernel paging request at virtual address fffffff0
[175329.900478] pgd = 80004000
[175329.903311] [fffffff0] *pgd=8fdf6861, *pte=00000000, *ppte=00000000
[175329.909757] Internal error: Oops: 37 [#2] ARM
[175329.914238] Modules linked in: virtio_rpmsg_bus vf610_rpmsg virtio virtio_ring flexcan can_dev libcomposite configfs
[175329.925071] CPU: 0 PID: 66 Comm: kworker/u2:2 Tainted: G D W 4.4.88-2.7.4+ge0f2806 #3
[175329.934017] Hardware name: Freescale Vybrid VF5xx/VF6xx (Device Tree)
[175329.940606] task: 8f68a140 ti: 8f75e000 task.ti: 8f75e000
[175329.946147] PC is at kthread_data+0x10/0x18
[175329.950460] LR is at wq_worker_sleeping+0x14/0xc0
[175329.955297] pc : [<8003b53c>] lr : [<80036c14>] psr: 20010193
[175329.955297] sp : 8f75fc38 ip : 8f75fc48 fp : 8f75fc44
[175329.967028] r10: 807b8528 r9 : 8f68a38c r8 : 00000000
[175329.972381] r7 : 8f75fcc8 r6 : 00000000 r5 : 808bdd98 r4 : 00000000
[175329.979040] r3 : 00000000 r2 : 1eb7d748 r1 : 00000000 r0 : 8f68a140
[175329.985697] Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none
[175329.993083] Control: 10c5387d Table: 8db60059 DAC: 00000051
[175329.998964] Process kworker/u2:2 (pid: 66, stack limit = 0x8f75e208)
[175330.005447] Stack: (0x8f75fc38 to 0x8f760000)
[175330.009934] fc20: 8f75fc5c 8f75fc48
[175330.018280] fc40: 80036c14 8003b538 8f68a140 808bdd98 8f75fcac 8f75fc60 80648894 80036c0c
[175330.026628] fc60: 8f75fc84 8f75fc70 8f75fc84 8f75fc78 8005607c 8005601c 8f75fcc4 80648b98
[175330.034977] fc80: 800228e8 ffffe000 8f75fae4 8f450000 8f75fcc8 00000000 8f68a308 807b8528
[175330.043325] fca0: 8f75fcc4 8f75fcb0 80648b98 80648650 8f68a140 8f75fae4 8f75fce4 8f75fcc8
[175330.051674] fcc0: 80022f38 80648b60 8f75fcc8 8f75fcc8 80021f7c 00000001 8f75fd6c 8f75fce8
[175330.060022] fce0: 80013c28 80022988 8f75e208 0000000b 8f75fd14 8003553d 00000008 00000000
[175330.068371] fd00: 00000000 60010193 6575fd24 36333132 20343030 30613165 30303035 63333120
[175330.076720] fd20: 66303633 35652066 30333839 28206330 36393565 34303032 80002029 8008b01c
[175330.085068] fd40: 80819dec 00000004 8f75fe90 00000017 00000000 00000004 8f402000 8f74a800
[175330.093417] fd60: 8f75fd84 8f75fd70 8001e89c 800138c8 8f75fe90 00000017 8f75fddc 8f75fd88
[175330.101765] fd80: 8001a610 8001e83c 808be250 808ebec0 800519b8 808c8b38 00000000 00000000
[175330.110114] fda0: 00000001 8f406000 90003100 8f68a140 8f75fdd4 808bb894 00000017 8001a268
[175330.118463] fdc0: 00000004 8f75fe90 8f402000 8f74a800 8f75fe8c 8f75fde0 80009290 8001a274
[175330.126811] fde0: 9000210c 90002100 8f75fe28 90003100 8f75fe24 8f75fe00 80009400 8004dc04
[175330.135160] fe00: 808bf840 808bf848 8f75fe54 8f75fe18 8f68a170 8f68a140 0000d430 00000000
[175330.143508] fe20: 8f75fe5c 8f75fe30 80043598 8004348c 0000d430 00000000 8f402000 808bdd98
[175330.151857] fe40: 808bdd98 8f68a140 808bdd98 8f68a170 8f75fe84 8f75fe60 80043c18 80043218
[175330.160205] fe60: ffffffff 00000000 8f75fe9c 8003553c 00010093 ffffffff 8f75fec4 8f402000
[175330.168554] fe80: 8f75ff1c 8f75fe90 800142e0 80009258 8f74a800 8f40200c 8f402010 00000004
[175330.176902] fea0: 8f40200c 8f74a800 00000000 8f74a818 8f402000 8f402000 8f74a800 8f75ff1c
[175330.185251] fec0: 8f75ff20 8f75fee0 80036560 8003553c 00010093 ffffffff 00000051 80035440
[175330.193599] fee0: ffffe000 8f402014 8f402000 00000088 8f402000 8f402000 ffffe000 8f402014
[175330.201948] ff00: 8f74a818 00000088 8f402000 8f74a800 8f75ff5c 8f75ff20 80036560 8003551c
[175330.210297] ff20: 807bcc28 8f402160 808bf040 808eb49b 00000000 00000000 8f7497c0 8f75e000
[175330.218645] ff40: 8f74a800 800364e8 00000000 00000000 8f75ffac 8f75ff60 8003b1c4 800364f4
[175330.226993] ff60: 00000000 00000000 8003b0bc 8f74a800 00000000 8f75ff74 8f75ff74 00000001
[175330.235342] ff80: 8f75ff80 8f75ff80 80040a14 8f7497c0 8003b0bc 00000000 00000000 00000000
[175330.243690] ffa0: 00000000 8f75ffb0 8000fa78 8003b0c8 00000000 00000000 00000000 00000000
[175330.252038] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[175330.260386] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[175330.268719] Backtrace:
[175330.271321] [<8003b52c>] (kthread_data) from [<80036c14>] (wq_worker_sleeping+0x14/0xc0)
[175330.279596] [<80036c00>] (wq_worker_sleeping) from [<80648894>] (__schedule+0x250/0x510)
[175330.287841] r5:808bdd98 r4:8f68a140
[175330.291582] [<80648644>] (__schedule) from [<80648b98>] (schedule+0x44/0xa4)
[175330.298757] r10:807b8528 r9:8f68a308 r8:00000000 r7:8f75fcc8 r6:8f450000 r5:8f75fae4
[175330.306814] r4:ffffe000
[175330.309497] [<80648b54>] (schedule) from [<80022f38>] (do_exit+0x5bc/0x904)
[175330.316591] r5:8f75fae4 r4:8f68a140
[175330.320335] [<8002297c>] (do_exit) from [<80013c28>] (die+0x36c/0x3cc)
[175330.326996] r7:00000001
[175330.329680] [<800138bc>] (die) from [<8001e89c>] (__do_kernel_fault.part.0+0x6c/0x1f0)
[175330.337758] r10:8f74a800 r9:8f402000 r8:00000004 r7:00000000 r6:00000017 r5:8f75fe90
[175330.345822] r4:00000004
[175330.348505] [<8001e830>] (__do_kernel_fault.part.0) from [<8001a610>] (do_bad_area+0x0/0x90)
[175330.357103] r7:00000017 r4:8f75fe90
[175330.360842] [<8001a268>] (do_page_fault) from [<80009290>] (do_DataAbort+0x44/0xc0)
[175330.368652] r10:8f74a800 r9:8f402000 r8:8f75fe90 r7:00000004 r6:8001a268 r5:00000017
[175330.376717] r4:808bb894
[175330.379393] [<8000924c>] (do_DataAbort) from [<800142e0>] (__dabt_svc+0x40/0x60)
[175330.386948] Exception stack(0x8f75fe90 to 0x8f75fed8)
[175330.392132] fe80: 8f74a800 8f40200c 8f402010 00000004
[175330.400482] fea0: 8f40200c 8f74a800 00000000 8f74a818 8f402000 8f402000 8f74a800 8f75ff1c
[175330.408828] fec0: 8f75ff20 8f75fee0 80036560 8003553c 00010093 ffffffff
[175330.415575] r8:8f402000 r7:8f75fec4 r6:ffffffff r5:00010093 r4:8003553c
[175330.422493] [<80035510>] (process_one_work) from [<80036560>] (worker_thread+0x78/0x564)
[175330.430738] r10:8f74a800 r9:8f402000 r8:00000088 r7:8f74a818 r6:8f402014 r5:ffffe000
[175330.438802] r4:8f402000
[175330.441483] [<800364e8>] (worker_thread) from [<8003b1c4>] (kthread+0x108/0x110)
[175330.449039] r10:00000000 r9:00000000 r8:800364e8 r7:8f74a800 r6:8f75e000 r5:8f7497c0
[175330.457096] r4:00000000
[175330.459777] [<8003b0bc>] (kthread) from [<8000fa78>] (ret_from_fork+0x14/0x3c)
[175330.467158] r8:00000000 r7:00000000 r6:00000000 r5:8003b0bc r4:8f7497c0
[175330.474073] Code: e1a0c00d e92dd800 e24cb004 e5903220 (e5130010)
[175330.480301] —[ end trace 644d96da60d71751 ]—
[175330.485041] Fixing recursive fault but reboot is needed!
[175330.490492] Unable to handle kernel paging request at virtual address fffffff0
[175330.497871] pgd = 80004000
[175330.500701] [fffffff0] *pgd=8fdf6861, *pte=00000000, *ppte=00000000
[175330.507148] Internal error: Oops: 37 [#3] ARM

Can some one let me know how can I get it fixed and make VF61 work smothly without crash

Thannks in advance,
Lad Dhawal Umesh

Hi @nikunjp26

Could you provide the software and hardware versions of your setup?
Is this a custom image or toradex evaluation image downloaded from our website?
Also, provide the steps to reproduce the issue at our end.

Hardware Version : V1.2A Col. VF61 256 MB IT
Software : Yocto with rocko branch
Image Type: Custom
Issue has been reproduced with different ways as follows:

  • When continiously the device gets reboot
  • When any external interface attached to device
  • Any time when device is in running mode

Hi @nikunjp26

I checked with the latest BSP v2.8b7 and was not able to reproduce it. Could you share the custom image here [support.in@toradex.com]

The issue has been resolved by changing the VF61 chipset

But, Can any let me know how can we trace back the issue occurance in the device.
Basically not identified why the issue has been resolved

Thanks,
Lad Dhawal Umesh