Apalis TK1 dual camera crashes

Hello,

I am using TK1 setup with two MIPI-CSI2 OV5640 cameras using Evaluation Board and Mezzanine Board. My main goal is to be able to capture images from both cameras simultaneously.

I’ve tested the setup with several different configurations in terms of resolution, formats etc. using both Gstreamer and OpenCV, however, crashes always occur whenever the pipeline stops and tries to release the cameras. The images are captured and saved correctly. Below you can find log produced after I interrupted, for example, the following command:

gst-launch v4l2src device=/dev/video0 ! 'video/x-raw-yuv,width=640,height=480' ! ffmpegcolorspace ! ximagesink v4l2src device=/dev/video1 ! 'video/x-raw-yuv,width=640,height=480' ! ffmpegcolorspace ! ximagesink

Error log:

Interrupt: Stopping pipeline ...

(gst-launch-0.10:890): GLib-CRITICAL **: Source ID 46 was not found when attempting to remove it
Execution ended after 2755087500 ns.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
[   94.371314] Unable to handle kernel NULL pointer dereference at virtual address 00000008
[   94.380537] pgd = ec4a4000
[   94.383895] [00000008] *pgd=abe8b003, *pmd=ad3fc003, *pte=00000000
[   94.391983] Internal error: Oops: 207 [#1] PREEMPT SMP ARM
[   94.397472] Modules linked in: ar0261_v4l2 soc_camera_platform adv7280 tc358743 ap1302 ar0330_v4l2 imx135_v4l2 tegra_camera ov5640 videobuf2_dma_contig joydev apalis_tk1_k20_can apalis_tk1_k20_adc gpio_apalis_tk1_k20 apalis_tk1_k20_ts bluetooth atmel_mxt_ts apalis_tk1_k20
[   94.421796] CPU: 0 PID: 890 Comm: gst-launch-0.10 Tainted: G        W    3.10.40-2.8.5+gccecdb3d6b3b #1
[   94.431183] task: eb401580 ti: ebef6000 task.ti: ebef6000
[   94.436588] PC is at kthread_stop+0x80/0x17c
[   94.440869] LR is at tegra_camera_stop_streaming+0x34/0x6c [tegra_camera]
[   94.447657] pc : [<c007c30c>]    lr : [<bf094710>]    psr: 60000013
[   94.447657] sp : ebef7d88  ip : ecc07a40  fp : c0524ae0
[   94.459122] r10: ed08f810  r9 : ed08f810  r8 : ed6a0a10
[   94.464346] r7 : 00000001  r6 : 00000000  r5 : 00000008  r4 : 00000000
[   94.470868] r3 : 00000000  r2 : c0b85684  r1 : 00000001  r0 : 00000000
[   94.477391] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   94.484521] Control: 30c5387d  Table: ac4a4000  DAC: fffffffd
[   95.229689] [<c007c30c>] (kthread_stop+0x80/0x17c) from [<bf094710>] (tegra_camera_stop_streaming+0x34/0x6c [tegra_camera])
[   95.240828] [<bf094710>] (tegra_camera_stop_streaming+0x34/0x6c [tegra_camera]) from [<c0532c6c>] (__vb2_queue_cancel+0x30/0x90)
[   95.252390] [<c0532c6c>] (__vb2_queue_cancel+0x30/0x90) from [<c0533c00>] (vb2_streamoff+0x84/0xb4)
[   95.261438] [<c0533c00>] (vb2_streamoff+0x84/0xb4) from [<c0536ba8>] (soc_camera_streamoff+0xb8/0xd0)
[   95.270661] [<c0536ba8>] (soc_camera_streamoff+0xb8/0xd0) from [<c0528308>] (__video_do_ioctl+0x20c/0x29c)
[   95.280314] [<c0528308>] (__video_do_ioctl+0x20c/0x29c) from [<c0527fd0>] (video_usercopy+0x288/0x3a0)
[   95.289625] [<c0527fd0>] (video_usercopy+0x288/0x3a0) from [<c0523704>] (v4l2_ioctl+0x68/0x11c)
[   95.298331] [<c0523704>] (v4l2_ioctl+0x68/0x11c) from [<c0145020>] (vfs_ioctl+0x30/0x44)
[   95.306424] [<c0145020>] (vfs_ioctl+0x30/0x44) from [<c01456d0>] (do_vfs_ioctl+0x80/0x538)
[   95.314690] [<c01456d0>] (do_vfs_ioctl+0x80/0x538) from [<c0145c54>] (SyS_ioctl+0xcc/0x168)
[   95.323045] [<c0145c54>] (SyS_ioctl+0xcc/0x168) from [<c000e5e0>] (ret_fast_syscall+0x0/0x30)
[   95.331570] Code: e3130002 0a000000 eb1b2df1 e2845008 (e1953f9f) 
[   95.344274] ---[ end trace f099d7e496e440dc ]---

Same problem occurs whenever filesinks are used or an OpenCV application gets to the moment the cameras are released.

Another type of error I’ve encountered so far is when I leave the two cameras for a longer period of time (tens of seconds) to keep capturing images continuously. Following logs were captured using an OpenCV application:

[   80.686599] vi vi.0: CSI_A syncpt timeout, syncpt = 1420, err = -11
[   80.692980] TEGRA_CSI_CSI_CIL_A_STATUS 0x00000011
[   80.697681] TEGRA_CSI_CSI_CILA_STATUS 0x00050050
[   80.702294] TEGRA_CSI_CSI_CIL_B_STATUS 0x00000000
[   80.706994] TEGRA_CSI_CSI_CIL_C_STATUS 0x00000001
[   80.711690] TEGRA_CSI_CSI_CIL_D_STATUS 0x00000000
[   80.716385] TEGRA_CSI_CSI_CIL_E_STATUS 0x00000000
[   80.721082] TEGRA_CSI_CSI_PIXEL_PARSER_A_STATUS 0x00000000
[   80.726559] TEGRA_CSI_CSI_PIXEL_PARSER_B_STATUS 0x00000000
[   80.732033] TEGRA_VI_CSI_0_ERROR_STATUS 0x00000000
[   80.736816] TEGRA_VI_CSI_1_ERROR_STATUS 0x00000000
[   80.741600] vi vi.0: Error capturing frame. Stopping capture

[   81.677566] vi vi.0: CSI_A syncpt timeout, syncpt = 1422, err = -11
[   81.683839] TEGRA_CSI_CSI_CIL_A_STATUS 0x00000011
[   81.688558] TEGRA_CSI_CSI_CILA_STATUS 0x00050050
[   81.693175] TEGRA_CSI_CSI_CIL_B_STATUS 0x00000000
[   81.697875] TEGRA_CSI_CSI_CIL_C_STATUS 0x00000001
[   81.702619] TEGRA_CSI_CSI_CIL_D_STATUS 0x00000000
[   81.707320] TEGRA_CSI_CSI_CIL_E_STATUS 0x00000000
[   81.712020] TEGRA_CSI_CSI_PIXEL_PARSER_A_STATUS 0x00000000
[   81.717498] TEGRA_CSI_CSI_PIXEL_PARSER_B_STATUS 0x00000000
[   81.722976] TEGRA_VI_CSI_0_ERROR_STATUS 0x00000000
[   81.727761] TEGRA_VI_CSI_1_ERROR_STATUS 0x00000000
[   81.732548] vi vi.0: Error capturing frame. Stopping capture

After this point, the cameras stop capturing images and when I try to close the application the same error as before occurs:

[  116.707840] Unable to handle kernel NULL pointer dereference at virtual address 00000008
[  116.715938] pgd = c0003000
[  116.718652] [00000008] *pgd=80000080004003, *pmd=00000000
[  116.724104] Internal error: Oops: 207 [#1] PREEMPT SMP ARM
[  116.729576] Modules linked in: ar0261_v4l2 soc_camera_platform adv7280 tc358743 ap1302 ar0330_v4l2 imx135_v4l2 tegra_camera ov5640 videobuf2_dma_contig joydev gpio_apalis_tk1_k20 apalis_tk1_k20_adc apalis_tk1_k20_can apalis_tk1_k20_ts bluetooth atmel_mxt_ts apalis_tk1_k20
[  116.753758] CPU: 3 PID: 879 Comm: fps-dual-cam Tainted: G        W    3.10.40-2.8.5+gccecdb3d6b3b #1
[  116.762870] task: ed786580 ti: ebef6000 task.ti: ebef6000
[  116.768259] PC is at kthread_stop+0x80/0x17c
[  116.772520] LR is at tegra_camera_stop_streaming+0x34/0x6c [tegra_camera]
[  116.779291] pc : [<c007c30c>]    lr : [<bf094710>]    psr: 60000113
[  116.779291] sp : ebef7dc8  ip : 00000000  fp : ec1c4b48
[  116.790743] r10: eb43d0b0  r9 : eb43d0b0  r8 : 00000008
[  116.795954] r7 : ec1c4b40  r6 : 00000000  r5 : 00000008  r4 : 00000000
[  116.802464] r3 : 00000000  r2 : c0b85684  r1 : 00000001  r0 : 00000000
[  116.808976] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  116.816094] Control: 30c5387d  Table: abc54000  DAC: 55555555
[  117.470360] [<c007c30c>] (kthread_stop+0x80/0x17c) from [<bf094710>] (tegra_camera_stop_streaming+0x34/0x6c [tegra_camera])
[  117.481474] [<bf094710>] (tegra_camera_stop_streaming+0x34/0x6c [tegra_camera]) from [<c0532c6c>] (__vb2_queue_cancel+0x30/0x90)
[  117.493017] [<c0532c6c>] (__vb2_queue_cancel+0x30/0x90) from [<c0535560>] (vb2_queue_release+0x1c/0x2c)
[  117.502393] [<c0535560>] (vb2_queue_release+0x1c/0x2c) from [<c0537364>] (soc_camera_close+0x74/0xe0)
[  117.511594] [<c0537364>] (soc_camera_close+0x74/0xe0) from [<c052359c>] (v4l2_release+0x38/0x70)
[  117.520362] [<c052359c>] (v4l2_release+0x38/0x70) from [<c01359a8>] (__fput+0xf0/0x1fc)
[  117.528352] [<c01359a8>] (__fput+0xf0/0x1fc) from [<c0078a78>] (task_work_run+0xbc/0xd8)
[  117.536426] [<c0078a78>] (task_work_run+0xbc/0xd8) from [<c005c8a4>] (do_exit+0x3d8/0x888)
[  117.544674] [<c005c8a4>] (do_exit+0x3d8/0x888) from [<c005dc3c>] (do_group_exit+0x60/0xbc)
[  117.552923] [<c005dc3c>] (do_group_exit+0x60/0xbc) from [<c006b5e0>] (get_signal_to_deliver+0x4d8/0x514)
[  117.562386] [<c006b5e0>] (get_signal_to_deliver+0x4d8/0x514) from [<c001186c>] (do_work_pending+0xf8/0x3e8)
[  117.572108] [<c001186c>] (do_work_pending+0xf8/0x3e8) from [<c000e620>] (work_pending+0xc/0x20)
[  117.580787] Code: e3130002 0a000000 eb1b2df1 e2845008 (e1953f9f) 
[  117.586897] ---[ end trace 3750526362d5fe76 ]---

[  117.599685] Fixing recursive fault but reboot is needed!

The cameras work fine when I use only one of them at the time and in that case I can successfully stop the pipeline/application. Although, even with single camera use I have also sometimes encountered “vi vi.0: CSI_A syncpt timeout” error after which the camera would stop working but at least kernel wouldn’t crash.

What exact BSP version are you talking about?

BSP 2.8b5 for Apalis TK1

Just to add another example. I have performed more test with gstreamer today everything works very well with a single camera either video0 or video1 with this pipeline:
gst-launch-0.10 v4l2src device=/dev/video0 num-buffers=100 ! ‘video/x-raw-yuv,format=(fourcc)UYVY,width=1920,height=1080’ ! jpegenc ! multifilesink location=img_%d.jpg

However, when I extend it to use two cameras simultaneously with this one:
gst-launch-0.10 v4l2src device=/dev/video0 num-buffers=100 ! ‘video/x-raw-yuv,format=(fourcc)UYVY,width=1920,height=1080’ ! jpegenc ! multifilesink location=img1_%d.jpg v4l2src device=/dev/video1 num-buffers=100 ! ‘video/x-raw-yuv,format=(fourcc)UYVY,width=1920,height=1080’ ! jpegenc ! multifilesink location=img2_%d.jpg

I get the kernel null pointer error as shown above.

In either case, all the images are captured successfully.

hi @AdamS

I can reproduce this Issue and we will discuss it internally.

Meanwhile, may I know, what is your application?
Why do you need to capture on two cameras at same time?
When do you want to have this issue fixed?

Best regards,
Jaski

Hey @jaski.tx

we are working on a mobile platform that will utilize multiple cameras primarily for machine vision (SLAM etc). At the moment, this issue does not strongly impact our workflow. It will probably still take a month or two before we move forward with the first test that would require actual hardware.

How would a timeline to fix this issue look like in your opinion?

Kind regards,
Adam

hi Adam

Thanks for your Input. We will discuss the timeline and come within this week to you.

Best regards,
Jaski

Hi @AdamS

A patch to fix the dual camera crashes Issue was added uploaded. You can get it from here.

Thanks and best regards,
Jaski

Hi @AdamS

Did that actually work for you?

Cheers
Marcel
Platform Manager Embedded Linux

Hi @jaski.tx @marcel.tx

thank you for the patch guys. I applied it to my image using a .bbappend file and rerun tests using Gstreamer pipelines. Unfortunately, the kernel still crashes the same way as before. The crash occurs whenever two cameras are in use at the same time at the moment when they are being closed.

Individually, cameras still appear to work just fine.

Did this fix work on your side?

Kind regards,

Adam

hi @AdamS

Yes, this worked perfectly on our side. Could you share your .bbappend file?

Hi @jaski.tx

my .bbappend file is placed inside: {OE-CORE}/layers/meta-custom/recipes/kernel/linux/
Content of the file:

FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI += "file://dual-camera-fix.patch"

I tried files named as: ‘linux-toradex_git.bbappend’ or ‘linux-toradex_3.10.40.bbappend’ (I am currently using 3.10.40 kernel)

My layer should be added/configured properly to bitbake as I have been able to add various libraries using custom recipes.

Thanks for the info. Could you first just compile the kernel from source code and check if issue is still happening?