MIPI CSI-2 Camera Gstreamer error - Virtual channels with TP2855

[sjjin]

ear all,

i’m working with the following configuration:
Apalis iMX8QM 4GB WB IT V1.1D
Ixora Carrier Board V 1.2A
Linux BSP 6.3.0

I would like to improve the ability to load video via MIPI CSI-2.
When you run the MIPI CSI-2 driver, a capture sub device is created, but /dev/videoX is not created.

I got the following results:

# v4l2-ctl --list-devices
amphion vpu decoder (platform: amphion-vpu):
        /dev/video0
        /dev/video1
        /dev/media0

mxc-jpeg codec (platform:58400000.jpegdec):
        /dev/video2

mxc-jpeg codec (platform:58450000.jpegenc):
        /dev/video3

FSL Capture Media Device (platform:mxc-md):
        /dev/media1

# ls -al /dev/v4l-subdev*
crw-rw---- 1 root video 81, 2 Feb 23 06:33 /dev/v4l-subdev0
crw-rw---- 1 root video 81, 3 Feb 23 06:33 /dev/v4l-subdev1

# media-ctl -d /dev/media1 -p
Media controller API version 5.15.77

Media device information
------------------------
driver          mxc-md
model           FSL Capture Media Device
serial
bus info
hw revision     0x0
driver version  5.15.77

Device topology
- entity 1: mxc-mipi-csi2.0 (8 pads, 4 links)
            type Node subtype V4L flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                <- "tp2855_mipi 5-0044":0 [ENABLED,IMMUTABLE]
        pad1: Sink
                <- "tp2855_mipi 5-0044":1 [ENABLED,IMMUTABLE]
        pad2: Sink
                <- "tp2855_mipi 5-0044":2 [ENABLED,IMMUTABLE]
        pad3: Sink
                <- "tp2855_mipi 5-0044":3 [ENABLED,IMMUTABLE]
        pad4: Source
        pad5: Source
        pad6: Source
        pad7: Source

- entity 10: tp2855_mipi 5-0044 (4 pads, 4 links)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev1
        pad0: Source
                [fmt:YUYV8_1X16/1920x1080 field:none colorspace:jpeg]
                -> "mxc-mipi-csi2.0":0 [ENABLED,IMMUTABLE]
        pad1: Source
                -> "mxc-mipi-csi2.0":1 [ENABLED,IMMUTABLE]
        pad2: Source
                -> "mxc-mipi-csi2.0":2 [ENABLED,IMMUTABLE]
        pad3: Source
                -> "mxc-mipi-csi2.0":3 [ENABLED,IMMUTABLE]

Can someone help me with this, pls?
Best regards,

[alvaro.tx]

What camera are you using? Do you know if the driver is being recognized?

[sjjin]

Hi @alvaro.tx,

We use AHD camera.

AHD camera input was converted to mipi-csi2 through TP2855 (AHD Decoder).

I was able to create /dev/video2 node in mxc_isi.4.capture.

- entity 18: mxc_isi.4.capture (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video2
        pad0: Sink
                <- "mxc_isi.4":12 [ENABLED]

However, when I tried to play it with gstreamer, it stopped in the following state:

# gst-launch-1.0 v4l2src device=/dev/video2 ! autovideosink
[ 1337.132101] usb 3-1.2: reset high-speed USB device number 4 using ci_hdrc
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

Can you help me solve this problem?

Best regards,
sjjin

[alvaro.tx]

Hi @sjjin , I’m still not convinced that the driver is properly recognized. How did you add the TP2855 driver? Can you share the device tree and kernel config?

Thanks,
Alvaro.

[sjjin]

Hi @alvaro.tx,

Thank you for answer.
I will provide the tp2855 device tree:

/* Apalis I2C3 (CAM) */                                                    
&i2c3 {                                                                    
    #address-cells = <1>;                                                  
    #size-cells = <0>;                                                     
    status = "okay";                                                       
                                                                           
    tp2855_mipi@44  {                                                      
        compatible = "techpoint,tp2855_mipi";                              
        reg = <0x44>;                                                      
        pinctrl-names = "default";                                         
        pinctrl-0 = <&pinctrl_gpio1>;                                      
        clocks = <&clk_dummy>;                                             
        clock-names = "capture_mclk";                                      
        mclk = <27000000>;                                                 
        mclk_source = <0>;                                                 
        mipi_csi;                                                          
        rst-gpios = <&lsio_gpio0 8 GPIO_ACTIVE_HIGH>;   /* Apalis GPIO1 */ 
        virtual-channel;                                                   
        status = "okay";                                                   
        port {                                                             
            tp2855_1_ep: endpoint {                                        
                remote-endpoint = <&mipi_csi1_ep>;                         
                data-lanes = <1 2 3 4>;                                    
            };                                                             
        };                                                                 
    };                                                                     
};                                                                         
                                                                           
&isi_4 {                                                                   
    status = "okay";                                                       
                                                                           
    cap_device {                                                           
        status = "okay";                                                   
    };                                                                     
};                                                                         
                                                                           
                                                                           
&mipi_csi_1 {                                                              
    #address-cells = <1>;                                                  
    #size-cells = <0>;                                                     
    status = "okay";                                                       
    virtual-channel;                                                       
                                                                           
    /* Camera 1  MIPI CSI-2 (CSIS1) */                                     
    port@1 {                                                               
        reg = <1>;                                                         
        mipi_csi1_ep: endpoint {                                           
            remote-endpoint = <&tp2855_1_ep>;                              
            data-lanes = <1 2 3 4>;                                        
        };                                                                 
    };                                                                     
};                                                                         

The tp2855 provides OSC 27MHz clk.

Can you help me solve this problem?

Kind regards,
sjjin

[alvaro.tx]

Dear @sjjin , please send the following:

• Output of dmesg
• Output of lsmod
• Output of zcat /proc/config.gz
• Output of ls -l /dev/

Thanks,
Alvaro.

[sjjin]

Hi @alvaro.tx,

Thank you for answer.
We will respond to your requests(Attached as a txt file):

• Output of dmesg
  → dmeg.txt (39.9 KB)

• Output of lsmod
  → lsmod.txt (1.4 KB)

• Output of zcat /proc/config.gz
  → config.gz.txt (198.8 KB)

• Output of ls -l /dev/
  → ls.txt (15.7 KB)

Do you have any additional requests for debugging?

Best regards,
sjjin

[alvaro.tx]

Hi @sjjin, sorry for the delay.

Well, the TP2855 is there according to the Kernel configuration. Do you have any information or reference on how this was added? This is not in our default BSP and I couldn’t find any information in Google about a Linux driver.

When you run the MIPI CSI-2 driver, a capture sub device is created, but /dev/videoX is not created.

I could see 6 video devices (video0-5) so I understand that this is not a problem anymore. Do you get the same response.

gst-launch-1.0 v4l2src device=/dev/video2 ! autovideosink

Checking the NXP Linux User Guide for 5.15 (Used in BSP 6.3.0), this is the correct Gstreamer pipeline for a TV decoder however, I don’t think this is your usecase. Have you tried any other pipelines for Webcam preview like the following?

gst-launch-1.0 v4l2src device=/dev/video2 ! 'video/x-raw, format=(string)UYVY,width=640,height=480,framerate=(fraction)30/1' ! autovideosink

Also, pls check the output for gst-inspect-1.0 v4l2src

Apalis iMX8QM 4GB WB IT V1.1D
Ixora Carrier Board V 1.2A
Linux BSP 6.3.0

Are you using our Multimedia Image 6.3.0? Do you have a HDMI display connected? What do you see in the screen?

Kind regards,
Alvaro.

[sjjin]

Hi @alvaro.tx, Thank you for answer.

Well, the TP2855 is there according to the Kernel configuration. Do you have any information or reference on how this was added? This is not in our default BSP and I couldn’t find any information in Google about a Linux driver.

1. Permissions are limited, but you can find it on the NXP Community:

• i.mx8x ADAS support - NXP Community

2. We can find information the kernel driver for the TP2855 at rockchip. share Git hub:

• https://github.com/armbian/linux-rockchip/blob/rk-5.10-rkr5.1/drivers/media/i2c/tp2855.c
• https://github.com/armbian/linux-rockchip/blob/rk-5.10-rkr5.1/drivers/media/i2c/techpoint/techpoint_tp2855.c

I could see 6 video devices (video0-5) so I understand that this is not a problem anymore. Do you get the same response.

1. You are right.

Checking the NXP Linux User Guide for 5.15 (Used in BSP 6.3.0), this is the correct Gstreamer pipeline for a TV decoder however, I don’t think this is your usecase. Have you tried any other pipelines for Webcam preview like the following?

1. I also tried other pipelines.
   However, when I tried to play it with gstreamer, it stopped in the following state:

# gst-launch-1.0 v4l2src device=/dev/video2 ! 'video/x-raw, format=(string)UYVY,width=1920,height=1080,framerate=(fraction)30/1' ! autovideosink
[ 1835.096744] usb 3-1.2: reset high-speed USB device number 3 using ci_hdrc
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

Also, pls check the output for gst-inspect-1.0 v4l2src

1. results of gst-inspect-1.0 v4l2src:
   gst-inspect-1.0 v4l2src.txt (10.8 KB)

Are you using our Multimedia Image 6.3.0? Do you have a HDMI display connected? What do you see in the screen?

1. I am using Multimedia Image 6.3.0.

TDX Wayland with XWayland 6.3.0-devel-20230920070229+build.0 (kirkstone) apalis-imx8-14759928 ttyLP1
Apalis-iMX8_Reference-Multimedia-Image

apalis-imx8-14759928 login:

2. HDMI is connected to the monitor.
3. The Weston screen is displayed. Share as a photo file:
   
   

Additionally, when I check v4l2-ctl --list-formats-ext -d /dev/video2, Interval: Discrete nans (nan fps) is repeated:

# v4l2-ctl --list-formats-ext -d /dev/video2
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'RGBP' (16-bit RGB 5-6-5)
                Size: Discrete 0x0
                        Interval: Discrete nans (nan fps)
                        Interval: Discrete nans (nan fps)
                        Interval: Discrete nans (nan fps)
                        Interval: Discrete nans (nan fps)
                        Interval: Discrete nans (nan fps)

Can you help me solve this problem?

Best regards,
sjjin

[alvaro.tx]

Hi @sjjin , thanks for all the information. Based on the logs, I think you have a USB/UVC camera. Does Gstreamer work for this USB camera? That should be very straightforward.

Kindly confirm.

The TP2855 seems to be correctly identified so my suspicion is with the Gstreamer pipeline. Let’s confirm that the USB camera is working.

[sjjin]

Hi @alvaro.tx, Thank you for answer.

I got USB/UVC information using v4l2-ctl --list-devices:

UVC Camera (046d:0825) (usb-ci_hdrc.1-1.2):
        /dev/video5
        /dev/video6
        /dev/media2

The video was confirmed normally when played with gstreamer.

# gst-launch-1.0 v4l2src device=/dev/video5 ! autovideosink
[  211.928876] usb 3-1.2: reset high-speed USB device number 3 using ci_hdrc
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...
0:00:04.4 / 99:99:99.

Do you have any additional requests for debugging?

Best regards,
sjjin

[alvaro.tx]

Hi @sjjin , sorry for the delay on this. I think we have everything needed for debugging. My suspicion is that the device tree is not correctly configured. I’m currently checking more in detail. Let me get back to you with any update.

FYI I’m checking ADV7280 implementation, since should be fairly similar.

Alvaro.

[sjjin]

Hi @alvaro.tx, Thank you for answer.

Please share any additional requests and updates.

Best regards,
sjjin

[sjjin]

Hi @alvaro.tx, You must be busy Are there any updates?

modifying the tp2855 driver, I got the following result:

# v4l2-ctl --list-formats-ext -d /dev/video2
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'RGBP' (16-bit RGB 5-6-5)
        [1]: 'RGB3' (24-bit RGB 8-8-8)
        [2]: 'BGR3' (24-bit BGR 8-8-8)
        [3]: 'YUYV' (YUYV 4:2:2)
        [4]: 'YUV4' (32-bit A/XYUV 8-8-8-8)
        [5]: 'NV12' (Y/CbCr 4:2:0)
        [6]: 'NM12' (Y/CbCr 4:2:0 (N-C))
        [7]: 'YM24' (Planar YUV 4:4:4 (N-C))
        [8]: 'XR24' (32-bit BGRX 8-8-8-8)
        [9]: 'AR24' (32-bit BGRA 8-8-8-8)

However, when playing with gstreamer, the video is not output.

# gst-launch-1.0 v4l2src device=/dev/video2 ! 'video/x-raw,width=1920,height=1080,framerate=(fraction)30/1' ! autovideosink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[  188.765976] bypass csc
[  188.768397] input fmt YUV4
[  188.771104] output fmt YUYV

Can you help me solve this problem?

Best regards,
sjjin

[alvaro.tx]

Sorry @sjjin , unfortunately no updates from my side… Any update from yours?

We don’t have any tp2855 here to further debug or test, so we cannot help you much directly apart from pointing you to similar devices like the AR0521 or ADV7280 for reference. We can present you a development partner if you need some further help.

Alvaro.

[sjjin]

Hi @alvaro.tx,We don’t have any updates…

We were able to confirm MIPI-CSI2 video output using AR1335.
However, our ultimate goal is to convert the input from 4 AHD cameras to MIPI-CSI2 and supply it to the carrier board to view the video through virtual channels.

Do you have any experience with this? In the case of ADV7280M, it appears to provide a 1-lane MIPI CSI-2 output, but is it possible to achieve virtual channel 4xHD@30 output?

Best regards,
sjjin

[alvaro.tx]

Hi @sjjin , I see. Virtual channels are a whole different beast. From Toradex we don’t add anything special on that so one should just use whatever NXP uses. i.MX8QM should be able to handle 4x 1080p@30 given that 4k@30 is supported clock-wise (AR1335) (For uncompressed UYVY Output Format)

I personally don’t have any experience with virtual channels. I asked my colleagues.

BTW I just found this, hopefully it gives some idea: https://community.nxp.com/t5/i-MX-Processors/iMX8QM-MIPI-CSI2-virtual-channel-setting/m-p/1731714

It uses max9286, but you may be able to apply how the virtual channel is defined in the device tree.

Kind regards,
Alvaro.

[sjjin]

Hi @alvaro.tx,

Thank you for sharing the information.

Best regards,
sjjin

[alvaro.tx]

Hi @sjjin thanks for your email the other day, I believe you got this working. Can you share the device tree and the Gstreamer pipeline you used at the end? This would provide great value to other users.

Thanks a lot,
Alvaro.

[sjjin]

Hi @alvaro.tx ,

The GStreamer pipeline I used is as follows:

// 1 channel
gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-raw,width=1920,height=1080,framerate=30/1 ! fpsdisplaysink video-sink=waylandsink text-overlay=false sync=false -v
// 4 channel
gst-launch-1.0 imxcompositor_g2d name=comp \
  sink_0::xpos=0 sink_0::ypos=0 sink_0::width=960 sink_0::height=540 \
  sink_1::xpos=0 sink_1::ypos=540 sink_1::width=960 sink_1::height=540 \
  sink_2::xpos=960 sink_2::ypos=0 sink_2::width=960 sink_2::height=540 \
  sink_3::xpos=960 sink_3::ypos=540 sink_3::width=960 sink_3::height=540 ! \
  video/x-raw,framerate=30/1 ! fpsdisplaysink video-sink=waylandsink text-overlay=false sync=false -v \
  v4l2src device=/dev/video2 ! video/x-raw,width=1920,height=1080,framerate=30/1 ! comp.sink_0 \
  v4l2src device=/dev/video3 ! video/x-raw,width=1920,height=1080,framerate=30/1 ! comp.sink_1 \
  v4l2src device=/dev/video4 ! video/x-raw,width=1920,height=1080,framerate=30/1 ! comp.sink_2 \
  v4l2src device=/dev/video5 ! video/x-raw,width=1920,height=1080,framerate=30/1 ! comp.sink_3

The device tree I used is as follows:

// SPDX-License-Identifier: GPL-2.0-or-later OR MIT
/*
 * Copyright 2022 Toradex
 */

/dts-v1/;
/plugin/;

#include <dt-bindings/gpio/gpio.h>

/ {
        compatible = "toradex,apalis-imx8";
};

/* Apalis I2C3 (CAM) */
&i2c3 {
        #address-cells = <1>;
        #size-cells = <0>;
        status = "okay";

        tp2855_mipi@44 {
                compatible = "techpoint,tp2855_mipi";
                reg = <0x44>;
                pinctrl-names = "default";
                pinctrl-0 = <&pinctrl_gpio1>;
                clocks = <&clk_dummy>;
                clock-names = "capture_mclk";
                mclk = <27000000>;
                mclk_source = <0>;
                rst-gpios = <&lsio_gpio0 8 GPIO_ACTIVE_HIGH>;   /* Apalis GPIO1 */
                virtual-channel;
                status = "okay";

                port {
                        tp2855_1_ep: endpoint {
                                remote-endpoint = <&mipi_csi1_ep>;
                                data-lanes = <1 2 3 4>;
                        };
                };
        };
};

/* <Input MIPI_VCx Output>
Input:  0-DC0, 1-DC1, 2-MIPI CSI0, 3-MIPI CSI1, 4-HDMI, 5-MEM
VCx:    0-VC0, 1-VC1, 2-VC2, 3-VC3, MIPI CSI only
Output: 0-DC0, 1-DC1, 2-MEM */
&isi_4 {
        interface = <3 0 2>;
        status = "okay";

        cap_device {
                status = "okay";
        };
};

&isi_5 {
        interface = <3 1 2>;
        status = "okay";

        cap_device {
                status = "okay";
        };
};

&isi_6 {
        interface = <3 2 2>;
        status = "okay";

        cap_device {
                status = "okay";
        };
};

&isi_7 {
        interface = <3 3 2>;
        status = "okay";

        cap_device {
                status = "okay";
        };
};

&mipi_csi_1 {
        #address-cells = <1>;
        #size-cells = <0>;
        virtual-channel;
        status = "okay";

        /* Camera 1 MIPI CSI-2 (CSIS0) */
        port@1 {
                reg = <1>;

                mipi_csi1_ep: endpoint {
                        remote-endpoint = <&tp2855_1_ep>;
                        data-lanes = <1 2 3 4>;
                };
        };
};

I referred to the NXP Community for the device tree I provided last time:

iMX8QM MIPI CSI2 virtual channel setting

Best regards,
sjjin