IMX219 Not Linking with MIPI-CSI on i.MX8MP

[Hardik]

Hello,

I am currently working on integrating the IMX219 camera sensor with the Verdin i.MX8MP board. I have added the required device tree configuration as shown below.

DTS Configurations:

&i2c3 {
        status = "okay";
        
        imx219: imx219@10 {
                compatible = "sony,imx219";
                reg = <0x10>;
                status = "okay";

                pinctrl-names = "default";
                pinctrl-0 = <&pinctrl_gpio_hog3>;

                clock-names = "xclk";
                clocks = <&clk IMX8MP_CLK_IPP_DO_CLKO2>;
                assigned-clocks = <&clk IMX8MP_CLK_IPP_DO_CLKO2>;
                assigned-clock-parents = <&clk IMX8MP_CLK_24M>;
                assigned-clock-rates = <24000000>;
                mclk = <24000000>;
                mclk_source = <0>;
                csi_id = <1>;
                mipi-csi;

                reset-gpios = <&gpio1 1 GPIO_ACTIVE_HIGH>;

                #address-cells = <1>;
                #size-cells = <0>;

                port@0 {
                    reg = <0>;
                    imx219_to_mipi_csi2: endpoint {
                        remote-endpoint = <&mipi_csi_1_in>;
                        bus-type = <4>;
                        clock-lanes = <0>;
                        data-lanes = <1 2>;
                        link-frequencies = /bits/ 64 <456000000>;

                        status = "okay";
                    };
               };
        };
};

&cameradev {
        status = "okay";
};

&mipi_csi_1 {
        #address-cells = <1>;
        #size-cells = <0>;
        status = "okay";

        port@0 {
                reg = <0>;
                mipi_csi_1_in: endpoint {
                        remote-endpoint = <&imx219_to_mipi_csi2>;
                        data-lanes = <1 2>;
                        csis-hs-settle = <13>;
                        csis-clk-settle = <2>;
                        csis-wclk;
                };
        };
};

&isp_1 {
        status = "okay";
};

&dewarp {
        status = "okay";
};

Issue: Although the camera sensor sub-device and MIPI-CSI node are created, the media link is not established. Below are the relevant kernel logs:

[    9.306696] mxc-md 32c00000.bus:camera: deferring csi device registration
[    9.365638] mxc-mipi-csi2-sam 32e50000.csi: supply mipi-phy not found, using dummy regulator
[    9.457142] mxc-mipi-csi2-sam 32e50000.csi: lanes: 1, hs_settle: 0, clk_settle: 0, wclk: 0, freq: 266000000
[    9.458979] mx8-img-md: Registered sensor subdevice: imx219 2-0010 (1)
[    9.458995] mxc-md 32c00000.bus:camera: mxc_md_create_links
[    9.524001] mxc-mipi-csi2.1: mipi_csis_imx8mp_phy_reset, No remote pad found!

media-ctl -p output:

Media device information
------------------------
driver          mxc-md
model           FSL Capture Media Device
serial          
bus info        platform:32c00000.bus:camera
hw revision     0x0
driver version  6.6.74

Device topology
- entity 1: mxc-mipi-csi2.1 (8 pads, 0 link)
            type Node subtype V4L flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
        pad1: Sink
        pad2: Sink
        pad3: Sink
        pad4: Source
        pad5: Source
        pad6: Source
        pad7: Source

- entity 10: imx219 2-0010 (1 pad, 0 link, 0 routes)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev1
        pad0: Source
                [stream:0 fmt:SRGGB10_1X10/3280x2464 field:none colorspace:raw xfer:none quantization:full-range
                 crop.bounds:(8,8)/3280x2464
                 crop:(8,8)/3280x2464]

It appears the media links between the IMX219 sensor and MIPI CSI interface are not being established. Could you please help me identify what might be missing or incorrectly configured?

Best Regards,
Hardik

[ToradexAI]

Hi, I’m the Toradex AI Assistant. The Toradex Support team will follow up with you shortly!

Did you like my answer?

---

Hello!

When working with the MIPI CSI-2 interface on the Verdin iMX8M Plus, there are several important aspects to consider:

1. MCLK Configuration: Ensure that the MCLK (Master Clock) is correctly set up, as this is crucial for the MIPI CSI-2 interface to function properly. You can find more details on this in the Toradex documentation here.

2. Camera Interface Verification:

   • The NXP i.MX 8M Plus supports a quad-lane MIPI CSI-2 interface, which is essential for high-speed data transfer.
   • It supports various color space definitions, including RGB, YUV, YCbCr, and RAW, and can handle 24-bit down to 8-bit per pixel formats. More information is available here.

3. I2C Configuration:

   • The Verdin standard allocates specific I2C ports for the MIPI CSI-2 camera. Ensure you are using the I2C_4_CSI (I2C3) for camera communication. Details can be found here.

4. Community Insights:

   • Some users have found that adding delays to the CSI enable sequence can resolve timing-related issues. Additionally, you might need to adjust power state initialization settings. You can read more about these experiences here.

To summarize, I recommend:

• Verifying the use of the correct I2C bus (I2C_4_CSI/I2C3) for camera communication.
• Reviewing the MCLK configuration and clock settings.
• Considering the addition of initialization delays if the camera unexpectedly enters a low power state.

If you need further assistance with debugging MIPI CSI issues, feel free to ask!

Best regards,
Toradex AI Assistant.

[Hardik]

Hello,

To address the issue of the link not being established between the IMX219 camera and the MIPI CSI node, I implemented the following changes in the IMX219 driver:

• Added the s_power() function to manage the power state of the sensor.
• Introduced the link_setup() function to support the media controller linking between the IMX219 sensor and the MIPI-CSI node.
  patch file :

diff --git a/drivers/media/i2c/imx219.c b/drivers/media/i2c/imx219.c
index d8cb1577207c..c64ebb54f5bc 100644
--- a/drivers/media/i2c/imx219.c
+++ b/drivers/media/i2c/imx219.c
@@ -444,6 +444,7 @@ static const struct imx219_mode supported_modes[] = {
 struct imx219 {
        struct v4l2_subdev sd;
        struct media_pad pad;
+       struct device *dev;
 
        struct regmap *regmap;
        struct clk *xclk; /* system clock to IMX219 */
@@ -468,6 +469,8 @@ struct imx219 {
        /* Streaming on/off */
        bool streaming;
 
+       struct mutex lock;
+       int power_count;
        /* Two or Four lanes */
        u8 lanes;
 };
@@ -1028,7 +1031,42 @@ static int imx219_identify_module(struct imx219 *imx219)
        return 0;
 }
 
+static int imx219_s_power(struct v4l2_subdev *sd, int on)
+{
+       struct imx219 *sensor = to_imx219(sd);
+       int ret = 0;
+
+       mutex_lock(&sensor->lock);
+
+       /*
+       * If the power count is modified from 0 to != 0 or from != 0 to 0,
+       * update the power state.
+       */
+       if (sensor->power_count == !on) {
+               if (!!on) {
+                       imx219_power_on(sensor->dev);
+               }
+               else{
+                       imx219_power_off(sensor->dev);
+               }
+       }
+
+       /* Update the power count. */
+       sensor->power_count += on ? 1 : -1;
+       WARN_ON(sensor->power_count < 0);
+
+       mutex_unlock(&sensor->lock);
+
+       if (on && !ret && sensor->power_count == 1) {
+               /* restore controls */
+               ret = v4l2_ctrl_handler_setup(&sensor->ctrl_handler);
+       }
+       return ret;
+}
+
+
 static const struct v4l2_subdev_core_ops imx219_core_ops = {
+        .s_power = imx219_s_power,
        .subscribe_event = v4l2_ctrl_subdev_subscribe_event,
        .unsubscribe_event = v4l2_event_subdev_unsubscribe,
 };
@@ -1052,6 +1090,16 @@ static const struct v4l2_subdev_ops imx219_subdev_ops = {
        .pad = &imx219_pad_ops,
 };
 
+static int imx219_link_setup(struct media_entity *entity,
+                             const struct media_pad *local,
+                             const struct media_pad *remote, u32 flags)
+{
+    return 0;
+}
+
+static const struct media_entity_operations imx219_media_ops = {
+        .link_setup = imx219_link_setup,
+};
 
 static unsigned long imx219_get_pixel_rate(struct imx219 *imx219)
 {
@@ -1233,12 +1281,15 @@ static int imx219_probe(struct i2c_client *client)
        struct device *dev = &client->dev;
        struct imx219 *imx219;
        int ret;
+
        printk(KERN_INFO "IMX219 : imx219_probe success\n");
 
        imx219 = devm_kzalloc(&client->dev, sizeof(*imx219), GFP_KERNEL);
        if (!imx219)
                return -ENOMEM;
 
+       imx219->dev = &client->dev;
+
        printk(KERN_INFO "IMX219 : devm_kzalloc success\n");
        v4l2_i2c_subdev_init(&imx219->sd, client, &imx219_subdev_ops);
 
@@ -1332,6 +1383,7 @@ static int imx219_probe(struct i2c_client *client)
 
        /* Initialize source pad */
        imx219->pad.flags = MEDIA_PAD_FL_SOURCE;
+       imx219->sd.entity.ops = &imx219_media_ops;
 
        ret = media_entity_pads_init(&imx219->sd.entity, 1, &imx219->pad);
        if (ret) {

DTS configuration :

&i2c3 {
        status = "okay";

        imx219: imx219@10 {
                compatible = "sony,imx219";
                reg = <0x10>;
                status = "okay";

                pinctrl-names = "default";
                pinctrl-0 = <&pinctrl_gpio_hog3>,<&pinctrl_gpio2>;

                clock-names = "xclk";
                clocks = <&clk IMX8MP_CLK_IPP_DO_CLKO2>;
                assigned-clocks = <&clk IMX8MP_CLK_IPP_DO_CLKO2>;
                assigned-clock-parents = <&clk IMX8MP_CLK_24M>;
                assigned-clock-rates = <24000000>;
                mclk = <24000000>;
                mclk_source = <0>;
                csi_id=<0>;

                reset-gpios = <&gpio1 1 GPIO_ACTIVE_HIGH>;

                port {
                    imx219_to_mipi_csi2: endpoint {
                        remote-endpoint = <&mipi_csi_1_in>;
                        clock-lanes = <0>;
                        data-lanes = <1 2>;
                        link-frequencies = /bits/ 64 <456000000>;
                    };
               };
        };
};

&cameradev {
        status = "okay";
};

&mipi_csi_0 {
        #address-cells = <1>;
        #size-cells = <0>;
        status = "okay";
        port@0 {
                reg = <0>;
                mipi_csi_1_in: endpoint {
                        remote-endpoint = <&imx219_to_mipi_csi2>;
                        data-lanes = <2>;
                        csis-hs-settle = <13>;
                        csis-clk-settle = <2>;
                        csis-wclk;
                };
        };
};

&isi_0 {
       status = "okay";
       cap_device {
               status = "okay";
       };

       m2m_device{
               status = "disabled";
       };
};

Issue Encountered:

When running the camera using GStreamer (gst-launch) and v4l2 commands, I am seeing a green screen on the display.

I would appreciate any assistance or suggestions on how to resolve this issue.

Thanks & Regards,
Hardik

[Hardik]

Hi @matthijs & @marcioalbano.tx

I hope you’re both doing well.

Could you please assist me in resolving the issue described above? Your support would be greatly appreciated.

Thanks & Regards,
Hardik

[matthijs]

In the past, I’ve succesfully tested with the IMX219 on the Verdin IMX8MP. We then switched to another camera, so I am not entirely sure if what I’m posting below still works, but I suspect it should.

I found two parts in my notes:

• A DTS overlay to configure the IMX219.
• A modified version of the staging-imx driver, to fix the lack of the s_power and link_setup that you fixed in the driver, plus some other fixes (that might not be relevant for imx219)

Note that I did not patch the staging-imx driver in the kernel tree, but used torizoncore-builder to compile it as an out-of-tree additional driver, with a modified name so it can co-exist with the in-tree driver (but also needing a different compatible string in the DTS to select it).

The driver is here: FM23-kernel-modules/staging-imx at main · 3devo/FM23-kernel-modules · GitHub

The DTS overlay is:

// SPDX-License-Identifier: GPL-2.0-or-later OR MIT
/*
 * Copyright 2020-2021 e-consystems
 */

// CSI Camera Module 5MP AR0521 orderable at Toradex and e-consystems.

/dts-v1/;
/plugin/;

#include <dt-bindings/clock/imx8mp-clock.h>
#include <dt-bindings/gpio/gpio.h>

/ {
	compatible = "toradex,verdin-imx8mp";
};

&cameradev {
	status = "okay";
};

/* Verdin I2C_4_CSI */
&i2c3 {
	#address-cells = <1>;
	#size-cells = <0>;
	status = "okay";

	imx219@10 {
		compatible = "sony,imx219";
		reg = <0x10>;
		status = "okay";

		/*
		Sensor board has its own crystal (and
		toradex-to-rpi-adapter-board does not have have the clock
		pin forwarded in any case), but the driver insists to
		have an xclk. This should probably refer to a dummy
		clock then, but this just sets up an actual clock.
		*/

		clocks = <&clk IMX8MP_CLK_IPP_DO_CLKO2>;
		clock-names = "xclk";
		/*

		assigned-clocks = <&clk IMX8MP_CLK_IPP_DO_CLKO2>;
		assigned-clock-parents = <&clk IMX8MP_CLK_24M>;
		assigned-clock-rates = <24000000>;
		*/

		/* All supplies are derived from the 3p3v on the sensor
		 * board. This should probably be modeled differently in
		 * DT, but this seems to work anyway. */
		VANA-supply = <&reg_3p3v>; /* 2.8v */
		VDIG-supply = <&reg_3p3v>; /* 1.8v */
		VDDL-supply = <&reg_3p3v>; /* 1.2v */

		port {
			imx219_mipi_0_ep: endpoint {
				clock-lanes = <0>;
				data-lanes = <1 2>;
				clock-noncontinuous;
				link-frequencies =
					/bits/ 64 <456000000>;
				//remote-endpoint = <&mipi_csi0_ep>;
			};
		};
	};
};

&isi_0 {
	status = "okay";

	cap_device {
		status = "okay";
	};

};

&mipi_csi_0 {
	#address-cells = <1>;
	#size-cells = <0>;
	status = "okay";
	csi = <0>; // https://community.nxp.com/t5/i-MX-Processors/Integrate-ADV7280-M-with-IMX8X/m-p/1661895/highlight/true#M206946

	port@0 {
		reg = <0>;

		mipi_csi0_ep: endpoint {
			/* NOTE: This is parsed by imx8-mipi-csi2-sam.c
			 * and uses different properties than the sensor
			 * endpoint...*/
			csis-hs-settle = <13>;
			csis-clk-settle = <2>;
			csis-wclk;
			data-lanes = <2>;
			clock-noncontinuous;
			remote-endpoint = <&imx219_mipi_0_ep>;
		};
	};
};

[Hardik]

Hi @matthijs

I tried exactly what you suggested, but I’m still getting a green screen.

I ran the following command:

gst-launch-1.0 v4l2src device=/dev/video2 ! 'video/x-raw,format=NV12, width=1920, height=1080, framerate=30/1' ! videoconvert ! autovideosink sync=false

Also, I found a post on the NXP community that says [1]:

It is not possible to perform Bayer to NV12 color conversion in the i.MX8MP ISI.
The reason is simple: There is no debayering logic in the ISI. Only the i.MX8MP embedded ISP features a debayer.

Link [1]:https://community.nxp.com/t5/i-MX-Processors/Capture-Image-Color-Format-Conversion-in-ISI-on-imx8mp/m-p/1584327

Can you please confirm if it’s possible to capture a visible (e.g., YUYV) image format using ISI?

[matthijs]

Right, the ISI indeed cannot do debayering, you need the ISP for that. I never really tried to get that working, because the kernel code for it had some non-standard EULA licensing stuff, needed shell scripts to set up things and was otherwise not very clear. Also, for our purposes, we did not really need RGB images, just the raw sensor images.

IIRC, what happens if you use the ISI and capture YUV or RGB or so images, then the ISI just assumes that the incoming images are already in that format (it can convert between YUV and RGB, but only if the sensor reports one and you report another, and I think the sensor reports some RAW format, so the ISI just falls back to pass-through mode, or maybe it does RGB->YUV conversion).

Pixels are passed through as-is, possibly with some reordering or shifting. In practice (looking back at my earlier notes), this means that if you read RGB, you will get pixels that look blue (because the RAW bits will be store as LSB’s, which end up in the blue field of an RGB pixel). In theory, this should mean that brighter pixels should overflow into the green field, but I think this did not happen in practice, because somehow the 10 bits from the sensor were compressed into 6 bits in the output.

Also looking back at my notes, I also observed a fully green image with YUV (or black with RGB), which seems to be what you get with zero or low pixel values. In my case, it turned out to be related to exposure settings - increasing gain and exposure time helped for me (also, making images with a lot of contrast, such as a window with daylight helped to confirm I was capturing something).

From my notes, the exposure settings can be controlled with:

v4l2-ctl -d /dev/v4l-subdev2 -c exposure=190,analogue_gain=921,digital_gain=256

These are the settings that we copied from the raspberry pi, with some (default or maybe tuned by us) libcamera script.

The kernel isi driver I linked also contains a commit/hack to enable RAW capture. This essentially lets the ISI driver announce it can emit RAW, and causes the ISI to be configured in passthrough (no YUV/RGB conversion) mode, which allows capturing the RAW frames with something like (I could not find any gstreamer commands in my notes, but that probably also supports it):

v4l2-ctl --device /dev/video2 --set-fmt-video=width=1920,height=1080,pixelformat=RG10,sizeimage=0,bytesperline=0 --stream-mmap --stream-to=/home/torizon/frame.raw --stream-count=1
convert -size 1920x1080 -depth 16 gray:frame.raw frame.png

In case it is useful, I’ve shared my notes (spanning a couple of months of work and exploration) here: Toradex Verdin IMX8MP camera support notes · GitHub That document collects my notes written while trying to get some different cameras (IMX219, IMX708, AR1335) working on the Toradex Verdin IMX8MP modules. They are mostly brain-dump / journal style, but maybe there is something in there that helps you.

[Hardik]

Hello @matthijs,

Thank you for sharing the detailed information. I truly appreciate your support and the time you took to help. Your guidance was very helpful.

Thanks & Regards,
Hardik