De-interlacing video with i.MX6 IPU

aurata-hl · April 18, 2024, 1:33pm

We have an ADV7280A-M video decoder on a custom board connected via MIPI-CSI
to an Apalis iMX6 on Ixora carrier.

Current kernel is 6.1.80-rt26-6.6.0-devel+git.ca3cd16f344f, using
Yocto build.

The ADV receives video in PAL format, and we receive the video via
ipu1_csi0 and route through ipu1_ic_prpvf for upscaling to 1000x800.

When using the de-interlacer (I2P) on the ADV, this is actually working,
with the following resulting media-controller pad configuration:

'adv7180 1-0021':0  [fmt:UYVY8_2X8/720x576@1/25 field:none colorspace:smpte170m]
'imx6-mipi-csi2':0  [fmt:UYVY8_2X8/720x576 field:none]
'imx6-mipi-csi2':1  [fmt:UYVY8_2X8/720x576 field:none]
'ipu1_csi0_mux':0   [fmt:UYVY8_2X8/720x576 field:none]
'ipu1_csi0_mux':2   [fmt:UYVY8_2X8/720x576 field:none]
'ipu1_csi0':0       [fmt:UYVY8_2X8/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range 
             crop.bounds:(0,0)/720x576
             crop:(0,0)/720x576
             compose.bounds:(0,0)/720x576
             compose:(0,0)/720x576]
'ipu1_csi0':1       [fmt:AYUV8_1X32/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_ic_prp':0     [fmt:AYUV8_1X32/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_ic_prp':2     [fmt:AYUV8_1X32/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_ic_prpvf':0   [fmt:AYUV8_1X32/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_ic_prpvf':1   [fmt:AYUV8_1X32/1000x800@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]

Due to the poor quality of the I2P on the ADV, we need to use the the
motion-compensated de-interlacer in the IPU (ipu1_vdic).

It is surprisingly difficult to find a working configuration, and so far
we have not been successful.

Current in-kernel documentation (Documentation/admin/media/imx.rst)
contains an example using an adv7180, but uses field:seq-bt on the
adv7180 pad, but since driver commit 6457b6263f0f, only field:alternate
is allowed.

The only media-controller pad configuration we have come up with that
uses field:alternate and is not rejected before streaming is the following:

'adv7180 1-0021':0  [fmt:UYVY8_2X8/720x288@1/25 field:alternate colorspace:smpte170m]
'imx6-mipi-csi2':0  [fmt:UYVY8_2X8/720x288 field:alternate]
'imx6-mipi-csi2':1  [fmt:UYVY8_2X8/720x288 field:alternate]
'ipu1_csi0_mux':0   [fmt:UYVY8_2X8/720x288 field:alternate]
'ipu1_csi0_mux':2   [fmt:UYVY8_2X8/720x288 field:alternate]
'ipu1_csi0':0       [fmt:UYVY8_2X8/720x288@1/25 field:alternate colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range
             crop.bounds:(0,0)/720x576
             crop:(0,0)/720x576
             compose.bounds:(0,0)/720x576
             compose:(0,0)/720x576]
'ipu1_csi0':1       [fmt:AYUV8_1X32/720x576@1/25 field:seq-tb colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_vdic':0       [fmt:AYUV8_1X32/720x576@1/25 field:seq-tb colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_vdic':2       [fmt:AYUV8_1X32/720x576@1/50 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_ic_prp':0     [fmt:AYUV8_1X32/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_ic_prp':2     [fmt:AYUV8_1X32/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_ic_prpvf':0   [fmt:AYUV8_1X32/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_ic_prpvf':1   [fmt:AYUV8_1X32/1000x800@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]

When trying to stream data, however, we get a
VIDIOC_DQBUF: failed: Input/output error with the following logged
in dmesg:

ipu1_ic_prpvf: EOF timeout
ipu1_ic_prpvf: wait last EOF timeout

The only differences between the (broken) ipu1_vdic configuration and the
(working, but poor quality) I2P configuration further above are

Use of field:alternate with 720x288 where appropriate
Use of ipu1_vdic and hence a slightly longer route.
A strange interval of 1/50 on the output of ipu1_vdic, which is
however enforced by the driver

What’s wrong here?

rudhi.tx · April 25, 2024, 9:50am

Hello @aurata-hl,

Thanks for reaching out to the Toradex Community! I’m discussing this internally to find a solution. I’ll let you know as soon as I have an answer.

aurata-hl · April 25, 2024, 10:13am

Hello @rudhi.tx,

Thanks for looking into this issue, I appreciate it!

Best regards,

Henrik Lauritzen

rudhi.tx · April 25, 2024, 3:01pm

Hello @aurata-hl,

This topic is related more to the iMX6 SOC and therefore you will find more related discussions on the NXP Community. Could you please try post this question there instead?

I also found some similar topics there:

aurata-hl · April 30, 2024, 12:13pm

Hi @rudhi.tx,

Although you are right that our topic is somewhat related to the i.MX6 SoC,
it is also based on the upstream BSP (BSP 6) provided by Toradex.

The issues in the two links you provide are somewhat similar (though not
a full match) to ours, but are also related to NXP’s BSP, which - at the
relevant level (kernel + V4L2) is a rather different software stack.

The best-matching topic we can find at NXP is the following:

https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/De-interlace-Capture-Device/ta-p/1114964.

On closer inspection we find the following problems with that topic:

The patch provided relates only to NXP’s BSP (and a rather old version of
it), not the the upstream
driver in Toradex’ BSP 6.
The post states that “This code is not an official delivery and as such no guarantee of support for this code is provided by Freescale”.
The patch seems to have been used with mixed success by the people involved,
and there are multiple indications of image-quality problems
(which is the very issue we are trying to avoid here).
There are multiple reports of stability problems (which would be completely
unacceptable in our use-case) from the users who succeeded in using
the patch - many, many years ago.
The topic was closed by NXP without responding to several of the issues
mentioned.

In other words, it does not look like NXP is the right support forum for us,
given our use of Toradex’ BSP 6 (containing a mainline kernel and V4L2 stack).

The question then is, what is the correct support forum?

Best regards,

Henrik

stefan_e.tx · May 8, 2024, 2:22pm

Hi @aurata-hl

Unfortunately, this driver is still in staging therefore risks are high there are still some bugs in there. However, just to be sure that there is no timeout, can you try to increase it?

diff --git a/drivers/staging/media/imx/imx-media.h b/drivers/staging/media/imx/imx-media.h
index f263fc3adbb94..82643069dbec3 100644
--- a/drivers/staging/media/imx/imx-media.h
+++ b/drivers/staging/media/imx/imx-media.h
@@ -68,7 +68,7 @@ enum {
 };

 /* How long to wait for EOF interrupts in the buffer-capture subdevs */
-#define IMX_MEDIA_EOF_TIMEOUT       2000
+#define IMX_MEDIA_EOF_TIMEOUT       20000

 struct imx_media_pixfmt {
        /* the in-memory FourCC pixel format */

Also what happens if you increase the framerate on the other elements? E.g.

media-ctl -V "'ipu1_ic_prp':0[fmt:AYUV8_1X32/720x576@1/50]"
...

Regards,
Stefan

aurata-hl · May 9, 2024, 12:43am

Hi @stefan_e.tx ,

Thanks for your reply. You are right that IMX media driver is still in staging, but given the maturity and relatively large userbase of the i.MX6, the amount of bugs is (we hope) not too large!

Anyway, this is where your suggestions are helpful thanks

I am unfortunately away from the hardware for the next few days. Anyway, I’ll try out your suggestions and report back as soon as I have the possibility.

Best regards,
Henrik

aurata-hl · May 15, 2024, 2:56pm

Hi @stefan_e.tx,

First of all, it seems that there is a timeout: changing the
IMX_MEDIA_EOF_TIMEOUT adds 18 seconds more delay (as expected)
before the EOF timeout is logged and VIDIOC_DQBUF is returned.
The changed timing is however the only visible change.

It turns out that the [most likely same] EOF timeot occurs even in a
much simpler pipeline trying to get an interlaced stream from
ipu_csi0_capture: The only [visible] difference in failure mode is then
that the kernel log reports the timeout messages from ipu1_csi0.

In other words, the problem could possibly be unrelated to the de-interlacer,
and more fundamental still.

The following pipeline does produce live images (but using the low-quality
I2P deinterlacer on the ADV7280):

'adv7180 1-0021':0              [fmt:UYVY8_2X8/720x576@1/25 field:none colorspace:smpte170m]
'imx6-mipi-csi2':0              [fmt:UYVY8_2X8/720x576 field:none]
'imx6-mipi-csi2':1              [fmt:UYVY8_2X8/720x576 field:none]
'ipu1_csi0_mux':0               [fmt:UYVY8_2X8/720x576 field:none]
'ipu1_csi0_mux':2               [fmt:UYVY8_2X8/720x576 field:none]
'ipu1_csi0':0           [fmt:UYVY8_2X8/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range
                 crop.bounds:(0,0)/720x576
                 crop:(0,0)/720x576
                 compose.bounds:(0,0)/720x576
                 compose:(0,0)/720x576]
'ipu1_csi0':2           [fmt:AYUV8_1X32/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]

The EOF timeout is triggered by the following pipeline which changes only
the field and derived properties:

'adv7180 1-0021':0              [fmt:UYVY8_2X8/720x288@1/25 field:alternate colorspace:smpte170m]
'imx6-mipi-csi2':0              [fmt:UYVY8_2X8/720x288 field:alternate]
'imx6-mipi-csi2':1              [fmt:UYVY8_2X8/720x288 field:alternate]
'ipu1_csi0_mux':0               [fmt:UYVY8_2X8/720x288 field:alternate]
'ipu1_csi0_mux':2               [fmt:UYVY8_2X8/720x288 field:alternate]
'ipu1_csi0':0           [fmt:UYVY8_2X8/720x288@1/25 field:alternate colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range
                 crop.bounds:(0,0)/720x576
                 crop:(0,0)/720x576
                 compose.bounds:(0,0)/720x576
                 compose:(0,0)/720x576]
'ipu1_csi0':2           [fmt:AYUV8_1X32/720x576@1/25 field:seq-tb colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]

With v4l2-ctl -d 0 --stream-mmap --stream-count=1 --verbose the problem looks as follows:

 VIDIOC_QUERYCAP: ok
                VIDIOC_REQBUFS returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_STREAMON returned 0 (Success)
VIDIOC_DQBUF: failed: Input/output error

To clarify, all pipelines above - and in preceding posts in this thread - are
the negotiated results read back after configuration with media-ctl,
where the exact input can be ignored or changed.
(For example the framerate is not accepted at all on the input format
to adv7180 1-0021, and the driver selects a framerate of 1/25 (for PAL)
regardless of interlacing.
In the same way, field:alternate is enforced by the ADV driver in any
other case than field:none. Likewise, field:seq-tb is what ipu1_csi0
enforces, regardless of whether field:alternate or field:interlaced
is specified).

Whether related to this case or not, I noticed the following message
from this link regarding an EOF timeout:

I actually suspect the IPU, specifically the CSI. In our experience
the CSI is rather sensitive to glitches and/or truncated frames on the
bt.656 bus and can easily loose vertical sync, and/or lock-up.

We use CSI-2, not bt.656, but it could be we have a similar problem.
The question is how best to debug/trace the problem, and any hints are
welcome.

Thanks again,

Henrik

stefan_e.tx · May 22, 2024, 3:01pm

Hi @aurata-hl

Unfortunately, it is hard to tell what is going wrong. Could you provide us with the dot file:

media-ctl --print-dot > csi2.dot

as well as the topology:

media-ctl -p > topology.txt

One pipeline that at least didn’t end in an error with the OV5640 (even thought didn’t show anything valid) was the following:

media-ctl -l "'ov5640 1-003c':0 -> 'imx6-mipi-csi2':0[1]"
media-ctl -l "'imx6-mipi-csi2':2 -> 'ipu1_csi1':0[1]"
media-ctl -l "'ipu1_csi1':1 -> 'ipu1_vdic':0[1]"
media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]"
media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]"
media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]"
# Configure pads
media-ctl -V "'ov5640 1-003c':0 [fmt:UYVY8_1X16/640x480 field:seq-tb]"
media-ctl -V "'imx6-mipi-csi2':2 [fmt:UYVY8_1X16/640x480]"
media-ctl -V "'ipu1_csi1':1 [fmt:AYUV32/640x480]"
media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/640x480]"
media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/640x480]"
media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/640x480]"

If you are interested in a company helping with such problems we made some good experience with Centricular.

Regards,
Stefan

aurata-hl · May 28, 2024, 1:44pm

csi2.dot (4.3 KB)
topology.txt (9.8 KB)

Thanks @stefan_e.tx.

In my latest post I describe a simpler case, which tries to capture
interlaced data from ipu1_csi0 capture (which would allow
de-interlacing in software outside V4L2), and as it has the same
failure mode, let’s focus on that setup.

To be explicit here, we use exactly the following configuration commands:

media-ctl --reset
v4l2-ctl -d /dev/v4l-subdev15 --set-standard PAL-BG

media-ctl -l "'adv7180 1-0021':0 -> 'imx6-mipi-csi2':0[1]"
media-ctl -l "'imx6-mipi-csi2':1 -> 'ipu1_csi0_mux':0[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"

media-ctl -V "'adv7180 1-0021':0 [fmt:UYVY8_2X8/720x288 field:alternate ]"
media-ctl -V "'imx6-mipi-csi2':0 [fmt:UYVY8_2X8/720x288 field:alternate ]"
media-ctl -V "'ipu1_csi0_mux':0 [fmt:UYVY8_2X8/720x288 field:alternate ]"
media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_2X8/720x288@1/25 field:alternate ]"
media-ctl -V "'ipu1_csi0':2 [fmt:UYVY8_2X8/720x576@1/25 field:interlaced ]"

v4l2-ctl -d "/dev/video0" --set-fmt-video=width=720,height=576,pixelformat=UYVY

and the resulting configuration is then (using my previous format):

'adv7180 1-0021':0              [fmt:UYVY8_2X8/720x288@1/25 field:alternate colorspace:smpte170m]
'imx6-mipi-csi2':0              [fmt:UYVY8_2X8/720x288 field:alternate]
'imx6-mipi-csi2':1              [fmt:UYVY8_2X8/720x288 field:alternate]
'ipu1_csi0_mux':0               [fmt:UYVY8_2X8/720x288 field:alternate]
'ipu1_csi0_mux':2               [fmt:UYVY8_2X8/720x288 field:alternate]
'ipu1_csi0':0           [fmt:UYVY8_2X8/720x288@1/25 field:alternate colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range
                 crop.bounds:(0,0)/720x576
                 crop:(0,0)/720x576
                 compose.bounds:(0,0)/720x576
                 compose:(0,0)/720x576]
'ipu1_csi0':2           [fmt:AYUV8_1X32/720x576@1/25 field:seq-tb colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]

The attached files contain the full topology and resulting configuration, as
per your request
(but I would expect the above information to be sufficient and succinct,
otherwise I’d have attached the more verbose output previously).

The interesting bit is what happens on 'ipu1_csi0':2: No matter the
combination of field and image size, the result will be as above,
i.e. field:seq-tb and 720x576 in resulting image size (including
the crop for 'ipu1_csi0':0); the above commands have then been updated
to differ the least from the result, but in reality my preference
(due to latency minimization) would be to capture field:alternate
at 720x288, but ipu1_csi0 pad doesn’t allow it.

Nevertheless, even though the commands and resulting configuration seems
coherent and valid (AFAICS) and passes validation the result from
v4l2-ctl -d 0 --stream-mmap is

VIDIOC_QUERYCAP: ok
VIDIOC_REQBUFS returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_STREAMON returned 0 (Success)
[...] ipu1_csi0: EOF timeout
VIDIOC_DQBUF: failed: Input/output error
[...] ipu1_csi0: wait last EOF timeout

where the delay before timeout precisely matches the value of
IMX_MEDIA_EOF_TIMEOUT.

Do you know of any working configuration where field:alternate is
passed into ipu1_csi0, or can we assume that this is the special case
that triggers the problem?

Thanks for your assistance,

Henrik

stefan_e.tx · May 29, 2024, 12:56pm

Hi @aurata-hl

I’m not exactly sure what in the end leads to the issue but it must be something in drivers/staging/media/imx/imx-media-csi.c. Most likely field:alternate is really not supported by one of the elements. There is this suspicious comment:

/*
 * This driver does not support alternate field mode, and
 * the CSI captures a whole frame, so the CSI never presents
 * alternate mode at its source pads. If user has not
 * already requested sequential, translate ALTERNATE at
 * sink pad to SEQ_TB or SEQ_BT at the source pad depending
 * on input height (assume NTSC BT order if 480 total active
 * frame lines, otherwise PAL TB order).
 */

Could it be that the adv7180 transfers the expected size of 720x288 but the imx-media-csi driver expects 720x576 and this leads to the timeout?
Can you try to apply this patch and see what the output will be? It will also hardly set the height to 288 for the input, maybe this will cause some side effects:

diff --git a/drivers/staging/media/imx/imx-media-csi.c b/drivers/staging/media/imx/imx-media-csi.c
index b2b1f4dd41d79..c7c2a113551f7 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -500,6 +500,12 @@ static int csi_idmac_setup_channel(struct csi_priv *priv)
 		break;
 	}
 
+	pr_info("%s - %s:%d: image width: %d, height: %d\n", __func__, __FILE__, __LINE__,
+			image.pix.width, image.pix.height);
+
+	if (image.pix.height == 576)
+		image.pix.height /= 2;
+
 	if (passthrough) {
 		if (priv->interweave_swap) {
 			/* start interweave scan at 1st top line (2nd line) */
@@ -507,6 +513,8 @@ static int csi_idmac_setup_channel(struct csi_priv *priv)
 			image.phys1 += image.pix.bytesperline;
 		}
 
+		pr_info("%s - %s:%d\n", __func__, __FILE__, __LINE__);
+
 		ipu_cpmem_set_resolution(priv->idmac_ch,
 					 image.rect.width * passthrough_cycles,
 					 image.rect.height);
@@ -521,6 +529,8 @@ static int csi_idmac_setup_channel(struct csi_priv *priv)
 			image.rect.top = 1;
 		}
 
+		pr_info("%s - %s:%d\n", __func__, __FILE__, __LINE__);
+
 		ret = ipu_cpmem_set_image(priv->idmac_ch, &image);
 		if (ret)
 			goto unsetup_vb2;

Regards,
Stefan

aurata-hl · May 30, 2024, 1:35pm

Hi @stefan_e.tx,

Thanks for your feedback.

I had previously assumed that the CSI simply enforced field:seq-bt
for simplicity (though at the cost of latency). Furthermore I have seen
various people suggesting that it is better to use a format with two
fields, to prevent mixups further down the line (although in theory
field information should be available and preserved, but interlaced formats
are getting less and less testing these days).

The question is whether “does not support alternate field mode”
(which it doesn’t at the source side) also applies to the sink side;
I had assumed it didn’t.

Anyway, the result of your patch is the same EOF timeout, with the extra
messages added:

VIDIOC_QUERYCAP: ok
VIDIOC_REQBUFS returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
[  251.685256] csi_idmac_setup_channel - drivers/staging/media/imx/imx-media-csi.c:503: image width: 720, height: 576
[  251.685289] csi_idmac_setup_channel - drivers/staging/media/imx/imx-media-csi.c:532
VIDIOC_STREAMON returned 0 (Success)
[  253.696176] ipu1_csi0: EOF timeout
VIDIOC_DQBUF: failed: Input/output error

Whether or not field:alternate is actually supported on the sink side
of the CSI, I’m getting more and more convinced that the combination
of adv7180, i.MX6 and CSI has not been used too much in the field…

…which reminds me of having seen the following
thread before reaching out to you.
The thread contains many subjects, and the one that’s relevant here
actually disappears unresolved from the discussion!

I’m referring to the part of the linked message starting with
“Let’s try an interlaced format…”
and ending with
“Sorry everything looks fine, I don’t know why that pipeline is failing to generate any data.”

I’m using a different sensor and different interlaced format, which is why
I did not mention it to begin with. It does mention that field:alternate
was working with adv7180 at that time, which is why I originally assumed
that the problem was not related, and that adv7180 would work with
field:alternate. Mabye things have changed since then…

What do you conclude? Your guess is probably better than mine.
(And no rush, I’ll be away from the hardware again and until Tuesday).

Thanks again,

Henrik

stefan_e.tx · June 5, 2024, 9:43am

Hi @aurata-hl

Unfortunately, I think this was never tested and therefore does not work. I assume the image never really arrives and thus can not be processed. The DMA should send an EOF interrupt when the image is read. However, this doesn’t seem to happen. Can you check if the eof interrupt is triggered when the pipeline fails?

cat /proc/interrupts |grep eof

Also the patch below might help to track down the issue:

diff --git a/drivers/staging/media/imx/imx-media-csi.c b/drivers/staging/media/imx/imx-media-csi.c
index b2b1f4dd41d79..73eeee38ea024 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -306,6 +306,8 @@ static irqreturn_t csi_idmac_eof_interrupt(int irq, void *dev_id)
 {
 	struct csi_priv *priv = dev_id;
 
+	pr_info("%s - %s:%d\n", __func__, __FILE__, __LINE__);
+
 	spin_lock(&priv->irqlock);
 
 	if (priv->last_eof) {
@@ -500,6 +502,12 @@ static int csi_idmac_setup_channel(struct csi_priv *priv)
 		break;
 	}
 
+	pr_info("%s - %s:%d: image width: %d, height: %d\n", __func__, __FILE__, __LINE__,
+			image.pix.width, image.pix.height);
+
+	if (image.pix.height == 576)
+		image.pix.height /= 2;
+
 	if (passthrough) {
 		if (priv->interweave_swap) {
 			/* start interweave scan at 1st top line (2nd line) */
@@ -507,6 +515,8 @@ static int csi_idmac_setup_channel(struct csi_priv *priv)
 			image.phys1 += image.pix.bytesperline;
 		}
 
+		pr_info("%s - %s:%d\n", __func__, __FILE__, __LINE__);
+
 		ipu_cpmem_set_resolution(priv->idmac_ch,
 					 image.rect.width * passthrough_cycles,
 					 image.rect.height);
@@ -521,6 +531,12 @@ static int csi_idmac_setup_channel(struct csi_priv *priv)
 			image.rect.top = 1;
 		}
 
+		pr_info("%s - %s:%d: rect width: %d, height: %d\n", __func__, __FILE__, __LINE__,
+				image.rect.width, image.rect.height);
+		if (image.rect.height == 576)
+			image.rect.height /= 2;
+		pr_info("%s - %s:%d\n", __func__, __FILE__, __LINE__);
+
 		ret = ipu_cpmem_set_image(priv->idmac_ch, &image);
 		if (ret)
 			goto unsetup_vb2;

However, I’m not sure if we can fix this driver without involving a partner company because it’s outside of our expertise. Even if the image is finally read, the processing will most likely not work afterward.

Regards,
Stefan

aurata-hl · June 6, 2024, 2:18pm

Hi @stefan_e.tx,

We agree that it seems likely that this scenario (field:alternate
across the i.MX6 ipu0_csi) is not supported in practice, although
nothing in advance indicates that this should be so.

There is no line matching “eof” in /proc/interrupts, neither before nor
after the pipeline failure.

I applied your latest patch, and it gives the following information trace:

VIDIOC_QUERYCAP: ok
VIDIOC_REQBUFS returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_STREAMON returned 0 (Success)
[ 1613.280702] csi_idmac_setup_channel - drivers/staging/media/imx/imx-media-csi.c:505: image width: 720, height: 576
ess)
[ 1613.280735] csi_idmac_setup_channel - drivers/staging/media/imx/imx-media-csi.c:534: rect width: 720, height: 576
[ 1613.280751] csi_idmac_setup_channel - drivers/staging/media/imx/imx-media-csi.c:538
VIDIOC_DQBUF: failed: Input/output error
[ 1615.354208] ipu1_csi0: EOF timeout
[ 1617.434552] ipu1_csi0: wait last EOF timeout

Based on the current state of affairs, the conclusion on our part is now
that we’ll not pursue this issure further, but rather accept the image
quality limitations of the ADV7280’s I2P and use field:none throughout.

Thanks again for your support on this issue.
(I’m not aware of any relevant place to flag the issue upstream, but please
feel free to do so).

Best regards,

Henrik

stefan_e.tx · June 10, 2024, 11:19am

Hi @aurata-hl

Thanks a lot for the update and sorry we could not help to resolve the issue properly.

Regards,
Stefan