De-interlacing video with i.MX6 IPU

We have an ADV7280A-M video decoder on a custom board connected via MIPI-CSI
to an Apalis iMX6 on Ixora carrier.

Current kernel is 6.1.80-rt26-6.6.0-devel+git.ca3cd16f344f, using
Yocto build.

The ADV receives video in PAL format, and we receive the video via
ipu1_csi0 and route through ipu1_ic_prpvf for upscaling to 1000x800.

When using the de-interlacer (I2P) on the ADV, this is actually working,
with the following resulting media-controller pad configuration:

'adv7180 1-0021':0  [fmt:UYVY8_2X8/720x576@1/25 field:none colorspace:smpte170m]
'imx6-mipi-csi2':0  [fmt:UYVY8_2X8/720x576 field:none]
'imx6-mipi-csi2':1  [fmt:UYVY8_2X8/720x576 field:none]
'ipu1_csi0_mux':0   [fmt:UYVY8_2X8/720x576 field:none]
'ipu1_csi0_mux':2   [fmt:UYVY8_2X8/720x576 field:none]
'ipu1_csi0':0       [fmt:UYVY8_2X8/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range 
             crop.bounds:(0,0)/720x576
             crop:(0,0)/720x576
             compose.bounds:(0,0)/720x576
             compose:(0,0)/720x576]
'ipu1_csi0':1       [fmt:AYUV8_1X32/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_ic_prp':0     [fmt:AYUV8_1X32/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_ic_prp':2     [fmt:AYUV8_1X32/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_ic_prpvf':0   [fmt:AYUV8_1X32/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_ic_prpvf':1   [fmt:AYUV8_1X32/1000x800@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]

Due to the poor quality of the I2P on the ADV, we need to use the the
motion-compensated de-interlacer in the IPU (ipu1_vdic).

It is surprisingly difficult to find a working configuration, and so far
we have not been successful.

Current in-kernel documentation (Documentation/admin/media/imx.rst)
contains an example using an adv7180, but uses field:seq-bt on the
adv7180 pad, but since driver commit 6457b6263f0f, only field:alternate
is allowed.

The only media-controller pad configuration we have come up with that
uses field:alternate and is not rejected before streaming is the following:

'adv7180 1-0021':0  [fmt:UYVY8_2X8/720x288@1/25 field:alternate colorspace:smpte170m]
'imx6-mipi-csi2':0  [fmt:UYVY8_2X8/720x288 field:alternate]
'imx6-mipi-csi2':1  [fmt:UYVY8_2X8/720x288 field:alternate]
'ipu1_csi0_mux':0   [fmt:UYVY8_2X8/720x288 field:alternate]
'ipu1_csi0_mux':2   [fmt:UYVY8_2X8/720x288 field:alternate]
'ipu1_csi0':0       [fmt:UYVY8_2X8/720x288@1/25 field:alternate colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range
             crop.bounds:(0,0)/720x576
             crop:(0,0)/720x576
             compose.bounds:(0,0)/720x576
             compose:(0,0)/720x576]
'ipu1_csi0':1       [fmt:AYUV8_1X32/720x576@1/25 field:seq-tb colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_vdic':0       [fmt:AYUV8_1X32/720x576@1/25 field:seq-tb colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_vdic':2       [fmt:AYUV8_1X32/720x576@1/50 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_ic_prp':0     [fmt:AYUV8_1X32/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_ic_prp':2     [fmt:AYUV8_1X32/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_ic_prpvf':0   [fmt:AYUV8_1X32/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
'ipu1_ic_prpvf':1   [fmt:AYUV8_1X32/1000x800@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]

When trying to stream data, however, we get a
VIDIOC_DQBUF: failed: Input/output error with the following logged
in dmesg:

ipu1_ic_prpvf: EOF timeout
ipu1_ic_prpvf: wait last EOF timeout

The only differences between the (broken) ipu1_vdic configuration and the
(working, but poor quality) I2P configuration further above are

  1. Use of field:alternate with 720x288 where appropriate
  2. Use of ipu1_vdic and hence a slightly longer route.
  3. A strange interval of 1/50 on the output of ipu1_vdic, which is
    however enforced by the driver

What’s wrong here?

Hello @aurata-hl,

Thanks for reaching out to the Toradex Community! I’m discussing this internally to find a solution. I’ll let you know as soon as I have an answer.

Hello @rudhi.tx,

Thanks for looking into this issue, I appreciate it!

Best regards,

Henrik Lauritzen

Hello @aurata-hl,

This topic is related more to the iMX6 SOC and therefore you will find more related discussions on the NXP Community. Could you please try post this question there instead?

I also found some similar topics there:

Hi @rudhi.tx,

Although you are right that our topic is somewhat related to the i.MX6 SoC,
it is also based on the upstream BSP (BSP 6) provided by Toradex.

The issues in the two links you provide are somewhat similar (though not
a full match) to ours, but are also related to NXP’s BSP, which - at the
relevant level (kernel + V4L2) is a rather different software stack.

The best-matching topic we can find at NXP is the following:

https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/De-interlace-Capture-Device/ta-p/1114964.

On closer inspection we find the following problems with that topic:

  1. The patch provided relates only to NXP’s BSP (and a rather old version of
    it), not the the upstream
    driver in Toradex’ BSP 6.
  2. The post states that “This code is not an official delivery and as such no guarantee of support for this code is provided by Freescale”.
  3. The patch seems to have been used with mixed success by the people involved,
    and there are multiple indications of image-quality problems
    (which is the very issue we are trying to avoid here).
  4. There are multiple reports of stability problems (which would be completely
    unacceptable in our use-case) from the users who succeeded in using
    the patch - many, many years ago.
  5. The topic was closed by NXP without responding to several of the issues
    mentioned.

In other words, it does not look like NXP is the right support forum for us,
given our use of Toradex’ BSP 6 (containing a mainline kernel and V4L2 stack).

The question then is, what is the correct support forum?

Best regards,

Henrik

Hi @aurata-hl

Unfortunately, this driver is still in staging therefore risks are high there are still some bugs in there. However, just to be sure that there is no timeout, can you try to increase it?

diff --git a/drivers/staging/media/imx/imx-media.h b/drivers/staging/media/imx/imx-media.h
index f263fc3adbb94..82643069dbec3 100644
--- a/drivers/staging/media/imx/imx-media.h
+++ b/drivers/staging/media/imx/imx-media.h
@@ -68,7 +68,7 @@ enum {
 };

 /* How long to wait for EOF interrupts in the buffer-capture subdevs */
-#define IMX_MEDIA_EOF_TIMEOUT       2000
+#define IMX_MEDIA_EOF_TIMEOUT       20000

 struct imx_media_pixfmt {
        /* the in-memory FourCC pixel format */

Also what happens if you increase the framerate on the other elements? E.g.

media-ctl -V "'ipu1_ic_prp':0[fmt:AYUV8_1X32/720x576@1/50]"
...

Regards,
Stefan

Hi @stefan_e.tx ,

Thanks for your reply. You are right that IMX media driver is still in staging, but given the maturity and relatively large userbase of the i.MX6, the amount of bugs is (we hope) not too large!

Anyway, this is where your suggestions are helpful thanks :slight_smile:

I am unfortunately away from the hardware for the next few days. Anyway, I’ll try out your suggestions and report back as soon as I have the possibility.

Best regards,
Henrik

Hi @stefan_e.tx,

First of all, it seems that there is a timeout: changing the
IMX_MEDIA_EOF_TIMEOUT adds 18 seconds more delay (as expected)
before the EOF timeout is logged and VIDIOC_DQBUF is returned.
The changed timing is however the only visible change.

It turns out that the [most likely same] EOF timeot occurs even in a
much simpler pipeline trying to get an interlaced stream from
ipu_csi0_capture: The only [visible] difference in failure mode is then
that the kernel log reports the timeout messages from ipu1_csi0.

In other words, the problem could possibly be unrelated to the de-interlacer,
and more fundamental still.

The following pipeline does produce live images (but using the low-quality
I2P deinterlacer on the ADV7280):

'adv7180 1-0021':0              [fmt:UYVY8_2X8/720x576@1/25 field:none colorspace:smpte170m]
'imx6-mipi-csi2':0              [fmt:UYVY8_2X8/720x576 field:none]
'imx6-mipi-csi2':1              [fmt:UYVY8_2X8/720x576 field:none]
'ipu1_csi0_mux':0               [fmt:UYVY8_2X8/720x576 field:none]
'ipu1_csi0_mux':2               [fmt:UYVY8_2X8/720x576 field:none]
'ipu1_csi0':0           [fmt:UYVY8_2X8/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range
                 crop.bounds:(0,0)/720x576
                 crop:(0,0)/720x576
                 compose.bounds:(0,0)/720x576
                 compose:(0,0)/720x576]
'ipu1_csi0':2           [fmt:AYUV8_1X32/720x576@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]

The EOF timeout is triggered by the following pipeline which changes only
the field and derived properties:

'adv7180 1-0021':0              [fmt:UYVY8_2X8/720x288@1/25 field:alternate colorspace:smpte170m]
'imx6-mipi-csi2':0              [fmt:UYVY8_2X8/720x288 field:alternate]
'imx6-mipi-csi2':1              [fmt:UYVY8_2X8/720x288 field:alternate]
'ipu1_csi0_mux':0               [fmt:UYVY8_2X8/720x288 field:alternate]
'ipu1_csi0_mux':2               [fmt:UYVY8_2X8/720x288 field:alternate]
'ipu1_csi0':0           [fmt:UYVY8_2X8/720x288@1/25 field:alternate colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range
                 crop.bounds:(0,0)/720x576
                 crop:(0,0)/720x576
                 compose.bounds:(0,0)/720x576
                 compose:(0,0)/720x576]
'ipu1_csi0':2           [fmt:AYUV8_1X32/720x576@1/25 field:seq-tb colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]

With v4l2-ctl -d 0 --stream-mmap --stream-count=1 --verbose the problem looks as follows:

 VIDIOC_QUERYCAP: ok
                VIDIOC_REQBUFS returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_STREAMON returned 0 (Success)
VIDIOC_DQBUF: failed: Input/output error

To clarify, all pipelines above - and in preceding posts in this thread - are
the negotiated results read back after configuration with media-ctl,
where the exact input can be ignored or changed.
(For example the framerate is not accepted at all on the input format
to adv7180 1-0021, and the driver selects a framerate of 1/25 (for PAL)
regardless of interlacing.
In the same way, field:alternate is enforced by the ADV driver in any
other case than field:none. Likewise, field:seq-tb is what ipu1_csi0
enforces, regardless of whether field:alternate or field:interlaced
is specified).

Whether related to this case or not, I noticed the following message
from this link regarding an EOF timeout:

I actually suspect the IPU, specifically the CSI. In our experience
the CSI is rather sensitive to glitches and/or truncated frames on the
bt.656 bus and can easily loose vertical sync, and/or lock-up.

We use CSI-2, not bt.656, but it could be we have a similar problem.
The question is how best to debug/trace the problem, and any hints are
welcome.

Thanks again,

Henrik