Can not write video with audio on Colibri iMX6 using GStreamer 1.0

We have a problem with capturing Audio/Video from Parallel Camera Interface (CIF) and Line In on Colibri iMX6 using GStreamer 1.0.
We are using Colibri Evaluation Board v3.2 and Analogue Camera Adapter V2.0.
U-Boot/Kernel version and also startup log you can find in “StartUpLog.txt”.
GStreamer version, used pipeline and gst-launch output log you can find in “GstLaunch.txt”. Also you can find the graph of gst pipeline in appropriate DOT and PNG files and result of capturing in “video.avi”.
Our problem is that audio in output file is shorter than video and isn’t synchronized with video. We also tried to use other encoders for audio (vorbis) and other containers (mp4, matroska) and result is the same.
We can’t solve this problem and we need some help…
link text

Just to let you know we are looking at it.

Many thanks, without the function (writing video with audio) I can not close project for my client. The client have a lot of pre-orders for a device with the Colibri IMX6 board. I hope you will help me. Regards, Yuriy.

Hi

I see some issues with your pipeline.

  • Assuming you are using an ACM the kernel already does the interlacing. given that, why adding ‘imxipuvideotransform deinterlace=true’?
  • With ‘queue2 max-size-buffers=0 max-size-time=0 max-size-bytes=0’ you basically disable any queuing and the sourcing element gets stalled. Is there any reason for that?
  • I guess that this ‘imxg2dvideosink sync=false’ asks to not drop frames if the video sink cannot keep up with the datastream which might give ‘funny’ effects with a live source.
  • You do not force a audio format. Setting that in the pipeline might make the results more predictable.
  • Are you sure that /media/sda1/ can sustainably write 1MB/sec? Add least some of my USB mass storage is not well above that rate. ‘dd if=/dev/urandom of=/media/sda1/delete.me bs=100k count=1000’

I could not reproduce the issue on our stock V2.7 image and also not on the V2.6.1 image.

E.g. the following pipeline (on a V2.7 image, imxvpuenc_mpeg4 → vpuenc_mpeg4) gives me a valid video file with audio and video of the same lenght and in sync up to a small latency.

gst-launch-1.0 -e -v \
  imxv4l2videosrc do-timestamp=true crop-meta-x=16 crop-meta-y=16 crop-meta-width=660 crop-meta-height=522 ! \
    queue ! textoverlay text="Hello_World" shaded-background=1 font-desc="Arial 22px" ! \
  tee name=tx \
  tx. ! \
    queue ! vpuenc_mpeg4 bitrate=1000 ! queue ! mux. \
  tx. ! \
    queue ! imxeglvivsink \
  alsasrc do-timestamp=true ! audio/x-raw, format=S16LE, rate=44100, channels=2 ! \
    queue ! imxmp3audioenc bitrate=96 ! queue ! mux. \
  avimux name=mux ! \
    filesink location=video.avi

Max

Thanks for your answer!

For about your questions above, our current pipeline is the result of many iterations of experiments with GStreamer’s command line without success.
So it was added to the question as the best working example in our case.

We have tried to use your example of pipeline yesterday.

First of all we measured writing speed to four different Flash/SD Cards (see attachment) and it is at least 1.5 MBytes/s.

Also we used your pipeline on three different Linux Images:

  1. Our custom image 2.6 (we changed boot logo)
  2. Colibri_iMX6_LinuxImageV2.6_20160826
  3. Colibri_iMX6_LinuxImageV2.7Beta1_20170112

Images were downloaded from - http://developer.toradex.com/files/toradex-dev/uploads/media/Colibri/Linux/Images/ .

We also changed ‘textoverlay’ to ‘timeoverlay’ for better understanding what is going on.

And the result is the same in all three cases. Video is longer than audio. The duration of each video must be about 1 minute and timeoverlay corresponds to this time but video in the file is about two times slower at playback. Audio is exactly 1 minute long in each file.
We also tried to record only video without audio using your example pipeline and result was the same.

All debug outputs and recorded videofiles you can find in attachment. It contains 4 folders for appropriate experiments.

We stuck at this point.link text

And what are you using to provide the digital video data?

We bought next things:

Analog camera adapter:
https://www.toradex.com/accessories/analogue-camera-adapter
EV board:
https://www.toradex.com/products/carrier-board/colibri-evaluation-carrier-board
Display:
https://www.toradex.com/accessories/display-edt-7.0-tft-wvga-with-touch
Computer module
https://www.toradex.com/computer-on-modules/colibri-arm-family/nxp-freescale-imx6

and connect it together.

For analog camera using that:

https://ru.aliexpress.com/item/600TVL-1-4-1-8mm-CMOS-FPV-170-Degree-Wide-Angle-Lens-Camera-PAL-NTSC-3/32517927292.html?spm=2114.13010608.0.0.5hjGnG

I would like to add information what we are using Industrial Temperature (IT) version, which have frequency 800 Mhz vs 1 Ghz non industrial temperature, maybe IT version have yet some differences?

Hi, please let me know if you looking at the problem.

Analog camera adapter: https://www.toradex.com/accessories/analogue-camera-adapter EV board: https://www.toradex.com/products/carrier-board/colibri-evaluation-carrier-board Display: https://www.toradex.com/accessories/display-edt-7.0-tft-wvga-with-touch Computer module https://www.toradex.com/computer-on-modules/colibri-arm-family/nxp-freescale-imx6

please let me know if you looking at the problem.

We are working on it, but no results are only expected after the weekend.

Hello,

I have similar problem, do you have solution or ideas how to deal with it?

Hi

Your hardware list allowed me to reproduce your issue.
The deinterlace plugin made me use the older ACM on the first experiments.
When I switched to the ACA with the ADV7280 Video ADC I got the same results you have, the Audio plays at the correct rate and then stops in the middle of the video, the video is about twice as long as actually recorded.

The ADV7280 is able to do deinterlacing and in doing so does double the framerate.
However it looks that this does not get properly propagated up the V4L2 stack, so the gstreamer pipeline gets twice as many frames as expected.

I so far was not able to correct this, neither in the ADV7280 driver nor the gstreamer pipeline.

One thing which worked for me is to use the ADV7280 in its interlaced mode.
i.e. commenting the HW_DEINT define in the driver, recompile the kernel modules and deploy drivers/media/platform/mxc/capture/adv7280_tvin.ko to the Colibri iMX6.

--- a/drivers/media/platform/mxc/capture/adv7280.c
+++ b/drivers/media/platform/mxc/capture/adv7280.c
@@ -115,7 +115,7 @@ static DEFINE_MUTEX(mutex);
 #define ADV7280_SD_SATURATION_CB               0xe3
 #define ADV7280_SD_SATURATION_CR               0xe4
 
-#define HW_DEINT /* Enable hardware deinterlacer */
+//#define HW_DEINT /* Enable hardware deinterlacer */
 #define VPP_SLAVE_ADDRESS                      0x42
 
 #define VPP_DEINT_RESET                                0x41

I don’t expect a quick solution which uses the HW deinterlacing in the ADV7280. I will create a ticket so that we can properly schedule this in our development.

Max

P.S. I now moved to the IT version of the module, the behaviour is the same as on the non IT version.

Hi,
Many thanks for your answer. Maybe you have another similar solution which worked (for example with using IMX7 module, or something)? What hardware parts you used first time for test the issue and as you said it worked?

Hi

Well the solution which worked for me with your HW setup is outlined above.

Change one line in the kernel sources, recompile the kernel and the kernel modules and deploy them.

The Colibri T20/T30 are the only Colibri Modules which would currently work with the ACA, but I’m not sure if they suffer the same issue as the Colibri iMX6 does.

At first I used the ACM which did work because I thought you have it. It is the predecessor of the ACA and no longer available.

Max

Hi,
thanks for your help, we got properly writing video with audio.
Regards, Yuriy.