Artifacts when muxing audio and video with iMX8QP

I am trying to mux audio and video in my gstreamer pipeline from a usb streaming camera source. Here is the pipeline description I am using:

gst-launch-1.0 -v v4l2src device=/dev/video2 ! video/x-raw,width=1920,height=1080,framerate=30/1 ! videoconvert ! fisheyeptz ! videoconvert ! v4l2h264enc extra-controls="controls,video_bitrate_mode=1,video_bitrate=2500000,h264_profile=2" ! h264parse ! tee name=t alsasrc ! queue ! volume volume=1.0 ! audioconvert ! avenc_aac ! aacparse ! mux. t. ! queue ! matroskamux name=mux ! filesink location=gst.mkv

The resulting video file contains numerous “artifacts”: it is particularly blurry for things that are in motion.

When I remove the audio encoding portion and mux only the video, the video is sharp and responsive with zero artifacts.

I’ve attached two sample clips: one with audio that demonstrates that artifacts and one without. I am confused why the muxing process would cause these artifacts or how I can improve my pipeline or construct it different to avoid these artifacts. Any pointers appreciated!

Sample clips are here: gst_with_audio.mkv gst_no_audio.mkv

grh

P.S. - I’m using the tdx-xwayland distro and built the tdx-reference-multimedia-image .

Hi @grh

Thanks for writing to the Toradex Community!

Could you provide the exact software version of your module?

I’m using the tdx-xwayland distro and built the tdx-reference-multimedia-image

What options did you choose to create the image? Why aren’t using the weston image?

Best regards,
Jaski

@jaski.tx ,

Sure. Here is the Build Configuration:

 Build Configuration:
 BB_VERSION           = "1.46.0"
 BUILD_SYS            = "x86_64-linux"
 NATIVELSBSTRING      = "universal"
 TARGET_SYS           = "aarch64-tdx-linux"
 MACHINE              = "apalis-imx8"
 DISTRO               = "tdx-xwayland"
 DISTRO_VERSION       = "5.2.0-devel-20210706132650+build.0"
 TUNE_FEATURES        = "aarch64"
 TARGET_FPU           = ""
 meta-toradex-nxp     = "HEAD:a9ae3ea1f4ad181edbe940a99943a8256a811326"
 meta-freescale       = "HEAD:468e157b4cff033694837db7a8bf7f39be2ea6c7"
 meta-freescale-3rdparty = "HEAD:50551f0576c00bad71b1496ff2af8eaa6d7f9ca4"
 meta-toradex-tegra   = "HEAD:60a50b4d6f03a3b4b2d224c2cc2cd37f522dff75"
 meta-toradex-bsp-common = "HEAD:5234d3b8bc7f1fb6269a350213e93a0f8f66d6aa"
 meta-oe              
 meta-filesystems     
 meta-gnome           
 meta-xfce            
 meta-initramfs       
 meta-networking      
 meta-multimedia      
 meta-python          
 meta-webserver       = "HEAD:389757a7494221fea1d8437e9a01c6e4fb4700af"
 meta-freescale-distro = "HEAD:5d882cdf079b3bde0bd9869ce3ca3db411acbf3b"
 meta-toradex-demos   = "HEAD:82e3d7b2b56fccabcd52708314acc9dfd6bb24d1"
 meta-qt5             = "HEAD:0d8eb956015acdea7e77cd6672d08dce18061510"
 meta-toradex-distro  = "HEAD:632db61f177acf18df79d40fefdf1c662c6233a8"
 meta-poky            = "HEAD:febea5f6be380fed476552d741dc97e9c78ebdd9"
 meta                 = "HEAD:707036d4ec12ef1a260adcef78627b26e32e6540"
 meta-argus           = "dev:c467bdf688979a97f38f42f746479fa92378e963"
 workspace            = "<unknown>:<unknown>"

Does this help?

grh

HI @grh

Thanks for this information.

Could you provide also your changes to local.conf or to any other files, if any?

Best regards,
Jaski

I managed to isolate the issue to the matroskamux element. We are still not sure why it is doing that; it is almost as if there are video frames being dropped/not making it to the muxer, and so it’s trying to fill in those gaps with duplicate frames?

In any case, we found a workaround: I simply switched to using the mpegtsmux element to mux our audio/video into an MPEG2-TS format. This fixes the artifact issue, and it is better suited for streaming over a network, so win-win.

Hi @grh

Perfect that you found a solution.
Could you share a bit more about your solution, please?
Which commands exactly did you use?
What changes have you done to local.conf to build the image?

Thanks and best regards,
Jaski