Decoding high-resolution videos on i.mx8

Technical Support
, ,
1

Hello,

According to data sheets, I should be able to decode 4K video on i.mx8m+.

Using the “Toradex Embedded Linux Reference Multimedia Image”, version “6.6.0+build.12” I tried playing video streams using gstreamer with following resolutions:

  • 1280x960 → h.264 and h.265 decoding work
  • 1600x1200 → h.264 decoding works, h.265 decoding fails
  • 2048x1535 → h.264 and h.265 decoding fail
  • I also tried playing video files with 4K resolution, that also fails.

I tried decodebin / uridecodebin which uses v4l2h264dec0 / v4l2h265dec0 and also tried vpudec.

How can i play video with higher resolution?

Example log

root@verdin-imx8mp-15336033:~# GST_DEBUG=2 gst-launch-1.0 uridecodebin uri=rtsp://192.168.2.62/Streaming/channels/001/ ! videoconvert ! fpsdisplaysink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Pipeline is PREROLLED ...
Prerolled, waiting for progress to finish...
Progress: (connect) Connecting to rtsp://192.168.2.62/Streaming/channels/001/
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Redistribute latency...
Progress: (request) Sending PLAY request
Redistribute latency...
Progress: (request) Sent PLAY request
Redistribute latency...

====== V4L2DEC: 1.20.3 build on Sep 15 2022 02:27:06. ======
0:00:00.232983619  1085 0xffff9c01c060 WARN               decodebin gstdecodebin2.c:2447:connect_pad:<decodebin0> Element v4l2h264dec0 does not accept caps
====== VPUDEC: 4.7.2 build on Sep  1 2022 09:49:28. ======
	wrapper: 3.0.0 (VPUWRAPPER_ARM64_LINUX Build on Aug 31 2022 01:28:14)
	vpulib: 1.1.1
	firmware: 1.1.1.65535
0:00:00.247580000  1085 0xffff9c01c060 ERROR         vpu_dec_object gstvpudecobject.c:1415:gst_vpu_dec_object_decode:<vpudecobject0> failed to decode frame: failure
0:00:00.275309050  1085 0xaaaaeed78360 WARN                 basesrc gstbasesrc.c:3127:gst_base_src_loop:<udpsrc1> error: Internal data stream error.
0:00:00.275370924  1085 0xaaaaeed78360 WARN                 basesrc gstbasesrc.c:3127:gst_base_src_loop:<udpsrc1> error: streaming stopped, reason error (-5)
ERROR: from element /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstRTSPSrc:source/GstUDPSrc:udpsrc1: Internal data stream error.
Additional debug info:
../git/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstRTSPSrc:source/GstUDPSrc:udpsrc1:
streaming stopped, reason error (-5)
Execution ended after 0:00:00.158149382
Setting pipeline to NULL ...
0:00:00.281243775  1085 0xaaaaeed77aa0 WARN                 rtspsrc gstrtspsrc.c:6526:gst_rtsp_src_receive_response:<source> receive interrupted
0:00:00.281316524  1085 0xaaaaeed77aa0 WARN                 rtspsrc gstrtspsrc.c:6624:gst_rtspsrc_try_send:<source> receive interrupted
0:00:00.281348148  1085 0xaaaaeed77aa0 WARN                 rtspsrc gstrtspsrc.c:9037:gst_rtspsrc_pause:<source> PAUSE interrupted
0:00:00.285059304  1085 0xaaaaeed77aa0 WARN                 rtspsrc gstrtspsrc.c:6789:gst_rtspsrc_send:<source> got NOT IMPLEMENTED, disable method TEARDOWN
Freeing pipeline ...

System info

root@verdin-imx8mp-15336033:~# tdx-info 

Software summary
------------------------------------------------------------
Bootloader:               U-Boot
Kernel version:           5.15.148-6.6.0-6.6.0+git.23a8e831749d #1 SMP PREEMPT Thu Feb 29 20:25:21 UTC 2024
Kernel command line:      root=PARTUUID=c21eb82c-02 ro rootwait console=tty1 console=ttymxc2,115200 consoleblank=0 earlycon
Distro name:              NAME="TDX Wayland with XWayland"
Distro version:           VERSION_ID=6.6.0-build.12
Distro variant:           -
Hostname:                 verdin-imx8mp-15336033
------------------------------------------------------------

Hardware info
------------------------------------------------------------
HW model:                 Toradex Verdin iMX8M Plus on Verdin Development Board
Toradex version:          0063 V1.1B
Serial number:            15336033
Processor arch:           aarch64
------------------------------------------------------------
2

Hi, I’m the Toradex AI Assistant. I’m in early beta, and I would appreciate your feedback.
The Toradex Support team will follow up with you shortly!

Did you like my answer?

:+1: :-1:

Hello,

To address the issue of decoding high-resolution videos, including 4K, on the i.MX8M Plus using GStreamer within the Toradex Embedded Linux Reference Multimedia Image, it’s crucial to ensure that the GStreamer pipeline is optimized to leverage the hardware acceleration capabilities of the i.MX8M Plus SoC. The i.MX8M Plus integrates a Video Processing Unit (VPU) that supports hardware-accelerated decoding of video streams, which is essential for handling high-resolution content efficiently.

Given the difficulties you’ve encountered with certain resolutions and codecs, here are several recommendations and considerations:

  1. Ensure Hardware Acceleration is Enabled: For the i.MX8M Plus, make sure that the GStreamer pipeline explicitly uses elements that leverage the VPU for decoding. The v4l2h264dec and v4l2h265dec elements are designed for this purpose, as they utilize the Video4Linux2 API to access the hardware decoding features of the VPU.

  2. Optimize the GStreamer Pipeline: The example pipelines you’ve tried seem to be on the right track. However, for high-resolution video decoding, it’s critical to ensure that every element in the pipeline is capable of handling the throughput required for 4K video. This might involve using elements like imxvideoconvert_g2d for efficient colorspace conversion, which utilizes the GPU for acceleration.

  3. Check Codec Support and Limitations: The i.MX8M Plus VPU has specific capabilities and limitations regarding the video codecs it can decode in hardware. Ensure that the video streams you are trying to decode are within the supported specifications for H.264 and H.265 (HEVC) codecs. For instance, there might be limitations on the profile level or the maximum bitrate supported.

  4. Update and Verify Software Versions: Ensure that your system is running the latest versions of the BSP, the Linux kernel, and GStreamer. Updates might include important fixes or improvements that could affect video decoding performance.

  5. Debugging and Logging: Utilize GStreamer’s debugging capabilities (GST_DEBUG) to gain insights into what might be causing the decoding to fail. The logs you’ve shared indicate issues with caps negotiation and errors from the VPU decoder, suggesting that the decoder might be receiving data it cannot process or there’s a configuration issue.

  6. System Resources and Performance: Monitor the system’s resource utilization (CPU, memory) while attempting to decode high-resolution videos.

Please, also check if these links can help you solve your problem:

Video Encoding and Playback With GStreamer (Linux) | Toradex Developer Center.

3

Hello @sst,

In which datasheet did you get this information?

While the i.MX8MP supports a 4K display via HDMI, its VPU is not capable of decoding 4k video.
You can check this information on the Verdin iMX8MP datasheet, in sections 1.5.3 and 1.5.4.

The recommended pipeline element for decoding videos on the Verdin iMX8MP as of the Toradex BSP 6 is vpudec. While v4l2h264dec may work, it will not achieve optimal performance on this combination of hardware and software version.

Best Regards,
Bruno

4

Thank you for your reply. Indeed I looked in the wrong data sheet, IMX8MDQLQIEC instead of IMX8MPIEC.