Using a GPU for rendering using gstreamer

Technical Support
, ,
1

Hi Toradex,

We are using the Apalis iMX8 V1.1C (imx8qm without wifi model) on our custom carrier board with the configuration of 2x screens on 2x channel of LVDS and 1x screen connected to HDMI port (also with the touchscreen on it) - 3x screens in total.

We are trying to use some streaming using the GStreamer, but are having issues with the “imxvideoconvert_g2d” element.
I have two pipelines here:

1. gst-launch-1.0 filesrc location=/usr/data/MP4_HPL40_30fps_channel_id_51.mp4 ! qtdemux name=d d.video_0 ! h264parse ! avdec_h264 ! videoconvert ! waylandsink

2. gst-launch-1.0 filesrc location=/usr/data/MP4_HPL40_30fps_channel_id_51.mp4 ! qtdemux name=d d.video_0 ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! waylandsink

The first is using the “videoconvert” and it is displaying OK on the screen, but the video is laggy since it uses the CPU for rendering. The second pipeline is using the “imxvideoconvert_g2d” element and it is not working, the error is:

ERROR: from element /GstPipeline:pipeline0/imxvideoconvert_g2d:imxvideoconvert_g2d0: failed to activate bufferpool
Additional debug info:
../git/libs/gst/base/gstbasetransform.c(1678): default_prepare_output_buffer (): /GstPipeline:pipeline0/imxvideoconvert_g2d:imxvideoconvert_g2d0:
failed to activate bufferpool
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
Freeing pipeline ...

In the linux-toradex kernel defconfig we have enabled these options:

CONFIG_MXC_VPU_MALONE=y
CONFIG_ACPI_CPPC_CPUFREQ=y
CONFIG_STAGING=y
CONFIG_ION=y
CONFIG_ION_SYSTEM_HEAP=y
CONFIG_ION_CMA_HEAP=y
CONFIG_MXC_GPU_VIV=y

Before we have enabled the ION heap the errors we were getting that device /dev/ion is not available, now we get the problem with bufferpool sent above.

Can you help us to enable this GStreamer plugin or show us how to use the GPU for rendering to get the proper performance on the stream?

Software used:
U-Boot (u-boot-toradex) version: 2020.04-0+git.586f79f64f6a
Linux kernel (linux-toradex) version: 5.4.91+gitAUTOINC+590db576d0
Custom Yocto distribution with layers:
meta-toradex-nxp (version: dunfell-5.x.y - 599cd72b82723ac096bb5c9bea2d82bdc6b38185);
meta-toradex-bsp-common (version: dunfell-5.x.y - 2b830c7a4aaf39dc7ea971c638b5042290c9ee1e)

Kind regards,
Domagoj

2

Hi @domagoj.karl-ra !

I did some quick tests and your 2nd pipeline works for me when using the /home/root/video/testvideo_h264.mp4 that is shipped with the default Reference Multimedia Image from BSP 5.5 and Apalis iMX8QM V1.1C connected to only an HDMI output:

root@apalis-imx8-06980209:~# gst-launch-1.0 filesrc location="/home/root/video/testvideo_h264.mp4" ! qtdemux name=d d.video_0 ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! waylandsink

The following pipeline also worked with aiurdemux (that is part of NXP BSP):

root@apalis-imx8-06980209:~# gst-launch-1.0 filesrc location="/home/root/video/testvideo_h264.mp4" ! video/quicktime ! aiurdemux name=d d.video_0 ! h264parse ! v4l2h264dec ! queue ! imxvideoconvert_g2d ! waylandsink

The only difference I see is the video source. Could you share it so I can also try to playback it?

Do you face the same issue if you use only the HDMI display?

Also, I would like to share the NXP documentation about their BSP (which includes the Gstreamer elements)

From this Video Encoding and Playback With GStreamer (Linux) | Toradex Developer Center article, we have the link https://www.nxp.com/design/software/embedded-software/i-mx-software/embedded-linux-for-i-mx-applications-processors:IMXLINUX?tab=In-Depth_Tab NXP link.

From this link, we can get the documentation for the BSP that you are using. Supposing you are using BSP with a version greater than 5.2.0, from the table at Toradex’s article above mentioned, the related NXP BSP is L5.4.70_2.3.0.

Downloading its documentation (would be this link: https://www.nxp.com/webapp/Download?colCode=L5.4.70_2.3.0_LINUX_DOCS), i.MX_Linux_Release_Notes.pdf and i.MX_Linux_Users_Guide.pdf might be useful.

Best regards,

3

Hi,

First of all, sorry for the delayed answer.
I have tried these gstreamer pipelines and here are the results.

1.:

gst-launch-1.0 filesrc location="/usr/data/MP4_HPL40_30fps_channel_id_51.mp4" ! qtdemux name=d d.video_0 ! h264parse ! avdec_h264 ! videoconvert ! waylandsink

Video is working OK (the colors are OK), but is slow because it is using the CPU for rendering

2.:

gst-launch-1.0 filesrc location="/usr/data/MP4_HPL40_30fps_channel_id_51.mp4" ! qtdemux name=d d.video_0 ! h264parse ! v4l2h264dec ! videoconvert ! waylandsink

Video is displaying on the screen, but the picture is broken (lots of artefacts) and is really slow

3.:

gst-launch-1.0 filesrc location=/usr/data/MP4_HPL40_30fps_channel_id_51.mp4 ! qtdemux name=d d.video_0 ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! waylandsink
gst-launch-1.0 filesrc location=/usr/data/MP4_HPL40_30fps_channel_id_51.mp4 ! qtdemux name=d d.video_0 ! h264parse ! avdec_h264 ! imxvideoconvert_g2d ! waylandsink
gst-launch-1.0 filesrc location="/usr/data/MP4_HPL40_30fps_channel_id_51.mp4" ! video/quicktime ! aiurdemux name=d d.video_0 ! h264parse ! v4l2h264dec ! queue ! imxvideoconvert_g2d ! waylandsink

Does not work at all, show the same error:

Redistribute latency...
ERROR: from element /GstPipeline:pipeline0/imxvideoconvert_g2d:imxvideoconvert_g2d0: failed to activate bufferpool
Additional debug info:
../git/libs/gst/base/gstbasetransform.c(1678): default_prepare_output_buffer (): /GstPipeline:pipeline0/imxvideoconvert_g2d:imxvideoconvert_g2d0:
failed to activate bufferpool
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...

I am sending a full output in the txt attachment below (gstreamer_vpu-pipelines.txt).

I have also tried the pipelines without the “imxvideoconvert_g2d” element just to get something on the screens, but it does not work at all (nothing on the screens, just errors), log in the attachment (gstreamer_vpu-pipelines2.txt)

I am also sending you our “minimal-qml” that we use to start the Wayland surface and the mp4 video file.

BR,
Domagoj
gstreamer_vpu-pipelines.txt (31.3 KB)
gstreamer_vpu-pipelines2.txt (24.7 KB)

4

I now see that minimal-qml and video file is not uploaded since they are too large.
Will upload it somewhere else.

BR

5

I uploaded the minimal-qml and mp4 video file to Mediafire where you can download it and use it.

https://www.mediafire.com/file/mky7i545fm89szc/minimal-qml/file
https://www.mediafire.com/file/9capznsd1nwpzqw/MP4_HPL40_30fps_channel_id_51.mp4/file

BR
Domagoj

6

Hi @domagoj.karl-ra

Thanks for sharing the files!

We analyzed your MP4_HPL40_30fps_channel_id_51 video file and seems like it has some kind of stream error:

$ gst-discoverer-1.0 Downloads/MP4_HPL40_30fps_channel_id_51.mp4
Analyzing file:///home/user/Downloads/MP4_HPL40_30fps_channel_id_51.mp4
Done discovering file:///home/user/Downloads/MP4_HPL40_30fps_channel_id_51.mp4
An error was encountered while discovering the file
 Internal data stream error.

Properties:
  Duration: 0:01:16.543333333
  Seekable: yes
  Live: no
  container: Quicktime
    audio: E-AC-3 (ATSC A/52B)
      Stream ID: 4db383850236aa177c714701b2b898037bdcda8384d75dda165f6e5dbb52b447/002
      Language: <unknown>
      Channels: 6 (front-left, front-right, front-center, lfe1, side-left, side-right)
      Sample rate: 48000
      Depth: 32
      Bitrate: 0
      Max bitrate: 0
    video: H.264 (High Profile)
      Stream ID: 4db383850236aa177c714701b2b898037bdcda8384d75dda165f6e5dbb52b447/001
      Width: 1920
      Height: 1080
      Depth: 24
      Frame rate: 30/1
      Pixel aspect ratio: 1/1
      Interlaced: false
      Bitrate: 0
      Max bitrate: 0

This is certainly problematic for gstreamer pipelines.

Could you please execute the following tests:

  • Try using some other healthy video. You can check the video using gst-discover (just like the example above)
  • Try your pipelines on a Reference Multimedia Image.

Best regards,