Hardware-Accelerated JPEG Decoding with GStreamer on i.MX 8QM running Torizon OS 7

Technical Support
, , ,
1

I am working on a real-time video processing application using OpenCV and GStreamer. My current GStreamer pipeline is hitting a performance bottleneck, and I suspect the software-based JPEG decoding is the primary cause, preventing me from achieving the camera’s maximum frame rate of 120 FPS.

My GStreamer pipeline is as follows:

v4l2src device=/dev/video99 io-mode=4 extra-controls=“c,gamma=100,gain=0,auto_exposure=3” ! image/jpeg, width=1280, height=720, framerate=120/1 ! queue max-size-buffers=1 leaky=downstream ! jpegdec ! videocrop top=100 left=100 right=100 bottom=100 ! videoconvert ! video/x-raw,format=GRAY8 ! appsink max-buffers=1 drop=true sync=false

I have attempted to integrate i.MX specific GStreamer plugins into my Docker container by following How do I integrate IMX specific plugins for gstreamer in a docker container? - #13 by allan.tx . While I was able to successfully install v4l2h264dec, v4l2h265dec, and imxvideoconvert_g2d, I have not been able to install a hardware-accelerated equivalent for the jpegdec element.

Camera specs:
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture

    [0]: 'MJPG' (Motion-JPEG, compressed)
            Size: Discrete 1280x800
                    Interval: Discrete 0.008s (120.000 fps)
                    Interval: Discrete 0.010s (100.000 fps)
            Size: Discrete 1280x720
                    Interval: Discrete 0.008s (120.000 fps)
            Size: Discrete 800x600
                    Interval: Discrete 0.008s (120.000 fps)
                    Interval: Discrete 0.010s (100.000 fps)
            Size: Discrete 640x480
                    Interval: Discrete 0.008s (120.000 fps)
                    Interval: Discrete 0.010s (100.000 fps)
            Size: Discrete 320x240
                    Interval: Discrete 0.008s (120.000 fps)
                    Interval: Discrete 0.010s (100.000 fps)
    [1]: 'YUYV' (YUYV 4:2:2)
            Size: Discrete 1280x800
                    Interval: Discrete 0.100s (10.000 fps)
            Size: Discrete 1280x720
                    Interval: Discrete 0.100s (10.000 fps)
2

Hi, I’m the Toradex AI Assistant.

I might make mistakes. Our support team will step in if needed. If I answered your question, please tick the “Solution” checkbox. For follow-up questions to me, please use @ToradexAI in your post.

Did you like my answer? :+1: :-1:

Hardware-Accelerated JPEG Decoding Limitation

Unfortunately, there is currently no hardware-accelerated JPEG decoding plugin available for Toradex modules running on i.MX SoCs(1). The NXP i.MX SoCs offer hardware acceleration primarily for video codecs such as H.264, H.265, and VP8/9, but JPEG hardware decoding is not exposed via a GStreamer element(1).

In your pipeline, jpegdec is a software decoder, and there is no hardware-accelerated replacement currently available for this platform(1).

Alternative Approaches to Improve Performance

1. Use Raw Video Format Instead of MJPEG

Since your camera supports both MJPEG and YUYV formats, consider switching to the raw YUYV format to eliminate JPEG decoding entirely. Based on your camera specs, you can capture at 10 FPS with YUYV format:

v4l2src device=/dev/video99 ! "video/x-raw, format=YUY2, framerate=10/1, width=1280, height=720" ! videocrop top=100 left=100 right=100 bottom=100 ! videoconvert ! video/x-raw,format=GRAY8 ! appsink max-buffers=1 drop=true sync=false

(2)

2. Optimize Your Current Pipeline

For your existing MJPEG pipeline, you can optimize it using GStreamer’s debugging capabilities(3):

export GST_DEBUG=2

(3)

This will help identify bottlenecks in your pipeline(3).

3. Consider Hardware Acceleration for Other Elements

While JPEG decoding cannot be hardware-accelerated, you can leverage hardware acceleration for other video processing tasks. For example, if you need to encode the processed video, you can use hardware-accelerated encoders like vpuenc_h264(3):

gst-launch-1.0 videotestsrc num-buffers=300 ! video/x-raw, width=1920, height=1080, framerate=60/1 ! videoconvert ! vpuenc_h264 ! video/x-h264 ! h264parse ! qtmux ! filesink location=videotestsrc.mp4

(3)

Performance Optimization Recommendations

  1. Lower JPEG Quality from Source: If you can control the webcam settings, lowering the JPEG quality may reduce CPU usage(1)
  2. Downscale Before Processing: Capturing at a lower resolution will reduce the CPU load(1)
  3. Pipeline Buffer Management: Your current pipeline already uses appropriate buffer management with max-buffers=1 and drop=true

Future Considerations

Keep monitoring software updates from NXP or Toradex, as future releases may introduce new hardware-accelerated plugins or kernel features(1). However, there are currently no public plans or roadmap regarding the implementation of hardware-accelerated JPEG decoding(1).

Best regards, Toradex AI Assistant.