USB Camera inconsistent framerate

Technical Support
, ,
1

I have bought a usb camera with uvc drivers so it is plug and play on the mallow carrier board which we are using. I am noticing though when i run the camera, i get large scale fluctuations on my loop timings. The camera seems to jog between 25fps to 17fps. The loop is doing a bunch of other things, but when i run the script on a csi camera, i get a consistent 22 fps output. This is all done in python using opencv.

Any ideas on what could be causing this?

tdx-info output
Software summary

Bootloader: U-Boot
Kernel version: 5.15.129-6.4.0+git.67c3153d20ff #1-TorizonCore SMP PREEMPT Wed Sep 27 12:30:36 UTC 2023
Kernel command line: root=LABEL=otaroot rootfstype=ext4 quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3 ostree=/ostree/boot.0/torizon/460c0c1dde52ce6a4fd14e6ff7399ad1c879175a9ffa047acbb97523d9376591/0
Distro name: NAME=“TorizonCore”
Distro version: VERSION_ID=6.4.0-build.5
Hostname: verdin-imx8mp-15403321

Hardware info

HW model: Toradex Verdin iMX8M Plus on Verdin Development Board
Toradex version: 0063 V1.1B
Serial number: 15403321
Processor arch: aarch64

2

Hi, I’m the Toradex AI Assistant. The Toradex Support team will follow up with you shortly!

Did you like my answer?

:+1: :-1:

The inconsistent frame rate you’re experiencing with the USB camera could be due to several factors:

  1. USB Bandwidth and Latency: USB cameras can be affected by bandwidth limitations and latency, especially if other USB devices are connected to the same bus. This can cause fluctuations in frame rate.

  2. CPU Load: Since your loop is doing a bunch of other things, it’s possible that the CPU is being overloaded, which can affect the processing of video frames. The CSI camera might be less affected due to its direct connection to the SoC, which can handle data more efficiently.

  3. Driver and Kernel Version: Ensure that the UVC drivers are up-to-date and compatible with your kernel version. Sometimes, driver issues can cause performance inconsistencies.

  4. Power Management: Check if there are any power management settings that might be affecting the USB ports, causing them to throttle under certain conditions.

  5. OpenCV and Python Overhead: Python and OpenCV can introduce additional overhead compared to lower-level implementations. Consider optimizing your code or using a more efficient language for critical sections.

  6. GStreamer Pipeline: As mentioned in the context, using GStreamer might provide a more stable pipeline for video capture. You could try setting up a GStreamer pipeline to see if it offers more consistent performance.

You might want to monitor the system’s resource usage (CPU, memory, USB bandwidth) while running your application to identify any bottlenecks. Additionally, testing with a different USB port or reducing the load in your loop could help isolate the issue.

Please, also check if these links can help you solve your problem:

Video processing pipeline for i.MX8.

3

  1. USB Bandwidth and latency: No other usb devices are currently on the bus. The camera came with a USB3.1 cable as well.

  2. CPU Load: The unit does not run above 25% CPU usage ever

  3. Driver and Kernel Version:
    v4l2-ctl --list-devices --device /dev/video2 -D output:

Driver Info:
Driver name : uvcvideo
Card type : See3CAM_CU55
Bus info : usb-xhci-hcd.1.auto-1.1
Driver version : 5.15.129
Capabilities : 0x84a00001
Video Capture
Metadata Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04200001
Video Capture
Streaming
Extended Pix Format
Media Driver Info:
Driver name : uvcvideo
Model : See3CAM_CU55
Serial : 28295101
Bus info : usb-xhci-hcd.1.auto-1.1
Media version : 5.15.129
Hardware revision: 0x00000000 (0)
Driver version : 5.15.129
Interface Info:
ID : 0x03000002
Type : V4L Video
Entity Info:
ID : 0x00000001 (1)
Name : See3CAM_CU55
Function : V4L2 I/O
Flags : default
Pad 0x01000007 : 0: Sink
Link 0x02000010: from remote pad 0x100000a of entity ‘Extension 3’ (Video Pixel Formatter): Data, Enabled, Immutable

  1. Power Management: I have disabled USB Power Management Globally
4

Hi @btj2 ,

Distro version: VERSION_ID=6.4.0-build.5

It is TorizonCore 6.4.0 on Verdin iMX8MP. Could you share how you bring up the container where the python camera application using opencv runs?

v4l2-ctl --device=/dev/video2 --list-formats-ext

I assume /dev/video2 is your USB camera device. The command above will display the supported formats for the USB camera.

5

Hi Benjamin,

We docker-compose up, using our python application in Torizon IDE V2. We used docker exec -it to run the command in the container.

Here is the output from the command above:

root@a397081843ad:/home/torizon/app# v4l2-ctl --list-formats-ext --device /dev/video2 ioctl: VIDIOC_ENUM_FMT Type: Video Capture [0]: ‘UYVY’ (UYVY 4:2:2) Size: Discrete 640x480 Interval: Discrete 0.017s (60.000 fps) Interval: Discrete 0.033s (30.000 fps) Size: Discrete 1280x720 Interval: Discrete 0.017s (60.000 fps) Interval: Discrete 0.033s (30.000 fps) Size: Discrete 1280x960 Interval: Discrete 0.017s (60.000 fps) Interval: Discrete 0.033s (30.000 fps) Size: Discrete 1920x1080 Interval: Discrete 0.017s (60.000 fps) Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.067s (15.000 fps) Size: Discrete 2560x1440 Interval: Discrete 0.031s (32.000 fps) Interval: Discrete 0.062s (16.000 fps) Size: Discrete 2592x1944 Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.083s (12.000 fps) [1]: ‘MJPG’ (Motion-JPEG, compressed) Size: Discrete 640x480 Interval: Discrete 0.017s (60.000 fps) Size: Discrete 1280x720 Interval: Discrete 0.017s (60.000 fps) Size: Discrete 1280x960 Interval: Discrete 0.017s (60.000 fps) Size: Discrete 1920x1080 Interval: Discrete 0.017s (60.000 fps) Size: Discrete 2560x1440 Interval: Discrete 0.017s (60.000 fps) Size: Discrete 2592x1944 Interval: Discrete 0.017s (60.000 fps)

6

Hi Benedict,

UYVY 4:2:2 and MJPEG formats are supported by your cameras, with resolutions ranging from 640x480 to 2592x1944. Are the format and resolution settings configured in the Python application?

7

Hi Benedict,

MJPEG decoding is not accelerated by iMX8M Plus VPU. This also affects the USB camera performance.

8

Hi Benedict,

I do more tests with a USB camera.
Build a container with VPU accelerated according to this page.
set ARG BASE_VERSION to 3.2.1 so that it builds successfully.

diff --git a/vpu/generic/vpuGeneric/Dockerfile b/vpu/generic/vpuGeneric/Dockerfile
index 5fd6a80..06078a1 100644
--- a/vpu/generic/vpuGeneric/Dockerfile
+++ b/vpu/generic/vpuGeneric/Dockerfile
@@ -1,4 +1,5 @@
-ARG BASE_VERSION=3
+#ARG BASE_VERSION=3
+ARG BASE_VERSION=3.2.1

 ##
 # Board architecture

add command: tail -f /dev/null to the built container in the docker-compose file. It will stop the default vputests.sh from launching and allow you to show video from the USB camera with gst pipeline.

diff --git a/vpu/generic/vpuGeneric/docker-compose.yml b/vpu/generic/vpuGeneric/docker-compose.yml
index 7723b7e..54b3dd4 100644
--- a/vpu/generic/vpuGeneric/docker-compose.yml
+++ b/vpu/generic/vpuGeneric/docker-compose.yml
@@ -5,6 +5,7 @@ services:
       context: .
       dockerfile: Dockerfile
     image: ${DOCKER_LOGIN}/vpugeneric:${TAG}
+    command: tail -f /dev/null
     environment:
       #- MACHINE=verdin-imx8mm
       - MACHINE=verdin-imx8mp

attach to the container and launch the pipe. With sync disabled, the USB camera shows 1920*1080 video at an average of 29fps.

$ docker exec -it torizon-vpugeneric-1 bash

# gst-launch-1.0 v4l2src device=/dev/video2 ! image/jpeg,width=1920,height=1080,framerate=30/1 ! jpegparse ! jpegdec ! fpsdisplaysink video-sink=waylandsink sync=false
...
Total showed frames (1325), playing for (0:00:46.353280645), fps (28.585).
9

Hi Benedict, has the USB camera framerate issue been resolved?