Gstreamer support for TK1 with USB 3.0 uncompressed cameras

Hello,

I am trying to use your image 2.8b2.97.

I have USB 3.0 cameras that can output uncompressed YUV422 1080p60 video. I was trying to capture this using gstreamer and I am getting some issues.

First, with your image, you have no gstreamer plugin in order to get hardware acceleration to render on HDMI.
It also seem that this image does not contains any nVidia specifc plugins in order to decode/encode using hardware GPU. What are the steps to includes them in custom image using bitbake?

Second, it seems the CPU is overloaded even when issuing the following pipeline that renders stream to fakesink:

gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080' ! fakesink sync=false

This is what I am getting on console when I start this command as a sample:

[  332.203323] tegra-xusb 70090000.usb: ERROR unknown event type 37
[  332.237711] tegra-xusb 70090000.usb: ERROR unknown event type 37
[  332.272157] tegra-xusb 70090000.usb: ERROR unknown event type 37
[  332.306568] tegra-xusb 70090000.usb: ERROR unknown event type 37
[  332.340960] tegra-xusb 70090000.usb: ERROR unknown event type 37
[  332.375361] tegra-xusb 70090000.usb: ERROR unknown event type 37
[  332.409775] tegra-xusb 70090000.usb: ERROR unknown event type 37
[  332.444162] tegra-xusb 70090000.usb: ERROR unknown event type 37
[  332.478621] tegra-xusb 70090000.usb: ERROR unknown event type 37
[  332.513067] tegra-xusb 70090000.usb: ERROR unknown event type 37
[  332.547497] tegra-xusb 70090000.usb: ERROR unknown event type 37
[  332.581943] tegra-xusb 70090000.usb: ERROR unknown event type 37

I know there are issues using kernel 3.10.xx with USB 3.0 devices with other platforms so I was hoping that your TK1 module was working better using mainline kernel 4.14.30.

I was testing this module in late 2016 and I had some problems to do this specific setup. I was hoping this kind of problem went away.

Are you aware of such issues with USB 3.0/gstreamer/nVidia plugins?

Are there any other way to capture USB 3.0 content and render it on HDMI output at a decent frame rate?

Thanks,

Jérôme

First, the mainline image does not contain any gstreamer plugins as there is no multimedia support in mainline so far.

Second, while basic USB 3.0 SuperSpeed should work fine in mainline I don’t think we ever tried much camera use cases.

We are not aware of any USB 3.0 SuperSpeed issues in downstream Linux kernel 3.10 other than the missing USBO1 support which we just implemented.

Are you aware of such issues with USB 3.0/gstreamer/nVidia plugins?

No, not really. I suggest for you to ask a new question about any specific issues you may encounter with downstream and we will have a look at it.

Thanks for the details.
Can you tell me how to include gstreamer plugins to the mainline release? Is it possible?
Why multimedia is only supported by this older kernel?
Like I said, we tried multiple platforms like x86 and i.MX6 (using PCIe-to-USB 3.0 bridge) and all of them showed performance issues over USB 3.0 using this older kernel.
Thanks for your help.
Jerome

Thanks for the details.

You are very welcome.

Can you tell me how to include gstreamer plugins to the mainline release?

The question is what gstreamer plugin(s)? There are hundreds if not thousand thereof which may be included by adding resp. recipe’s package(s) aka build artifacts to your image recipe or e.g. via IMAGE_INSTALL_append in your local.conf.

Is it possible?

Basic gstreamer support is certainly possible. However, as mentioned above the ones supporting any kind of hardware acceleration are unfortunately NVIDIA proprietary and not really open source.

Why multimedia is only supported by this older kernel?

That has to do with the NVIDIA proprietary IP involved for which e.g. also no documentation is available. NVIDIA only implemented and optimised this in their Linux for Tegra aka L4T based on that older downstream Linux kernel.

Like I said, we tried multiple platforms like x86 and i.MX6 (using PCIe-to-USB 3.0 bridge) and all of them showed performance issues over USB 3.0 using this older kernel.

Like I said, we are not aware of any such issue(s) and would need more concrete information to investigate any such.

Thanks for your help. Jerome

We were looking for v4l2src, nvhdmioverlaysink, nvoverlaysink, nvvidconv and all NVIDIA decoders and encoders that are using hardware acceleration.

If we go with this module on our upcoming product, I will need to stick with this L4T image, which is quite big for the purpose of our next design. There will be no way to include the plugings to a Yocto custom image using IMAGE_INSTALL_append in local.conf? I tried the following without success.

IMAGE_INSTALL_append += " gstreamer gstreamer1.0-omx-tegra gst-plugins-base gst-plugins-good gst-plugins-bad v4l-utils"

Maybe this is because what you just explained it is only supported by the downstream Linux kernel.

Thanks.

We were looking for v4l2src, nvhdmioverlaysink, nvoverlaysink, nvvidconv and all NVIDIA decoders and encoders that are using hardware acceleration.

Yes, other than v4l2src those are only supported in downstream as far as I know.

If we go with this module on our upcoming product, I will need to stick with this L4T image,

No, not at all. Our regular LXDE demo images also includes all of those and is a regular Angstrom distribution/OpenEmbedded/Yocto project based image.

which is quite big for the purpose of our next design. There will be no way to include the plugings to a Yocto custom image using IMAGE_INSTALL_append in local.conf?

No, as mentioned above those should really already be included.

I tried the following without success.

IMAGE_INSTALL_append += " gstreamer gstreamer1.0-omx-tegra gst-plugins-base gst-plugins-good gst-plugins-bad v4l-utils"
Maybe this is because what you just explained it is only supported by the downstream Linux kernel.

I believe you are confusing upstream aka mainline vs. downstream Linux kernel with L4T vs. Yocto Project image. Of course, while L4T is always downstream, you may build a Yocto Project image either based on the mainline or downstream Linux kernel.