GStreamer usage in a docker container for a video player plugin

Hello,

This somewhat follows this post: Flutter application in a docker container on an IMX8 on Torizon - Technical Support - Toradex Community

To Reproduce

  • Verdin iMX8M Plus on Dahlia Carrier Board
  • TorizonCore 6.1.0 (There is just a device tree overlay modification to run the DSI to LVDS adapter on a 7 inch panel)

Essentially, I have a bundled flutter application that uses the Sony video player plugin extension for Flutter. In the background it just uses gstreamer. So here is what (so far) the Dockerfile looks like:

####################################################################################################################
FROM --platform=linux/arm64/v8 arm64v8/ubuntu:20.04 AS build
ARG DEBIAN_FRONTEND=noninteractive

# Build environment dependencies for flutter-elinux
RUN apt-get -y update && apt-get install -y \
         build-essential clang cmake \
         curl git pkg-config \
         ninja-build unzip wget \
         libgtk2.0-dev libglib2.0-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev \
         libegl1-mesa-dev libxkbcommon-dev libgles2-mesa-dev \
         libwayland-dev wayland-protocols \
         locales

# Set the locale
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && \
    locale-gen
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

# Non root user
RUN groupadd -r -g 1441 flutter && useradd --no-log-init -r -u 1441 -g flutter -m flutter
USER flutter:flutter
WORKDIR /home/flutter

# Flutter 3.3.10 needs to be used (at this sha in particular) because 3.7 has this open issue:
# https://github.com/flutter/flutter/issues/116703 (as of 07.03.2023)
RUN git clone https://github.com/sony/flutter-elinux.git &&\
    cd flutter-elinux &&\
    git reset --hard 19a1b05ea20675c36c705955c4148ad40b205161 &&\
    cd ..
    
# Add flutter-elinux to the path
ENV PATH="${PATH}:/home/flutter/flutter-elinux/bin"

# While not necessary, running flutter-elinux for the first time builds it
RUN flutter-elinux devices
RUN flutter-elinux doctor

# Add the video player application
RUN git clone https://github.com/sony/flutter-elinux-plugins.git

# Get the Gallery Application
RUN cd flutter-elinux-plugins/packages/video_player/example &&\
    flutter-elinux create . &&\
    flutter-elinux upgrade && flutter-elinux clean && flutter-elinux pub get &&\
    flutter-elinux build elinux

####################################################################################################################
FROM --platform=linux/arm64/v8 torizon/weston-vivante:2 AS runtime
ARG IMAGE_ARCH

RUN mkdir -p /app/gui

RUN apt-get -y update && apt-get install -y \
         alsa-utils \
         libasound2 \
         psmisc \
         procps pciutils \
         libgstreamer1.0-0 \
	 gstreamer1.0-plugins-base \
	 gstreamer1.0-plugins-good \
	 gstreamer1.0-plugins-bad \
	 gstreamer1.0-plugins-ugly \
	 gstreamer1.0-libav \
	 gstreamer1.0-doc \
	 gstreamer1.0-tools \
	 gstreamer1.0-x \
	 gstreamer1.0-alsa \
	 gstreamer1.0-gl \
         gstreamer1.0-gtk3 \
	 gstreamer1.0-pulseaudio \
         v4l-utils \
         locales
         
# Set the locale
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && \
    locale-gen
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

# Get the flutter bundle built in the build environment
COPY --from=build /home/flutter/flutter-elinux-plugins/packages/video_player/example/build/elinux/arm64/release/bundle /app/gui/bundle

# The .so libraries (notably libflutter_engine.so) must be in the library path
# to properly execute the flutter_app. Some plugin (like the elinux video_player plugin)
# also generate a .so library, and NEED to be in the library path.
COPY --from=build /homeflutter/flutter-elinux-plugins/packages/video_player/example/build/elinux/arm64/release/bundle/lib /lib/

WORKDIR /app/gui/bundle/
ENV PATH="/app/gui:${PATH}"

ENTRYPOINT ["./video_player_example"]
CMD [ "-f", "-b", "../bundle/"]

To build the docker image containing the flutter application, from the host, on the folder where the Dockerfile is:

docker run --rm -it --privileged torizon/binfmt
docker buildx build --platform=linux/arm64 -t flutter-videoplayer:latest .
docker save flutter-videoplayer:latest > ./flutter-vp.tar
scp flutter-vp.tar torizon@ip.address:/home/torizon

If I try to launch the flutter application, I get the following error:

torizon@verdin-imx8mp-sn:~$ docker load < flutter-vp.tar
torizon@verdin-imx8mp-sn:~$ docker stop $(docker ps -a -q)
torizon@verdin-imx8mp-sn:~$ docker run -e ACCEPT_FSL_EULA=1 -d --rm --name=weston --net=host --cap-add CAP_SYS_TTY_CONFIG \
     -v /dev:/dev -v /tmp:/tmp -v /run/udev/:/run/udev/ \
     --device-cgroup-rule='c 4:* rmw' --device-cgroup-rule='c 13:* rmw' \
     --device-cgroup-rule='c 199:* rmw' --device-cgroup-rule='c 226:* rmw' \
     torizon/weston-vivante:$CT_TAG_WESTON_VIVANTE --developer weston-launch \
     --tty=/dev/tty7 --user=torizon
torizon@verdin-imx8mp-sn:~$ docker run -e ACCEPT_FSL_EULA=1 --rm -it --name=vivante-hello \
    -v /tmp:/tmp -v /dev/dri:/dev/dri \
    -v /dev/galcore:/dev/galcore \
    --device /dev/snd \
    --device-cgroup-rule='c 199:* rmw' \
    --device-cgroup-rule='c 226:* rmw' \
    --entrypoint bash flutter-videoplayer:latest
root@2acf38bcfe96:/app/gui/bundle# ./video_player_example -b ../bundle/
xdg_surface@16: error 3: xdg_surface has never been configured
EGL: errno=71 (Protocol error)
EGL: errno=71 (Protocol error)
EGL: errno=71 (Protocol error)
EGL: errno=71 (Protocol error)
...

But, the weird thing is that, if I run:

gst-launch-1.0 videotestsrc ! videoconvert ! waylandsink

And then, I run the flutter-app: ./video_player_example -b ../bundle/, this works flawlessly

Question is, what does launching a gstreamer video pipeline does that then enable a flutter app using a gstreamer plugin to then work?

Other information

With Yocto, this never happenned. Also, we can use the NXP’s imx gstreamer plugins with Yocto, but how can we do that with TorizonCore?

Hi @mnano ,

Question is, what does launching a gstreamer video pipeline does that then enable a flutter app using a gstreamer plugin to then work?

I was able to reproduce your issue on a Verdin iMX8M Plus, including the part where the application works after running gst-launch-1.0, both on TorizonCore 6.1 and on TorizonCore 5.7 LTS. While I don’t know exactly why this happens, I suspect gst-launch-1.0 sets a config file/option on Weston/Wayland that allows the app to run.

I’ve checked if any environment variables inside the container change when running gst-launch-1.0, but it doesn’t seem to be the case. I’ll continue to investigate this and I’ll update here if I find anything.

With Yocto, this never happenned. Also, we can use the NXP’s imx gstreamer plugins with Yocto, but how can we do that with TorizonCore?

You can refer to this article: How to use Gstreamer on Torizon OS | Toradex Developer Center

Keep in mind that for the moment we don’t have VPU support on TorizonCore for iMX8 devices.

Best regards,
Lucas Akira

Thank you for the quick answer and support.

I am also investigating and will report if I find anything. In the meantime, I found a small workaround:
An entrypoint.sh script:

#!/bin/bash
# Launch a pipeline on a fakesink. This does something (unknown so far) that enable
# a flutter app using the video player plugin to launch without an immediate error
gst-launch-1.0 videotestsrc ! videoconvert ! fakesink &

# Wait 5 seconds for the pipeline to properly starts and shut it down
sleep 5
pkill gst-launch-1.0

# Called the content of CMD[] situated after ENTRYPOINT[]
./$@

And I replaced the entrypoint of the Dockerfile with it:

# Entrypoint script that launch a gstreamer pipeline before starting the flutter app. Without it, 
# "xdg_surface@16: error 3: xdg_surface has never been configured" error occurs when launching the flutter application
ADD entrypoint.sh /app/gui/bundle/entrypoint.sh
RUN chmod +x entrypoint.sh

ENTRYPOINT ["./entrypoint.sh"]
CMD ["video_player_example","-f", "-b", "../bundle/"]

To launch on the target, you have to have a weston image already running of course. And, it seems we have to wait for the pipeline to properly starts. Nowhere near perfect but does the job for now.

1 Like

Hi @mnano,

Thanks for sharing this workaround.

Best Regards
Kevin

Hi @mnano ,

I think I have a clue about this issue: Someone recently reported on the Flutter GitHub page a similar problem - xdg_surface has never been configured · Issue #330 · sony/flutter-embedded-linux · GitHub .

Apparently it’s a bug in Flutter and it happens if the contents of ~/.cache/gstreamer-1.0/ don’t exist when using the video player plugin.

gst-launch-1.0 creates the missing files on ~/.cache/gstreamer-1.0/, so this explains why the app works after running it.

Until this is fixed by the Flutter team I suggest using your workaround or maybe adding the needed files in the container image.

Best regards,
Lucas Akira

1 Like

Thank you for the quick support.

It seems to be the problem. I will continue using the workaround in the meantime and track this issue on the flutter-embedded-linux repository.

Glad I was able to help!

Best regards,
Lucas Akira