NPU delgate & Dockerfile Creation HELP!

btj2 · September 16, 2024, 12:42pm

Hi there,

I would like some help with out tensorflow python application where the Dockerfile.debug is working (NPU delegate is working and UINT8 model loaded) and running our camera at 20FPS. When trying to build a production image, the NPU delegate is not loaded and defaults to CPU making the camera inferencing run at 1FPS. I have posted the working Dockerfile.debug below and the not working Dockerfile also (all of the requirements.txt files and torizonPackages.json are the same). I believe the issue is the .venv and how the APP_ROOT dir is handled between them but I can’t quite figure it out.

Please help.

Kind regards,

btj2.

Dockerfile.debug:

ARGUMENTS --------------------------------------------------------------------

Board architecture

ARG IMAGE_ARCH=

Base container version

ARG BASE_VERSION=3.3.1-bookworm

Debug port

ARG DEBUG_SSH_PORT=

Run as

ARG SSHUSERNAME=

Directory of the application inside container

ARG APP_ROOT=

Build image ----------------------------------------------------------------

FROM --platform=linux/${IMAGE_ARCH} \
torizon/debian:${BASE_VERSION} AS Base

Install Python

RUN apt-get -y update && apt-get install -y
python3 python3-dev python3-numpy python3-pybind11
&& apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

Install Pip and Wheel

RUN apt-get -y update && apt-get install -y
python3-pip python3-setuptools python3-wheel
&& apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

Install Build tools

RUN apt-get -y update && apt-get install -y
cmake build-essential gcc g++ git wget unzip patchelf imx-gpu-viv-wayland-dev
&& apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

I uncommented this section to build tensorflow lite from source ###4:38PM

RUN apt-get -y update && apt-get install -y
libovxlib-dev libneuralnetworks libnnrt-dev
&& apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

Prepare build output directory

COPY global_variables.sh global_variables.sh

Build nn-imx

COPY nn-imx_1.3.0.sh nn-imx_1.3.0.sh
RUN chmod +x nn-imx_1.3.0.sh && ./nn-imx_1.3.0.sh

Build tim vx

COPY tim-vx.sh tim-vx.sh
RUN chmod +x tim-vx.sh && ./tim-vx.sh

Build tensorflow lite

COPY tensorflow-lite_2.9.1.sh tensorflow-lite_2.9.1.sh
RUN chmod +x tensorflow-lite_2.9.1.sh && ./tensorflow-lite_2.9.1.sh

Build tflite vx delegate

COPY tflite-vx-delegate.sh tflite-vx-delegate.sh
RUN chmod +x tflite-vx-delegate.sh && ./tflite-vx-delegate.sh

Deploy image ---------------------------------------------------------------

FROM --platform=linux/${IMAGE_ARCH}
commontorizon/debian:${BASE_VERSION} AS debug

ARG IMAGE_ARCH
ARG DEBUG_SSH_PORT
ARG APP_EXECUTABLE
ARG SSHUSERNAME
ENV APP_EXECUTABLE ${APP_EXECUTABLE}
ARG APP_ROOT

SSH for remote debug

EXPOSE ${DEBUG_SSH_PORT}
EXPOSE 6512

Make sure we don’t get notifications we can’t answer during building.

ENV DEBIAN_FRONTEND=“noninteractive”

your regular RUN statements here

Install required packages

RUN apt-get -q -y update &&
apt-get -q -y install
openssl
openssh-server
rsync
file
screen
python3-minimal
python3-pip
python3-setuptools
python3-venv
&& apt-get clean && apt-get autoremove &&
rm -rf /var/lib/apt/lists/*

automate for torizonPackages.json

RUN apt-get -q -y update &&
apt-get -q -y install \

DO NOT REMOVE THIS LABEL: this is used for VS Code automation

# __torizon_packages_dev_start__
    python3-debugpy:arm64 \
    python3-opencv:arm64 \
    python3-libgpiod:arm64 \
    python3-setuptools:all \
    python3-pybind11:all \
    python3-scipy:arm64 \
    python3-psutil:arm64 \
    lsof:arm64 \
    v4l-utils:arm64 \
    bc:arm64 \
    python3-spidev:arm64 \
# __torizon_packages_dev_end__

DO NOT REMOVE THIS LABEL: this is used for VS Code automation

&& \
apt-get clean && apt-get autoremove && \
rm -rf /var/lib/apt/lists/*

Create virtualenv

RUN python3 -m venv ${APP_ROOT}/.venv --system-site-packages

Install pip packages on venv

COPY requirements-debug.txt /requirements-debug.txt
RUN . ${APP_ROOT}/.venv/bin/activate &&
pip3 install --upgrade pip && pip3 install -r requirements-debug.txt &&
rm requirements-debug.txt &&
chown -R ${SSHUSERNAME}:${SSHUSERNAME} ${APP_ROOT}

Install TF Lite

Rebember to unzip build.zip before running it

COPY build /build

If you want to build it from source, comment the line above and uncomment the line below

#COPY --from=Base /build /build
RUN cp -r /build/* /
RUN . ${APP_ROOT}/.venv/bin/activate && pip3 install tflite_runtime-2.9.1-cp311-cp311-linux_aarch64.whl && rm -rf *.whl

Print current working directory and list files

RUN . ${APP_ROOT}/.venv/bin/activate &&
echo “Python path:” && which python3 &&
python3 --version &&
echo “Pip path:” && which pip3 &&
pip3 --version &&
pwd > /home/torizon/dir_check.txt && ls -la . >> /home/torizon/dir_check.txt

RUN apt-get -y update && apt-get install -y
libneuralnetworks libovxlib-dev libnnrt-dev
&& apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

Install GStreamer and rtsp libraries

RUN apt-get -y update && apt-get install -y --no-install-recommends libgstreamer1.0-0
gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad
gstreamer1.0-plugins-ugly gstreamer1.0-tools gstreamer1.0-OpenCV
python3-gst-1.0 gstreamer1.0-rtsp libgstrtspserver-1.0-0 gir1.2-gst-rtsp-server-1.0 &&
apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

Set the environment variable to enable NPU acceleration

ENV USE_HW_ACCELERATED_INFERENCE=1
ENV USE_GPU_INFERENCE=0
ENV ACCEPT_FSL_EULA=1
ENV CAPTURE_DEVICE=/dev/video2

DEBUG PURPOSES ONLY!!

THIS CONFIGURES AN INSECURE SSH CONNECTION

create folders needed for the different components

configures SSH access to the container and sets environment by default

RUN mkdir /var/run/sshd &&
sed ‘s@session\srequired\spam_loginuid.so@session optional pam_loginuid.so@g’
-i /etc/pam.d/sshd &&
if test $SSHUSERNAME != root ;
then mkdir -p /home/$SSHUSERNAME/.ssh ;
else mkdir -p /root/.ssh ; fi &&
echo “PermitUserEnvironment yes” >> /etc/ssh/sshd_config &&
echo “PermitRootLogin yes” >> /etc/ssh/sshd_config &&
echo “Port ${DEBUG_SSH_PORT}” >> /etc/ssh/sshd_config &&
echo “PasswordAuthentication yes” >> /etc/ssh/sshd_config &&
echo “PermitEmptyPasswords yes” >> /etc/ssh/sshd_config &&
su -c “env” $SSHUSERNAME > /etc/environment &&
echo “$SSHUSERNAME:” | chpasswd -e

RUN rm -r /etc/ssh/ssh*key &&
dpkg-reconfigure openssh-server

Copy the application source code in the workspace to the $APP_ROOT directory

path inside the container, where $APP_ROOT is the torizon_app_root

configuration defined in settings.json

Print current working directory and list files

RUN . ${APP_ROOT}/.venv/bin/activate &&
echo “Python path:” && which python3 &&
python3 --version &&
echo “Pip path:” && which pip3 &&
pip3 --version &&
pwd > /home/torizon/dir_check.txt && ls -la . >> /home/torizon/dir_check.txt

COPY --chown=$SSHUSERNAME:$SSHUSERNAME ./src ${APP_ROOT}/src

CMD [ “/usr/sbin/sshd”, “-D” ]

Dockerfile:

ARGUMENTS --------------------------------------------------------------------

Board architecture

ARG IMAGE_ARCH=

Base container version

ARG BASE_VERSION=3.3.1-bookworm

Directory of the application inside container

ARG APP_ROOT=app

Build image ----------------------------------------------------------------

FROM --platform=linux/${IMAGE_ARCH}
torizon/debian:${BASE_VERSION} AS Base

Install Python

RUN apt-get -y update && apt-get install -y
python3 python3-dev python3-numpy python3-pybind11
&& apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

Install Pip and Wheel

RUN apt-get -y update && apt-get install -y
python3-pip python3-setuptools python3-wheel
&& apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

Install Build tools

RUN apt-get -y update && apt-get install -y
cmake build-essential gcc g++ git wget unzip patchelf imx-gpu-viv-wayland-dev
&& apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

I uncommented this section to build tensorflow lite from source ###4:38PM

RUN apt-get -y update && apt-get install -y
libovxlib-dev libneuralnetworks libnnrt-dev
&& apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

Prepare build output directory

COPY global_variables.sh global_variables.sh

Build nn-imx

COPY nn-imx_1.3.0.sh nn-imx_1.3.0.sh
RUN chmod +x nn-imx_1.3.0.sh && ./nn-imx_1.3.0.sh

Build tim vx

COPY tim-vx.sh tim-vx.sh
RUN chmod +x tim-vx.sh && ./tim-vx.sh

Build tensorflow lite

COPY tensorflow-lite_2.9.1.sh tensorflow-lite_2.9.1.sh
RUN chmod +x tensorflow-lite_2.9.1.sh && ./tensorflow-lite_2.9.1.sh

Build tflite vx delegate

COPY tflite-vx-delegate.sh tflite-vx-delegate.sh
RUN chmod +x tflite-vx-delegate.sh && ./tflite-vx-delegate.sh

Make sure we don’t get notifications we can’t answer during building.

ENV DEBIAN_FRONTEND=“noninteractive”

your regular RUN statements here

Install required packages

RUN apt-get -q -y update &&
apt-get -q -y install
openssl
rsync
file
screen
python3-minimal
python3-pip
python3-setuptools
python3-venv
&& apt-get clean && apt-get autoremove &&
rm -rf /var/lib/apt/lists/*

automate for torizonPackages.json

RUN apt-get -q -y update &&
apt-get -q -y install \

DO NOT REMOVE THIS LABEL: this is used for VS Code automation

# __torizon_packages_dev_start__
    python3-debugpy:arm64 \
    python3-opencv:arm64 \
    python3-libgpiod:arm64 \
    python3-setuptools:all \
    python3-pybind11:all \
    python3-scipy:arm64 \
    python3-psutil:arm64 \
    lsof:arm64 \
    v4l-utils:arm64 \
    bc:arm64 \
    python3-spidev:arm64 \
# __torizon_packages_dev_end__

DO NOT REMOVE THIS LABEL: this is used for VS Code automation

&& \
apt-get clean && apt-get autoremove && \
rm -rf /var/lib/apt/lists/*

Create virtualenv inside the container (removed the COPY ./.venv line)

RUN python3 -m venv ${APP_ROOT}/.venv --system-site-packages

Install pip packages on venv

COPY requirements-release.txt /requirements-release.txt
RUN . ${APP_ROOT}/.venv/bin/activate &&
pip3 install --upgrade pip && pip3 install -r requirements-release.txt &&
rm requirements-release.txt

Install TF Lite

COPY build /build
RUN cp -r /build/* /

No need to COPY ./.venv anymore; create the virtualenv inside the container

RUN . ${APP_ROOT}/.venv/bin/activate && pip3 install tflite_runtime-2.9.1-cp311-cp311-linux_aarch64.whl && rm -rf *.whl

RUN apt-get -y update && apt-get install -y
libneuralnetworks libovxlib-dev libnnrt-dev
&& apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

Install GStreamer and rtsp libraries

RUN apt-get -y update && apt-get install -y --no-install-recommends libgstreamer1.0-0
gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad
gstreamer1.0-plugins-ugly gstreamer1.0-tools gstreamer1.0-OpenCV
python3-gst-1.0 gstreamer1.0-rtsp libgstrtspserver-1.0-0 gir1.2-gst-rtsp-server-1.0 &&
apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

Set the environment variable to enable NPU acceleration

ENV USE_HW_ACCELERATED_INFERENCE=1
ENV USE_GPU_INFERENCE=0
ENV ACCEPT_FSL_EULA=1
ENV CAPTURE_DEVICE=/dev/video3

Copy the application source code in the workspace to the $APP_ROOT directory

COPY --chown=${APP_ROOT}:${APP_ROOT} ./src ${APP_ROOT}/src

Final CMD for running the application using Python from the virtual environment

CMD ${APP_ROOT}/.venv/bin/python3 ${APP_ROOT}/src/main.py --no-sandbox

ToradexAI · September 16, 2024, 12:43pm

Hi, I’m the Toradex AI Assistant. The Toradex Support team will follow up with you shortly!

Did you like my answer?

It looks like the main issue is related to how the virtual environment (.venv) and the application root directory (APP_ROOT) are handled between the Dockerfile.debug and the production Dockerfile. Here are a few suggestions to help you troubleshoot and resolve the issue:

Ensure Consistent Environment Setup:
- Make sure that the environment variables and paths are consistently set up in both Dockerfiles. Specifically, check the APP_ROOT and .venv paths.
Check Virtual Environment Activation:
- Ensure that the virtual environment is correctly activated before running the application. In the production Dockerfile, you have:
```
CMD ${APP_ROOT}/.venv/bin/python3 ${APP_ROOT}/src/main.py --no-sandbox
```
  This should work if the virtual environment is correctly set up. However, you might want to add a step to explicitly activate the virtual environment and verify it.
Compare Installed Packages:
- Verify that the same packages are installed in both the debug and production environments. You can do this by comparing the requirements-debug.txt and requirements-release.txt files.
Check for Missing Dependencies:
- Ensure that all necessary dependencies for the NPU delegate are installed in the production image. This includes libraries like libneuralnetworks, libovxlib-dev, and libnnrt-dev.

Debugging Steps:

Add debugging steps to the production Dockerfile to print out the environment and paths. For example:

RUN . ${APP_ROOT}/.venv/bin/activate && \
    echo "Python path:" && which python3 && \
    python3 --version && \
    echo "Pip path:" && which pip3 && \
    pip3 --version && \
    pwd > /home/torizon/dir_check.txt && ls -la . >> /home/torizon/dir_check.txt

Environment Variables:
- Ensure that the environment variables for hardware acceleration are correctly set in the production Dockerfile. You have:
```
ENV USE_HW_ACCELERATED_INFERENCE=1
ENV USE_GPU_INFERENCE=0
ENV ACCEPT_FSL_EULA=1
ENV CAPTURE_DEVICE=/dev/video3
```
Permissions:
- Ensure that the permissions for the application root directory and virtual environment are correctly set. In the production Dockerfile, you have:
```
COPY --chown=${APP_ROOT}:${APP_ROOT} ./src ${APP_ROOT}/src
```

Here is a modified version of your production Dockerfile with some additional debugging steps:

# ARGUMENTS
ARG IMAGE_ARCH=
ARG BASE_VERSION=3.3.1-bookworm
ARG APP_ROOT=app

# Build image
FROM --platform=linux/${IMAGE_ARCH} torizon/debian:${BASE_VERSION} AS Base

# Install Python
RUN apt-get -y update && apt-get install -y \
    python3 python3-dev python3-numpy python3-pybind11 \
    && apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

# Install Pip and Wheel
RUN apt-get -y update && apt-get install -y \
    python3-pip python3-setuptools python3-wheel \
    && apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

# Install Build tools
RUN apt-get -y update && apt-get install -y \
    cmake build-essential gcc g++ git wget unzip patchelf imx-gpu-viv-wayland-dev \
    && apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

# Install additional libraries
RUN apt-get -y update && apt-get install -y \
    libovxlib-dev libneuralnetworks libnnrt-dev \
    && apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

# Prepare build output directory
COPY global_variables.sh global_variables.sh

# Build nn-imx
COPY nn-imx_1.3.0.sh nn-imx_1.3.0.sh
RUN chmod +x nn-imx_1.3.0.sh && ./nn-imx_1.3.0.sh

# Build tim vx
COPY tim-vx.sh tim-vx.sh
RUN chmod +x tim-vx.sh && ./tim-vx.sh

# Build tensorflow lite
COPY tensorflow-lite_2.9.1.sh tensorflow-lite_2.9.1.sh
RUN chmod +x tensorflow-lite_2.9.1.sh && ./tensorflow-lite_2.9.1.sh

# Build tflite vx delegate
COPY tflite
            
Please, also check if these links can help you solve your problem:

https://community.toradex.com/t/apollox-creating-production-docker-image-fails/18516

https://community.toradex.com/t/npu-acceleration-for-tflite-python-application/21002

https://developer.toradex.com/linux-bsp/application-development/machine-learning/building-machine-learning-software-with-reference-images-for-yocto-project/.

jeremias.tx · September 16, 2024, 8:44pm

Greetings @btj2,

It’s very difficult to parse what you’ve shared. Could you please format the information properly so it’s easier to examine.

It looks like you’re basing your project off this example: torizon-samples/tflite/tflite-rtsp at bookworm · toradex/torizon-samples · GitHub

Is that correct?

One thing I noticed is that you’re setting the following variables in the Dockerfile:

ENV USE_HW_ACCELERATED_INFERENCE=1
ENV USE_GPU_INFERENCE=0
ENV ACCEPT_FSL_EULA=1
ENV CAPTURE_DEVICE=/dev/video3

If you look at the sample, these were meant to be set in your docker run command or in your docker-compose.yml. What does your docker-compose.yml file look like?

Again it would be appreciated if you could neatly format your Dockerfiles. It’s difficult to compare the debug and production Dockerfiles in this state.

Best Regards,
Jeremias

btj2 · October 2, 2024, 6:34am

Hi Jeremias,

I resolved the issue thank you. I am now onto trying to access more GPIO pins on the Mallow carrier (Verdin IMX8MP), than the default 4 and 2 PWM pins. How do I change this so I can use as a digital 1.8V output, also are there any pins that would be ideal for this on the three types of headers?

Kind regards

btj2 · October 2, 2024, 6:34am

Within our python application.

jeremias.tx · October 2, 2024, 6:08pm

Since this is a new topic could you please create a new post for this, just to keep the questions organized.

In general if you want additional GPIO/PWM or any other extra interfaces you’ll need to modify the device tree to enable these: Device Tree Documentation Overview | Toradex Developer Center

To figure out what pins can serve what functions you should consult the module datasheet. There is a table that shows each pin and every function it can serve as.

Best Regards,
Jeremias