GPU performance in Chromium container on Torizon OS iMX8M-Plus

Hi @jeremias.tx and @leo.tx,
sorry for opening a new post but there is a mess on this topic which I would try to summarize here.
These topic would merge these four discussions (which could be closed if possibile):

  1. torizoncore-6-1-cannot-render-anymore-3d-content-on-verdin-imx8m-mini
  2. chromium-container-cannot-render-3d-content-on-verdin-imx8m-mini
  3. gpu-acceleration-in-chromium-container-on-torizon-os-imx8m-plus
  4. gpu-monitoring-on-torizoncore

Now I try to summarize the point here.
Several months ago, we start testing GPU performance of a chromium container initially on the imx8-mini and then on the imx8-plus.
To evaluate the GPU, we use the webglsamples - fishtank running in the chromium container.

Initially we test on the imx8-plus and we get poor performance (14 FPS with 1 fish):

We thought that these were the ‘expected’ performance until we found this foundries.io blog in which is quite clear that they were able to get better GPU performance (22 FPS with 10 fishes and sharks) even on the imx8-mini (which has a lower performance GPU). So we start investigating with you (toradex) on the main cause of this poor performance on the imx8-plus, we get some gpu-top screenshot to log the situation.

In the meantime, we open a ticket to NXP to point the same issue and they focus two things:

  1. we are using the 6.4.3.[something] vivante driver version while there is a new version available (should be 6.11.[something]) which could improve performance and fix bugs.
  2. since gpu-top reports 100% Rasterizer load, they says the word ‘driver bug’ (we were testing using QT to take chromium out of the loop)


But in the same period, also toradex starts talking about some ‘GPU related problems’ in chromium so we pause this task waiting for feedback. Some days ago, @jeremias.tx (here) says ‘they did some fixes and have an early version of the improved Chromium container. Please try […] The developer was able to get much better performance on the aquarium demo then you reported prior

So we try again with the linked chromium container on the same fishtank demo and we get absolutely no performance improvements (again, 14 FPS with 1 fish)
image

And this should complete the whole recap.

We are testing performance from the ‘user point of view’, so we need to get hardware accelerated performance in chromium to improve our web-based application; for better animations and response speed.

Let me know if I can do some other test, actually this is a blocking task for our project.

Thanks

Greetings @vix,

I spoke heavily with our developers and we get the same/similar numbers you are seeing. ~14FPS with 1 fish. At the moment this is what we have. We can confirm that the GPU is being used however. I can’t really comment on the blog from foundries since much of their setup there is abstracted. It’s hard to say what resolution or display interface they’re even showing in their example. Also they just have static screenshot showing 22fps, it’s hard to say if it’s even stable at 22fps. I can’t really do a one to one comparison with this.

Now moving onto your requirements. You said this is a blocking issue for you. Have you already tested the performance here with your own application stack? Is the performance not satisfactory for your needs?

Best Regards,
Jeremias

Yes, I tested my application and the performance are not satisfactory.
As an example, they are better on an old Intel Celeron J1900 running Windows and so I expect the iMX8M-Plus hardware is powerful enough to run the application.

One thing that I can say is that before and after the fixes on Chromium container, the FPS has not changed (~14FPS some months agfo, and ~14FPS today).
This is more or less what I wrote to Toradex engineers some months ago: probably Chromium has space for improvement, but the key point seems not be there.
I tested with Qt5 examples container, and 3D acceleration is not good. And this container is not Chromium.
All what I saw and I see seems related to some issues in vivante driver.

Is there a possibility to upgrade the vivante driver from 6.4.3 to 6.11.x as suggested by NXP in the answer here?

In the meanwhile I’m going on with investigation here, because I think there is a missing point somewhere IMHO.

/---- EDIT ----/
To see if the issue is either in the browser or in the driver, it could be interesting comparing FPS of other browsers (Cog, firefox, …)
I don’t know if someone did this kind of test.

One more thing: the FPS of fishtank sample stays at ~14 FPS even if I start Chromium with the parameter --disable-gpu and so this seems to indicate that GPU is not used by Chromium.

Do you agree to my analysis?

I can bring this up for more discussion internally but I can’t promise anything on any timeline. What we have currently is what we have.

Is there a possibility to upgrade the vivante driver from 6.4.3 to 6.11.x as suggested by NXP in the answer here ?

This would not be a quick and easy change on our side. At best this would be something we do for the next major version (i.e. Torizon OS 7.X).

One more thing: the FPS of fishtank sample stays at ~14 FPS even if I start Chromium with the parameter --disable-gpu and so this seems to indicate that GPU is not used by Chromium.

This is actually an issue with our script the --disable-gpu wasn’t actually passing anything. If you look at our startup-script for chromium: torizon-containers/debian-docker-images/chromium/start-chromium.sh at bookworm · toradex/torizon-containers · GitHub

You can see the --disable-gpu argument sets the ENABLE_GPU variable, which then does nothing. This was found recently by a developer and will actually be removed since it’s not doing anything.

Best Regards,
Jeremias

Hi @jeremias.tx,
thank you for your interest and your help.
I really appreciate it.

I can add one more thing:
I’ve just tried with cog-imx8 container (instead of Chromium) and I have 14 FPS (so, the same as Chromium).
Moreover, cog is blinking and is completely unstable and unusable (this is a different problem, and I’m not so interested in cog).

Since both Chromium and Cog show the same performances (14 FPS on FishTank), I think this can be only for one of the following reasons:

  • either this ios the hardware limit
  • or this is the GPU driver (vivante) limit

But here you can read the answer from NXP tech support when I asked for a hardware expected benchmarking:

There is no official benchmark to show you the fps numbers we have in the mx8mplus, but for the fishtank you can obtain at least 60fps since is pure opengl.

I asked NXP for more details on this, and I keep you informed about their answer.
If they’re right, there is a big issue in the GPU driver.

I see that there is a nre rc container torizon/gputop-imx8.
How can I run it to perform the GPU?

How can I run it to perform the GPU?

Leon explained this over in his comment on the other thread here: TorizonCore 6.1 cannot render anymore 3D content on Verdin iMX8M-Mini - #58 by leon.tx

You need to install gputop in the same container as the process you want to measure otherwise it won’t “see” the process and give misleading results.

Best Regards,
Jeremias

Hi @jeremias.tx
hi @leon.tx

I’ve just seen this commit on toradex/github.
It’s related to GPU acceleration in Chromium for iMX8 platform.
Great news!
I rebuilt the chromium container for iMX8, and so

ARG IMAGE_ARCH=linux/arm64

ARG BASE_NAME=wayland-base-vivante

ARG IMAGE_TAG=3

Now I see some promising messages in the log of the container:

docker-compose-kiosk-1   | EGL: Warning: No default display support on wayland
docker-compose-kiosk-1   | [58:58:0304/074817.234078:ERROR:gl_surface_egl.cc(1448)] EGL display query failed with error EGL_SUCCESS
docker-compose-kiosk-1   | [58:58:0304/074817.286198:ERROR:gl_surface_egl.cc(1473)] eglInitialize Default failed with error EGL_BAD_DISPLAY
docker-compose-kiosk-1   | [58:58:0304/074817.288136:ERROR:gl_ozone_egl.cc(21)] GLSurfaceEGL::InitializeOneOff failed.
docker-compose-kiosk-1   | [1:39:0304/074817.288478:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
docker-compose-kiosk-1   | [1:39:0304/074817.288586:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
docker-compose-kiosk-1   | [58:58:0304/074817.317335:ERROR:viz_main_impl.cc(185)] Exiting GPU process due to errors during initialization
docker-compose-kiosk-1   | [1:92:0304/074817.448732:ERROR:object_proxy.cc(623)] Failed to call method: org.freedesktop.DBus.Properties.Get: object_path= /org/freedesktop/UPower: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.UPower was not provided by any .service files
docker-compose-kiosk-1   | [1:92:0304/074817.450856:ERROR:object_proxy.cc(623)] Failed to call method: org.freedesktop.UPower.GetDisplayDevice: object_path= /org/freedesktop/UPower: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.UPower was not provided by any .service files
docker-compose-kiosk-1   | [1:92:0304/074817.453575:ERROR:object_proxy.cc(623)] Failed to call method: org.freedesktop.UPower.EnumerateDevices: object_path= /org/freedesktop/UPower: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.UPower was not provided by any .service files
docker-compose-kiosk-1   | EGL: Warning: No default display support on wayland
docker-compose-kiosk-1   | [91:91:0304/074817.473985:ERROR:gl_surface_egl.cc(1448)] EGL display query failed with error EGL_SUCCESS
docker-compose-kiosk-1   | [91:91:0304/074817.477597:ERROR:gl_surface_egl.cc(1473)] eglInitialize Default failed with error EGL_BAD_DISPLAY
docker-compose-kiosk-1   | [91:91:0304/074817.478164:ERROR:gl_ozone_egl.cc(21)] GLSurfaceEGL::InitializeOneOff failed.
docker-compose-kiosk-1   | [91:91:0304/074817.505853:ERROR:viz_main_impl.cc(185)] Exiting GPU process due to errors during initialization
docker-compose-kiosk-1   | [126:126:0304/074817.572667:ERROR:gpu_init.cc(481)] Passthrough is not supported, GL is disabled, ANGLE is 
docker-compose-kiosk-1 exited with code 0

I say promising because now, for the first time, I see that chromium tries to use the GPU.
Chromium is not shown on the display, but I think this is for the errors.
Now I think that the focus can be moved to the investigation of these errors.
I found some topics that seems similar to what happens:

Do you agreee to what I wrote?
Thanks

Our chromium container images on Dockerhub should already have the latest commits: Docker

You shouldn’t need to build a chromium container manually like this. When I run weston & chromium as per our documentation, it starts fine for me without any of these errors you are seeing here. The fishtank example still has the same “performance” of about 14fps at 1 fish. So nothing has changed in this regard. That said, as far as we can tell the GPU is being used for this.

Best Regards,
Jeremias

Hi @jeremias.tx
I don’t agree with your answer, but we need the help of one of the developers. Let me explain:

  • docker container torizon/chromium can be used for generic linux/arm64 architecture, so it’s not specific to iMX8
  • for this reason I imagine that, even with the latest commit, it’s built over wayland-base (becasue this is ok for a generic linux/arm64
  • from the logs of this container I don’t see any message related to GPU, and the performace is 14 FPS
  • I suspect that the “under-test” container torizon/chromium-imx8 is built over wayland-base-vivante. Can @leon.tx confirm this?
  • If I run it (or the container I built - the behavior is the same) with --in-process-gpu I can see the logs I attached in the previous message, where you can see a lot of issues related to GL. And chromium crashes.
  • If I run it without --in-process-gpu, I can see the following log
    chromium-log.txt (4.6 KB)
    note the message
    [58:58:0305/092046.430776:ERROR:viz_main_impl.cc(185)] Exiting GPU process due to errors during initialization
    and this is the screen of chromium

    this is clear that GL (GPU) is not enabled

I suggest to look into this issue
https://github.com/OSSystems/meta-browser/issues/711

NXP Tech support confirmed in a private message that if the GPU Vivante GC7000 is used properly, the performance of fishtank are 60 FPS.
The repo OSSystems/meta-browser has some issues related to problems with HW acceleration on chromium.

@leon.tx can you give me a feedback from the developer side, please?

Well all my replies in this thread were made after consulting with the developers (including Leon). These are not just my opinions/observations but that of the developer team as well.

Though if you insist on speaking with a developer I can discuss this internally and see if we can arrange something.

Best Regards,
Jeremias

Yes, please.
I think it will be helpful.

Is there any news on this topic?

Hi @erik.weber,

There’s not really much to add here. After some discussion internally and with vix here it was determined there was not any serious performance issues with our Chromium container and that the issues were elsewhere.

If you’re having an issue could you open another thread with a description of what you’re experiencing please.

Best Regards,
Jeremias

I try to summarize the conclusion of the analysis.

  • the performance of the GPU is hard to measure, because the GPU itself has some different features (rasterization, shader, texture, …) and, moreover, iMX8M-Plus has two different GPU cores (one 3D GPU and one 2D GPU)
  • different samples from https://webglsamples.org show different FPS on the same HW and display, because it depends on the GPU features they use
  • docker containers by Toradex (weston vivante + Chromium) are not responsible for these performance, and so we can say they’re ok. Cog container doesn’t work at all and I opened a ticket here

The only thing that could bring some performace improvement (hard to be 100% sure) would be upgrading the GPU drivers inside Torizon OS.
But unfortunately this is not possible at the moment, because of a decision of NXP to release newer drivers on Yocto nanbield and not kirkstone (which is the one used by Torizon OS 6). The last bug fixing from NXP on the GPU driver was 3 weeks ago.
Toradex has been working actively on this topic (GPU drivers), but it would be a long term process.

At the moment you must live with this performance on iMX8M-Plus + Torizon OS, whatever they are.

@vix , completely out of curiosity and as a total stranger on this issue: What is the experience of this through Yocto only? I had some issues with HW acceleration (VPU, NPU) in Torizon while it was working fine with the standard BSP layers.

Again, just curious and mostly for my own reference. Thanks.

Alvaro.

@alvaro.tx I can’t give any feedback on Yocto side.
I use Torizon OS only, because Yocto requires some knowledge that I don’t have at the moment.

1 Like

Ok, thank you for all the information.