Hi @jeremias.tx and @leo.tx,
sorry for opening a new post but there is a mess on this topic which I would try to summarize here.
These topic would merge these four discussions (which could be closed if possibile):
Now I try to summarize the point here.
Several months ago, we start testing GPU performance of a chromium container initially on the imx8-mini and then on the imx8-plus.
To evaluate the GPU, we use the webglsamples - fishtank running in the chromium container.
Initially we test on the imx8-plus and we get poor performance (14 FPS with 1 fish):
We thought that these were the âexpectedâ performance until we found this foundries.io blog in which is quite clear that they were able to get better GPU performance (22 FPS with 10 fishes and sharks) even on the imx8-mini (which has a lower performance GPU). So we start investigating with you (toradex) on the main cause of this poor performance on the imx8-plus, we get some gpu-top screenshot to log the situation.
In the meantime, we open a ticket to NXP to point the same issue and they focus two things:
we are using the 6.4.3.[something] vivante driver version while there is a new version available (should be 6.11.[something]) which could improve performance and fix bugs.
since gpu-top reports 100% Rasterizer load, they says the word âdriver bugâ (we were testing using QT to take chromium out of the loop)
But in the same period, also toradex starts talking about some âGPU related problemsâ in chromium so we pause this task waiting for feedback. Some days ago, @jeremias.tx (here) says âthey did some fixes and have an early version of the improved Chromium container. Please try [âŚ] The developer was able to get much better performance on the aquarium demo then you reported priorâ
So we try again with the linked chromium container on the same fishtank demo and we get absolutely no performance improvements (again, 14 FPS with 1 fish)
And this should complete the whole recap.
We are testing performance from the âuser point of viewâ, so we need to get hardware accelerated performance in chromium to improve our web-based application; for better animations and response speed.
Let me know if I can do some other test, actually this is a blocking task for our project.
I spoke heavily with our developers and we get the same/similar numbers you are seeing. ~14FPS with 1 fish. At the moment this is what we have. We can confirm that the GPU is being used however. I canât really comment on the blog from foundries since much of their setup there is abstracted. Itâs hard to say what resolution or display interface theyâre even showing in their example. Also they just have static screenshot showing 22fps, itâs hard to say if itâs even stable at 22fps. I canât really do a one to one comparison with this.
Now moving onto your requirements. You said this is a blocking issue for you. Have you already tested the performance here with your own application stack? Is the performance not satisfactory for your needs?
Yes, I tested my application and the performance are not satisfactory.
As an example, they are better on an old Intel Celeron J1900 running Windows and so I expect the iMX8M-Plus hardware is powerful enough to run the application.
One thing that I can say is that before and after the fixes on Chromium container, the FPS has not changed (~14FPS some months agfo, and ~14FPS today).
This is more or less what I wrote to Toradex engineers some months ago: probably Chromium has space for improvement, but the key point seems not be there.
I tested with Qt5 examples container, and 3D acceleration is not good. And this container is not Chromium.
All what I saw and I see seems related to some issues in vivante driver.
Is there a possibility to upgrade the vivante driver from 6.4.3 to 6.11.x as suggested by NXP in the answer here?
In the meanwhile Iâm going on with investigation here, because I think there is a missing point somewhere IMHO.
/---- EDIT ----/
To see if the issue is either in the browser or in the driver, it could be interesting comparing FPS of other browsers (Cog, firefox, âŚ)
I donât know if someone did this kind of test.
One more thing: the FPS of fishtank sample stays at ~14 FPS even if I start Chromium with the parameter --disable-gpu and so this seems to indicate that GPU is not used by Chromium.
I can bring this up for more discussion internally but I canât promise anything on any timeline. What we have currently is what we have.
Is there a possibility to upgrade the vivante driver from 6.4.3 to 6.11.x as suggested by NXP in the answer here ?
This would not be a quick and easy change on our side. At best this would be something we do for the next major version (i.e. Torizon OS 7.X).
One more thing: the FPS of fishtank sample stays at ~14 FPS even if I start Chromium with the parameter --disable-gpu and so this seems to indicate that GPU is not used by Chromium.
You can see the --disable-gpu argument sets the ENABLE_GPU variable, which then does nothing. This was found recently by a developer and will actually be removed since itâs not doing anything.
Hi @jeremias.tx,
thank you for your interest and your help.
I really appreciate it.
I can add one more thing:
Iâve just tried with cog-imx8 container (instead of Chromium) and I have 14 FPS (so, the same as Chromium).
Moreover, cog is blinking and is completely unstable and unusable (this is a different problem, and Iâm not so interested in cog).
Since both Chromium and Cog show the same performances (14 FPS on FishTank), I think this can be only for one of the following reasons:
either this ios the hardware limit
or this is the GPU driver (vivante) limit
But here you can read the answer from NXP tech support when I asked for a hardware expected benchmarking:
There is no official benchmark to show you the fps numbers we have in the mx8mplus, but for the fishtank you can obtain at least 60fps since is pure opengl.
I asked NXP for more details on this, and I keep you informed about their answer.
If theyâre right, there is a big issue in the GPU driver.
I see that there is a nre rc container torizon/gputop-imx8.
How can I run it to perform the GPU?
You need to install gputop in the same container as the process you want to measure otherwise it wonât âseeâ the process and give misleading results.
Iâve just seen this commit on toradex/github.
Itâs related to GPU acceleration in Chromium for iMX8 platform.
Great news!
I rebuilt the chromium container for iMX8, and so
Now I see some promising messages in the log of the container:
docker-compose-kiosk-1 | EGL: Warning: No default display support on wayland
docker-compose-kiosk-1 | [58:58:0304/074817.234078:ERROR:gl_surface_egl.cc(1448)] EGL display query failed with error EGL_SUCCESS
docker-compose-kiosk-1 | [58:58:0304/074817.286198:ERROR:gl_surface_egl.cc(1473)] eglInitialize Default failed with error EGL_BAD_DISPLAY
docker-compose-kiosk-1 | [58:58:0304/074817.288136:ERROR:gl_ozone_egl.cc(21)] GLSurfaceEGL::InitializeOneOff failed.
docker-compose-kiosk-1 | [1:39:0304/074817.288478:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
docker-compose-kiosk-1 | [1:39:0304/074817.288586:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
docker-compose-kiosk-1 | [58:58:0304/074817.317335:ERROR:viz_main_impl.cc(185)] Exiting GPU process due to errors during initialization
docker-compose-kiosk-1 | [1:92:0304/074817.448732:ERROR:object_proxy.cc(623)] Failed to call method: org.freedesktop.DBus.Properties.Get: object_path= /org/freedesktop/UPower: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.UPower was not provided by any .service files
docker-compose-kiosk-1 | [1:92:0304/074817.450856:ERROR:object_proxy.cc(623)] Failed to call method: org.freedesktop.UPower.GetDisplayDevice: object_path= /org/freedesktop/UPower: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.UPower was not provided by any .service files
docker-compose-kiosk-1 | [1:92:0304/074817.453575:ERROR:object_proxy.cc(623)] Failed to call method: org.freedesktop.UPower.EnumerateDevices: object_path= /org/freedesktop/UPower: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.UPower was not provided by any .service files
docker-compose-kiosk-1 | EGL: Warning: No default display support on wayland
docker-compose-kiosk-1 | [91:91:0304/074817.473985:ERROR:gl_surface_egl.cc(1448)] EGL display query failed with error EGL_SUCCESS
docker-compose-kiosk-1 | [91:91:0304/074817.477597:ERROR:gl_surface_egl.cc(1473)] eglInitialize Default failed with error EGL_BAD_DISPLAY
docker-compose-kiosk-1 | [91:91:0304/074817.478164:ERROR:gl_ozone_egl.cc(21)] GLSurfaceEGL::InitializeOneOff failed.
docker-compose-kiosk-1 | [91:91:0304/074817.505853:ERROR:viz_main_impl.cc(185)] Exiting GPU process due to errors during initialization
docker-compose-kiosk-1 | [126:126:0304/074817.572667:ERROR:gpu_init.cc(481)] Passthrough is not supported, GL is disabled, ANGLE is
docker-compose-kiosk-1 exited with code 0
I say promising because now, for the first time, I see that chromium tries to use the GPU.
Chromium is not shown on the display, but I think this is for the errors.
Now I think that the focus can be moved to the investigation of these errors.
I found some topics that seems similar to what happens:
Our chromium container images on Dockerhub should already have the latest commits: Docker
You shouldnât need to build a chromium container manually like this. When I run weston & chromium as per our documentation, it starts fine for me without any of these errors you are seeing here. The fishtank example still has the same âperformanceâ of about 14fps at 1 fish. So nothing has changed in this regard. That said, as far as we can tell the GPU is being used for this.
Hi @jeremias.tx
I donât agree with your answer, but we need the help of one of the developers. Let me explain:
docker container torizon/chromium can be used for generic linux/arm64 architecture, so itâs not specific to iMX8
for this reason I imagine that, even with the latest commit, itâs built over wayland-base (becasue this is ok for a generic linux/arm64
from the logs of this container I donât see any message related to GPU, and the performace is 14 FPS
I suspect that the âunder-testâ container torizon/chromium-imx8 is built over wayland-base-vivante. Can @leon.tx confirm this?
If I run it (or the container I built - the behavior is the same) with--in-process-gpu I can see the logs I attached in the previous message, where you can see a lot of issues related to GL. And chromium crashes.
If I run it without--in-process-gpu, I can see the following log chromium-log.txt (4.6 KB)
note the message [58:58:0305/092046.430776:ERROR:viz_main_impl.cc(185)] Exiting GPU process due to errors during initialization
and this is the screen of chromium
NXP Tech support confirmed in a private message that if the GPU Vivante GC7000 is used properly, the performance of fishtank are 60 FPS.
The repo OSSystems/meta-browser has some issues related to problems with HW acceleration on chromium.
@leon.tx can you give me a feedback from the developer side, please?
Well all my replies in this thread were made after consulting with the developers (including Leon). These are not just my opinions/observations but that of the developer team as well.
Though if you insist on speaking with a developer I can discuss this internally and see if we can arrange something.
Thereâs not really much to add here. After some discussion internally and with vix here it was determined there was not any serious performance issues with our Chromium container and that the issues were elsewhere.
If youâre having an issue could you open another thread with a description of what youâre experiencing please.
I try to summarize the conclusion of the analysis.
the performance of the GPU is hard to measure, because the GPU itself has some different features (rasterization, shader, texture, âŚ) and, moreover, iMX8M-Plus has two different GPU cores (one 3D GPU and one 2D GPU)
different samples from https://webglsamples.org show different FPS on the same HW and display, because it depends on the GPU features they use
docker containers by Toradex (weston vivante + Chromium) are not responsible for these performance, and so we can say theyâre ok. Cog container doesnât work at all and I opened a ticket here
The only thing that could bring some performace improvement (hard to be 100% sure) would be upgrading the GPU drivers inside Torizon OS.
But unfortunately this is not possible at the moment, because of a decision of NXP to release newer drivers on Yocto nanbield and not kirkstone (which is the one used by Torizon OS 6). The last bug fixing from NXP on the GPU driver was 3 weeks ago.
Toradex has been working actively on this topic (GPU drivers), but it would be a long term process.
At the moment you must live with this performance on iMX8M-Plus + Torizon OS, whatever they are.
@vix , completely out of curiosity and as a total stranger on this issue: What is the experience of this through Yocto only? I had some issues with HW acceleration (VPU, NPU) in Torizon while it was working fine with the standard BSP layers.
Again, just curious and mostly for my own reference. Thanks.
Thanks for all the information you have gathered on this issue.
Iâm trying to develop an wxwidget+opencv based application which can utilize the GPU on iMX8MP but facing similar issues. First of all the framework is not able to access vivante drivers in the wayland vivante container itself and application is switching to cpu based rendering of camera data and it is painful to see the CPU struggling.
But running the webGL aquarium examples, Iâm getting only 22 FPS with 1 fish and 12-13 FPS with 100.
I wonder the GPU is even being utilized or not. Is that the best iMX8MP can do? If that is so I doubt Iâll be able to handle realtime video processing with openCV.
What can I do to overcome this issue.
How did you solved it?
Hi @Sudharshan
basically the issue has not been solved yet.
I changed a little bit my application, so that it doesnât need all the power of GPU, so this is ok for me.
As I wrote, I opened a ticket to NXP and they found a but in their Vivante driver.
The fixed release is available in Torizon OS 7 only (not possible in Torizon OS 6 since itâs kirkstone based).
But Chromium container (tag 4.x) is not stable yet (at the time of writing).
I think that you should wait for a stable Chromium container on Torizon OS 7 (Scarthgap) and see what happens.
I read a lot of people saying that cog is better, but for me cog flickers with aquarium example and it cannot be used.
As my main target is to achieve GPU acceleration in my wxWidgets based GUI application using opencv and openGL for GUI ,
will I be blocked by this issue?
I wonder how others are using the GPU on iMX8MP.
