Hi community,
I wanted to share a couple of findings when using OpenGL ES on the colibri with weston.
The configuration is as follows:
I’m using ImGui for the UI, SDL to generate the windows/inputs/etc. running on the Colibri on Weston with Torizon 5.7.0.
I’ve done a few tries and I finally managed to get some nice performances out of it even on 1080p.
The following statistics are done on a 1024x600 screen:
SDL with wayland backend, VNC enabled (no connections) → ~10-14 fps, 1 core almost all used by weston, lot of lag on pointer
SDL with wayland backend, patched and recompiled as explained below, VNC enabled (no connections) → ~30 fps, VNC broken/distorted, 60% of CPU used by weston, no pointer lag
SDL with wayland backend, patched and recompiled as explained below,VNC disabled → ~30-70 fps, mostly around the high 60s fps, 10% of CPU used by weston, no pointer lag
About SDL: I tried other libraries like GLFW but the performances were more or less the same.
The issue seems to be on the OpenGL function eglSwapBuffers on the Vivante GPU: it is very slow and it gets worse fast when the resolution of the screen increases.
The technical manual of the GPU has a list of the OpenGL support extensions and one of them solved the problem and increased drastically the framerate.
Using eglSwapBuffersWithDamageEXT instead of eglSwapBuffers, and specifying the damage rect, tripled the framerate (I directly modified the SDL sources to do this).
I’m attaching a .patch file of the SDL sources with the change. 0001-Use-tiles-and-EGL_EXT_swap_buffers_with_damage-to.patch (3.1 KB)
The damage rect doesn’t seems to matter that much, I’ve divided a 1080p resolution into 9 rectangles that are “damaged” one by one at each frame.
There is no noticeable flickering or distortion on the image on the real screen, but you can see the partially updated rectangles while using VNC, making VNC totally unusable.
Has someone else noticed any high CPU usage while using weston with VNC?
Are there any plans to optimize it?
Based on your findings, in summary you’re having issues with VNC affecting the performance of your UI and overall graphics. This seems to happen even when no VNC connection is present just having VNC enabled in Weston already presents a performance drop.
Did I more or less understand you correctly?
Has someone else noticed any high CPU usage while using weston with VNC?
Your findings here are consistent with what we have observed ourselves internally with this VNC implementation. The severity of the slowdown seems to depend on the exact graphics/UI use-case. But in almost every case there is a degree of slow-down observed when VNC is enabled.
Just to understand your use-case. In your end-product will you have VNC enabled always for your system? Is this to have a way to debug/monitor the UI remotely when it’s deployed?
Are there any plans to optimize it?
At the moment there are no plans to optimize this behavior. But given your report and previous reports I’ll see if I can make the case to take a look at this issue.
Hi @jeremias.tx ,
thank you for the fast response.
Yes I confirm that.
We usually keep it enabled in our other products to provide remote support when needed. Since we are not using any graphical demanding application and lag is not noticeable there (touchscreen so no cursor) it works just fine for that.
It would be great to be able to enable/disable VNC remotely when needed.
The only workaround I can think of at the moment is to keep two different docker-compose releases on Torizon OTA platform, one with the ENABLE_VNC: 1 and one with the ENABLE_VNC: 0 and “update” to the one with VNC only during support.
Anyway, I thank you again for your answer, if there are updates I’ll keep checking this thread and your issue tracker.
Thank you for sharing more details about your use-case. It seems pretty normal and sensible to me. Just one more question, is this something you would say is fairly urgent for your product/system or can you work around it for now?
It would be great to be able to enable/disable VNC remotely when needed.
The only workaround I can think of at the moment is to keep two different docker-compose releases on Torizon OTA platform, one with the ENABLE_VNC: 1 and one with the ENABLE_VNC: 0 and “update” to the one with VNC only during support.
I believe your assessment here is correct, you need to restart the container to change the VNC configuration.
Hi @jeremias.tx,
we can work around it for now.
We will probably start rolling it out to customers at the beginning of the next year and it will take some more time to “ramp up” the numbers (hopefully).
Thank you for letting me know, it helps with our own prioritization and such. Well as you said for the time being this isn’t an immediate issue, but I’ll try and inform you of any improvements or progress we make here.
Unfortunately this issue hasn’t gotten any attention due to lack of priority. That said, could you describe your use case here. Is it just simply you want to remotely-access the UI of your systems after they’re deployed in the field? Or is it something more than this?
Hey @jeremias.tx. I am facing a similar issue. When the VNC is not enabled in Weston, my UI application works smoothly. However, when I enable the VNC backend in Weston, my UI application works slowly with poor performance. The entire application slows down even when no VNC connections have been established. The VNC is used for debugging in my application, but this is still an important feature. Furthermore, when checking the CPU consumption, I see Weston is using lots of CPU (almost one entire CPU core unit). When I try to see the syscalls via “strace” command, I see it spends a lot of CPU time with “ioctl” syscalls, which might indicate it is an I/O issue (maybe a driver issue?). Nevertheless, when I use the “perf” profiling tool, I see lots of CPU consumption on the function gcoVX_MultiGPUSync from the lib libGAL.so (indicating that it might be a problem on the package libgal-imx package). Have you seen a similar issue? Do you know what makes this function be called so frequently? Do you see a way of avoiding my application to call this gcoVX_MultiGPUSync function? I am running my application throw chromium, so maybe I could use a chromium flag to fix it as an interim solution. I have tried the patch that @mag used, but I had no success. The version of Weston that I am using is 9.0.0.
Regarding the VNC issue, is this something that you’re using only during development for debugging or do you plan to use it in production as well? Furthermore, what version of TorizonCore are you on in your situation, and what module is this?
Furthermore, when checking the CPU consumption, I see Weston is using lots of CPU (almost one entire CPU core unit).
Is the high CPU-usage related to VNC or is this a different issue now? If it’s a different issue could you open a new thread with the details related to this please.
On a final note, could you try RDP instead of VNC for remote GUI access. Let me know if it’s better for your situation or not.
Hey @jeremias.tx, the VNC is used in production as well. I am using a personalized distro build with yocto which is not TorizonCore, but yet based on toradex packages. The high CPU-usage is related to VNC (when I disable VNC on weston, the CPU usage is way lower). Regarding the RDP, I have checked using RDP instead, but still the same problem. I will create another ticket then. Thanks.
Oh so you’re building and integrating Weston from Yocto then? How exactly do you have VNC support then? I don’t believe Weston 9.0.0 had native VNC support back then. In TorizonCore we’re using an out of tree patch to add VNC support.
Now there are efforts upstream to improve upon this, though they’re still in progress. We’re monitoring the upstream efforts to see if we can utilize anything in our own implementations, but there’s nothing concrete yet.
All that said, I can’t really give you a timeline at the moment of when this will be improved upon.
