Graphics accelerated X11 application on i.MX8 based SoM

Technical Support
,
1

Hey everyone,

I am currently trying to get a x11 application running (with accelerated graphics, 2d and 3d) on the verdin-imx8m-plus-quad and eventually apalis-imx8-quad-max. It uses GLFW which tries to access OpenGL through GLX. And i’m in the process of gauging what my option are going forward.

  1. Are the verdin and apalis modules (or imx8m and imx8) similar enough, so that getting my application to run on verdin, means it will run on apalis as well? Of course it would be better to test on the system it should run on but since i have physical access to a verdin module and only remote access to a apalis module, i’m testing on verdin for now.

  2. Which combinations of Protocols and APIs are supported to accelerate graphical application?

I did some further research and ran this across with the team internally. Unfortunately this might not be as simple as it seems.
To summarize, OpenGL with GLX is tied to X11. X11 isn’t something that the i.MX8 graphic drivers support. Not to say that it isn’t impossible but you’d require some additional logic or in-between software that uses opengles which should work with i.MX8 graphics. Either way that is to say this won’t work out of the box as is.

Here are the list of DISTRO configurations. Note that fsl-imx-fb is not supported on
i.MX 8 and fsl-imx-x11 is not supported anymore.
• fsl-imx-wayland: Wayland weston graphics.
• fsl-imx-xwayland: Wayland graphics and X11. X11 applications using EGL are not
supported.
• fsl-imx-fb: Frame Buffer graphics - no X11 or Wayland. Frame Buffer is not
supported on i.MX 8 and i.MX 9.

  1. Am i missing any or have i misunderstood any of the options?

Kind Regards
Ben

1 Like
2

Hello Ben, welcome to my world!

I assume you copied me on this because I have crawled this particular trail of tears . . . sans the need for any graphics acceleration.

Avoid Qt at all costs. The current definition of “OpenSource” for Qtc is “Everybody has to buy a license” so cost is the operative word.

Search the interest archive: The Interest Archives

=====
March 2021 archive

Yeah, we noticed when QtPdf license changed:
New QtLabs PDF module (LGPLv3)
Change in open-source licensing of Qt Wayland Compositor, Qt Application Manager and Qt PDF (Tukka’s own post)
[Development] [Interest] Change in open-source licensing of Qt Wayland Compositor, Qt Application Manager and Qt PDF (Not everyone was on board with the license change)
But it’s now under the marketplace license?
https://marketplace.qt.io/collections/most-popular/products/qtpdf ($49/ Marketplace license)

Shenannigans. I declare shenannigans.

Yep, Qt’s trying to become smth like Atlasians or alikes…
They (Digia or whoever) are not interested in the technology, they
just want to milk the cash cow with no long term plan.
Once the cows run out of milk, they’ll chop it down like an old dead
tree (Dirty old town).

IMHO, Qt is not a technology to be put in the hands of hedge funds…

=====

For my last project using this hardware I used NanoGUI inside of docker container running pure Buster with X11. I originally tried building NanoGUI native to use the GLES support Wayland claims to have. GLES under Wayland is a cruel and evil joke. The actual application people ended up using Elements also inside Docker container running pure X11.

Wayland is something that never should have been forced onto the embedded world. The medical device and SAFETY critical world long ago eliminated any X11 vulnerabilities . . . we only have a touch screen, no keyboard or mouse . . . TCP/IP is always disabled until we need to make an outbound connection . . . inbound connections are never allowed.

End of vulnerabilities.

If you need accelerated graphics, my first impulse is to say fuggedaboutit. Traditionally the only way to get that is to get access to the hardware. If you want 3d, forget about Qt for any and all lifetimes. 3d support in Qt is looking up at sad hoping to one day be that good.

I do know that FoxGUI (a.k.a. Fox-Toolkit) has some kind of 3d support. The UI library has a very Windows 95 look and feel but that is how it runs on soooo many platforms. I have never tried it on ARM.

CopperSpice does not currently support Wayland or officially support ARM. (Some people claim to have ported it to Apple ARM but haven’t submitted build scripts or pull requests.) You want to keep an eye on this because they forked Qt 4.8 long ago, ripped out all of the icky nasty QML stuff, and started to really make core library improvements.

This dude did an awful lot of research. You can scroll through the post for some ideas.

If you read this post you will get a few more ideas, LVGL being one of them.

NanoGUI was inspired by imGUI. I didn’t experiment with imGUI because I wanted something easier to get through FDA approval. imGUI is immediate mode and focused on gaming.

I’ve not personally coded anything with it, but wxWidgets claims support for Wayland. Naturally there were some issues along the way. I will probably be slapped for mentioning this, but a competitor of Toradex, Variscite, has some kind of tutorial/podcast on getting started with wxWidgets.

Like Qt, wxWidgets is an 800 pound gorilla of a library. I don’t have a publication date, but here is a free 744 page book on wxWidgets cross platform programming.

Unlike Qt and CopperSpice, wxWidgets has a much nicer license. If you (being you and your company) do use it, “you” should kick a few dollars their way.

Best I got on a Monday morning.

Summary

  1. Avoid Qt at all costs with costs being the operative word.

  2. wxWidgets will probably be your most straight forward option if you don’t need sexy widgets by default.

  3. LVGL, and the other immediate mode GUI libraries are worth a quick look at to see if they support both ARM and Wayland directly.

  4. GLES support under Wayland is a joke. Basically, what I found is that all of the other APIs Wayland claims to support in the ARM world is a microscopic fraction of those APIs. Just enough for “Hello World!” and not much more.

  5. Using pure X11 in a Docker container you can get just about any traditional Linux GUI library to work you want . . . sans acceleration.

1 Like
3

Hey @seasoned_geek
thanks a lot for your answer, not to bad for a monday morning i’d say :). Although i might not have been clear enough about what im looking for. I’m not looking for a GUI framework, i’m trying to get our own graphical application to run, which was developed for X11.

In regards to your summary

  1. i was told Qt wasn’t an option anyway, but now i better know why
  2. i don’t think wxWidgets is what im looking for
  3. same
  4. that’s good to know. where can i find what parts of the API are supported with GLES + Wayland + ARM?
  5. jeah, i suspected, that was the case. The software components that enable hardware acceleration are not getting virtualized in the container, right?. so a wayland container could potentially use hardware acceleration but a x11 container couldn’t if the interfaces outside the container are not supporting it. is that more or less correct?

Kind Regards
Ben

4

You can expose “some” of the hardware devices to a container on the launch command or in the YAML file. What you need access to??? I couldn’t say. That’s for the boys and girls at Toradex, but they are going to need to know exactly what your application needs access to prior to being able to answer.

This move to Wayland well and truly sucks. I really hope someone other than NXP starts making the modules and drops Wayland for X11. I wish it true! I wish it true! I wish it true! (Keep changing mantra profusely.)

I guess my answer wasn’t completely clear.

There is a 98.5% chance you are going to have to re-write from scratch using a different framework to get 3D acceleration under Wayland. There’s a 1% chance you will thieve the code from one of the frameworks you get to actually work on Wayland to add Wayland support to whatever it is you currently use under X11.

That last .5% that Unicorns and rainbows will somehow bestow a magic enchantment to allow your X11 application to “just work” with full acceleration under Wayland is going to be a loooong epic journey.

First you have to get one of “the desktop solutions” to actually work in a Wayland desktop environment.

https://www.reddit.com/r/swaywm/comments/rxrccc/wayland_hardware_acceleration_in_chromium/

https://bbs.archlinux.org/viewtopic.php?id=273468

Assuming the Unicorns and Rainbows smile on thee

Second you have to get 2d and 3d video hardware on your device that actually has the GPU for acceleration.

Third you have to get whatever worked on the desktop to provide an ARM version that works with the ARM driver for your video hardware.

The last link for Linux reviews might point you on a proper path. It talks a bit bout NVidia difficulties and it mentions “some” Xwayland support for X11 access to hardware acceleration.

Honestly, I gave up. I can’t answer the question you have about #4 because I gave up.

I started a fork of NanoGUI to actually solve this problem and gave up.

I got quite a ways there. The humiliating kick to the crotch was NanoGUI had its own GLES because someone wanted MAC/Apple support. I remember getting through that obstacle course, but cannot remember exactly how/where I left the project. We opted for containers and pure X11.

My personal coding project time has been devoted to RedDiamond and I haven’t revisited the WaylandGUI project.

If you (or anyone reading this) wants to fork and create pull requests with any fixes and/or updates from the main NanoGUI source, I whole-heartedly welcome such an endeavor. I’m not currently working on Toradex products so haven’t had great incentive to pursue. (Also no longer have a platform all set up for testing.)

NanoGUI was chosen because it is minimal minimal minimal, then I stripped out even more, choosing to support only Debian based Linux distros and their embedded versions.

Finishing my WaylandGUI journey (which I don’t think was that far from being completed). I don’t exactly remember what this comment in the ReadMe meant

GLES 3 is now used for the GLFW layer, soon that it will be EGL only.

I think I was whittling the library down even further, trying to remove all GLES so it was EGL only because there was some difference with the previous Toradex GLES stuff. Haven’t touched since new Toradex version.

Sorry I didn’t mention WaylandGUI before. Like I said, Monday morning before caffeine kicked in.

5

ah, didn’t realize you meant using parts of existing GUI framework to get hardware acceleration within wayland or possibly without it. good idea, though.

i had a short call with a toradex developer this morning and he confirmed more or less everything i suspected and you confirmed as well. as far as i can tell, this is how it looks:

  • native X11
    • only hope is that open source graphics driver (etnaviv) is eventually implementing X11 support
  • XWayland
    • hardware acceleration possible, but not out of the box. also can have performance issues or incomplete functionality depending on the application (touchscreen events are buggy, input devices may not work fully)
  • Direct Rendering
    • possible, but would mean to access EGL directly, which in my case, GLFW is not able to do.
  • X11 Docker
    • should run almost any X11 application, but has no hardware acceleration
  • native Wayland
    • here @seasoned_geek’s answer applies: rewrite, using glue-code from existing gui-frameworks or praying/reconfiguring existing interfaces/APIs/backends of the application

Thanks a ton, warning me of the pitfalls that lie ahead. I appreciate it :slight_smile:

6

No problem man.

I would recommend, if you have a few days for research, you take a stab at getting waylandGUI to work. Layer your own widget/2d/3d stuff and try to run native.

Here’s an option nobody talks about. Don’t use i.MX8.

The older Toradex/NXP/freescale core software was good ole reliable X11. No Wayland. If you can live with back-in-time and i.MX6 or other non-MX8, you can be happy.

1 Like