iMX8QM device-tree broken after migrating from TorizonCore 5 to TorizonOS 6

Hello everyone

We’ve been successfully using Apalis iMX8QM V1.1 on Apalis Evaluation Board (actually on an Ixora v1.2 on top of a custom board) together with TorizonCore 5.7.2 and earlier versions.

Recently, we wanted to upgrade to TorizonOS 6.5 quarterly for the wealth of great new features added. But in doing so found that our GPIO lines aren’t working anymore, because the pins seem not to be configured properly. Please find some debug information below.

We have spent quite some time trying different combinations of overlays but just can’t seem to bring the GPIOs back as they were before. One thing that brought them back was using the old kernel’s device tree, but then the HDMI didn’t work and the weston container would crash immediately.

Any help or hint in the right direction would be greatly appreciated. We’re happy to provide more information if necessary.

Thank you very much,
Kim

======

Here’s the tdx-info for SW/HW and device-tree as well as gpioinfo output from a device on TorizonCore 5.7.2

Software summary
------------------------------------------------------------
Bootloader:               U-Boot
Kernel version:           5.4.193-5.7.2+git.b60d3160fd04 #1-TorizonCore SMP PREEMPT Fri Dec 23 15:47:24 UTC 2022
Kernel command line:      pci=nomsi root=LABEL=otaroot rootfstype=ext4 quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3 ostree=/ostree/boot.0/torizon/020545d935d9f72873c9b5be7c5bb59b1c7cc1886ca7979b5e9e3950434b330e/0 kernel.sched_rt_runtime_us=-1
Distro name:              NAME="TorizonCore"
Distro version:           VERSION_ID=5.7.2-build.20
Distro variant:           VARIANT="Docker"
Hostname:                 *******
------------------------------------------------------------

Hardware info
------------------------------------------------------------
HW model:                 Toradex Apalis iMX8QM V1.1 on Apalis Evaluation Board
Toradex version:          0037 V1.1C
Serial number:            07240280
Processor arch:           aarch64
------------------------------------------------------------

Device tree
------------------------------------------------------------
Device tree enabled:      imx8qm-apalis-v1.1-eval.dtb
Compatible string:        toradex,apalis-imx8-v1.1-evaltoradex,apalis-imx8-evaltoradex,apalis-imx8fsl,imx8qm
Device trees available:
                          imx8qm-apalis-eval.dtb
                          imx8qm-apalis-ixora-v1.1.dtb
                          imx8qm-apalis-v1.1-eval.dtb
                          imx8qm-apalis-v1.1-ixora-v1.1.dtb
                          imx8qm-apalis-v1.1-ixora-v1.2.dtb
                          imx8qp-apalis-v1.1-eval.dtb
                          imx8qp-apalis-v1.1-ixora-v1.1.dtb
                          imx8qp-apalis-v1.1-ixora-v1.2.dtb
------------------------------------------------------------

Device tree overlays
------------------------------------------------------------
Overlays enabled:         fdt_overlays=apalis-imx8_hdmi_overlay.dtbo apalis-imx8_lvds_overlay.dtbo apalis-imx8_atmel-mxt_overlay.dtbo display-lt170410_overlay.dtbo custom-kargs_overlay.dtbo
Overlays available:
                          apalis-imx8_ar0521_overlay.dtbo
                          apalis-imx8_atmel-mxt_overlay.dtbo
                          apalis-imx8_hdmi_overlay.dtbo
                          apalis-imx8_lvds_overlay.dtbo
                          apalis-imx8_mezzanine-can_overlay.dtbo
                          apalis-imx8_mezzanine_lvds_overlay.dtbo
                          apalis-imx8_mezzanine_ov5640_overlay.dtbo
                          apalis-imx8_ov5640_overlay.dtbo
                          apalis-imx8_resistive-touch_overlay.dtbo
                          custom-kargs_overlay.dtbo
                          display-dpi-lt170410_overlay.dtbo
                          display-edt5.7_overlay.dtbo
                          display-edt7_overlay.dtbo
                          display-fullhd_overlay.dtbo
                          display-lt161010_overlay.dtbo
                          display-lt170410_overlay.dtbo
                          display-vga_overlay.dtbo
                          touch-atmel-mxt_overlay.dtbo
------------------------------------------------------------
gpiochip0 - 32 lines:
	line   0:   "MXM3_279"       unused   input  active-high
	line   1:   "MXM3_277"       unused   input  active-high
	line   2:   "MXM3_135"       unused   input  active-high
	line   3:   "MXM3_203"       unused   input  active-high
	line   4:   "MXM3_201"       unused   input  active-high
	line   5:   "MXM3_275"       unused   input  active-high
	line   6:   "MXM3_110"       unused   input  active-high
	line   7:   "MXM3_120"       unused   input  active-high
	line   8: "MXM3_1/GPIO1" unused output active-high
	line   9: "MXM3_3/GPIO2" unused output active-high
	line  10:   "MXM3_124"       unused   input  active-high
	line  11:   "MXM3_122"       unused   input  active-high
	line  12: "MXM3_5/GPIO3" unused output active-high
	line  13: "MXM3_7/GPIO4" unused output active-high
	line  14:      unnamed       unused   input  active-high
	line  15:      unnamed       unused   input  active-high
	line  16:     "MXM3_4"       unused   input  active-high
	line  17:   "MXM3_211"       unused   input  active-high
	line  18:   "MXM3_209"       unused   input  active-high
	line  19:     "MXM3_2"       unused   input  active-high
	line  20:   "MXM3_136"       unused   input  active-high
	line  21:   "MXM3_134"       unused   input  active-high
	line  22:     "MXM3_6"       unused   input  active-high
	line  23:     "MXM3_8"       unused   input  active-high
	line  24:   "MXM3_112"       unused   input  active-high
	line  25:   "MXM3_118"       unused   input  active-high
	line  26:   "MXM3_114"       unused   input  active-high
	line  27:   "MXM3_116"       unused   input  active-high
	line  28:      unnamed       unused   input  active-high
	line  29:      unnamed       unused   input  active-high
	line  30:      unnamed       unused  output  active-high
	line  31:      unnamed "usb3503 connect" output active-high [used]
gpiochip4 - 32 lines:
	line   0:    "MXM3_18"       unused   input  active-high
	line   1: "MXM3_11/GPIO5" unused output active-high
	line   2: "MXM3_13/GPIO6" unused output active-high
	line   3:   "MXM3_274"       unused   input  active-high
	line   4:    "MXM3_84" "regulator-usb-host-vbus" output active-high [used]
	line   5:   "MXM3_262"       unused   input  active-high
	line   6:    "MXM3_96"       unused   input  active-high
	line   7:      unnamed       unused   input  active-high
	line   8:      unnamed       unused   input  active-high
	line   9:      unnamed       unused   input  active-high
	line  10:      unnamed       unused   input  active-high
	line  11:      unnamed     "enable"  output  active-high [used]
	line  12:   "MXM3_190"         "cd"   input   active-low [used]
	line  13:      unnamed       unused   input  active-high
	line  14:      unnamed       unused   input  active-high
	line  15:      unnamed       unused   input  active-high
	line  16:   "MXM3_269"       unused   input  active-high
	line  17:   "MXM3_251"       unused   input  active-high
	line  18:   "MXM3_253"       unused   input  active-high
	line  19:   "MXM3_295"       unused   input  active-high
	line  20:   "MXM3_299"       unused   input  active-high
	line  21:   "MXM3_301"       unused   input  active-high
	line  22:   "MXM3_297"       unused   input  active-high
	line  23:   "MXM3_293"       unused   input  active-high
	line  24:   "MXM3_291"       unused   input  active-high
	line  25:   "MXM3_289"       unused   input  active-high
	line  26:   "MXM3_287"       unused   input  active-high
	line  27:      unnamed       unused   input  active-high
	line  28:      unnamed       unused   input  active-high
	line  29:      unnamed       unused   input  active-high
	line  30:      unnamed       unused   input  active-high
	line  31:      unnamed       unused   input  active-high

and the same for another device running TorizonOS 6.5. As far as we tested, everything except the GPIOs seems to work fine.

Software summary
------------------------------------------------------------
Bootloader:               U-Boot
Kernel version:           5.15.129-6.5.0+git.6f8fd49366db #1-TorizonCore SMP PREEMPT Fri Dec 22 11:15:52 UTC 2023
Kernel command line:      pci=nomsi root=LABEL=otaroot rootfstype=ext4 quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3 ostree=/ostree/boot.1/torizon/dce1dad5733ea7972c208c5d92f38666568c732b776074e69e2cd0b79eca6eac/0 kernel.sched_rt_runtime_us=-1
Distro name:              NAME="TorizonCore"
Distro version:           VERSION_ID=6.5.0-build.8
Distro variant:           VARIANT="Docker"
Hostname:                 *******
------------------------------------------------------------

Hardware info
------------------------------------------------------------
HW model:                 Toradex Apalis iMX8QM V1.1 on Apalis Evaluation Board V1.2
Toradex version:          0037 V1.1B
Serial number:            06738340
Processor arch:           aarch64
------------------------------------------------------------

Device tree
------------------------------------------------------------
Device tree enabled:      imx8qm-apalis-v1.1-eval.dtb
Compatible string:        toradex,apalis-imx8-v1.1-evaltoradex,apalis-imx8-v1.1fsl,imx8qm
Device trees available:
                          imx8qm-apalis-eval.dtb
                          imx8qm-apalis-ixora-v1.1.dtb
                          imx8qm-apalis-v1.1-eval.dtb
                          imx8qm-apalis-v1.1-ixora-v1.1.dtb
                          imx8qm-apalis-v1.1-ixora-v1.2.dtb
                          imx8qp-apalis-v1.1-eval.dtb
                          imx8qp-apalis-v1.1-ixora-v1.1.dtb
                          imx8qp-apalis-v1.1-ixora-v1.2.dtb
------------------------------------------------------------

Device tree overlays
------------------------------------------------------------
Overlays enabled:         fdt_overlays=apalis-imx8_hdmi_overlay.dtbo apalis-imx8_panel-cap-touch-10inch-lvds_overlay.dtbo custom-kargs_overlay.dtbo
Overlays available:
                          apalis-imx8_ar0521_overlay.dtbo
                          apalis-imx8_hdmi_overlay.dtbo
                          apalis-imx8_mezzanine_can_overlay.dtbo
                          apalis-imx8_mezzanine_ov5640_overlay.dtbo
                          apalis-imx8_mezzanine_panel-cap-touch-10inch-lvds_overlay.dtbo
                          apalis-imx8_ov5640_overlay.dtbo
                          apalis-imx8_panel-cap-touch-10inch-lvds_overlay.dtbo
                          apalis-imx8_panel-lvds-dual-channel-1080p_overlay.dtbo
                          apalis-imx8_resistive-touch_overlay.dtbo
                          apalis-imx8_spi1_spidev_overlay.dtbo
                          apalis-imx8_spi2_spidev_overlay.dtbo
                          custom-kargs_overlay.dtbo
------------------------------------------------------------
gpiochip0 - 32 lines:
	line   0:      unnamed       unused   input  active-high
	line   1:      unnamed       unused   input  active-high
	line   2:      unnamed       unused   input  active-high
	line   3:      unnamed       unused   input  active-high
	line   4:      unnamed       unused   input  active-high
	line   5:      unnamed       unused   input  active-high
	line   6:      unnamed       unused   input  active-high
	line   7:      unnamed       unused   input  active-high
	line   8:      unnamed       unused  output  active-high
	line   9:      unnamed       unused  output  active-high
	line  10:      unnamed       unused   input  active-high
	line  11:      unnamed       unused   input  active-high
	line  12:      unnamed       unused  output  active-high
	line  13:      unnamed       unused  output  active-high
	line  14:      unnamed       unused   input  active-high
	line  15:      unnamed       unused   input  active-high
	line  16:      unnamed       unused   input  active-high
	line  17:      unnamed       unused   input  active-high
	line  18:      unnamed       unused   input  active-high
	line  19:      unnamed       unused   input  active-high
	line  20:      unnamed       unused   input  active-high
	line  21:      unnamed       unused   input  active-high
	line  22:      unnamed       unused   input  active-high
	line  23:      unnamed       unused   input  active-high
	line  24:      unnamed       unused   input  active-high
	line  25:      unnamed       unused   input  active-high
	line  26:      unnamed       unused   input  active-high
	line  27:      unnamed       unused   input  active-high
	line  28:      unnamed       unused   input  active-high
	line  29:      unnamed       unused   input  active-high
	line  30:      unnamed       unused   input  active-high
	line  31:      unnamed       unused   input  active-high
gpiochip4 - 32 lines:
	line   0:      unnamed       unused   input  active-high
	line   1:      unnamed       unused   input  active-high
	line   2:      unnamed       unused   input  active-high
	line   3:      unnamed       unused   input  active-high
	line   4:      unnamed       unused   input  active-high
	line   5:      unnamed       unused   input  active-high
	line   6:      unnamed       unused   input  active-high
	line   7:   "MXM3_198"       unused   input  active-high
	line   8:    "MXM3_35"       unused   input  active-high
	line   9:   "MXM3_164"         "cd"   input   active-low [used]
	line  10:      unnamed       unused   input  active-high
	line  11:      unnamed     "enable"  output  active-high [used]
	line  12:      unnamed       unused   input  active-high
	line  13:      unnamed       unused   input  active-high
	line  14:   "MXM3_217"       unused   input  active-high
	line  15:   "MXM3_215"       unused   input  active-high
	line  16:      unnamed       unused   input  active-high
	line  17:      unnamed       unused   input  active-high
	line  18:   "MXM3_193"       unused   input  active-high
	line  19:   "MXM3_194"       unused   input  active-high
	line  20:    "MXM3_37"       unused   input  active-high
	line  21:      unnamed       unused   input  active-high
	line  22:   "MXM3_271"       unused   input  active-high
	line  23:   "MXM3_273"       unused   input  active-high
	line  24:   "MXM3_195"       unused   input  active-high
	line  25:   "MXM3_197"       unused   input  active-high
	line  26:   "MXM3_177"       unused   input  active-high
	line  27:   "MXM3_179"       unused   input  active-high
	line  28:   "MXM3_181"       unused   input  active-high
	line  29:   "MXM3_183"       unused   input  active-high
	line  30:   "MXM3_185"       unused   input  active-high
	line  31:   "MXM3_187"       unused   input  active-high

We use torizoncore-builder to build the images and also updated the sources for the device tree to be taken from the new kernel version 5.15.

input:
  easy-installer:
    toradex-feed:
      version: "6.5.0"
      release: quarterly
      machine: apalis-imx8
      distro: torizon
      variant: torizon-core-docker
      build-number: 8

customization:
  kernel:
    arguments:
      - kernel.sched_rt_runtime_us=-1
  filesystem:
     - changes
     - changes-manual
  device-tree:
    include-dirs:
      - linux/include
      - linux/arch/arm64/boot/dts/freescale
    custom: linux/arch/arm64/boot/dts/freescale/imx8qm-apalis-v1.1-eval.dts
    overlays:
      clear: false
      add:
        - device-trees-new/overlays/apalis-imx8_hdmi_overlay.dts
        - device-trees-new/overlays/apalis-imx8_panel-cap-touch-10inch-lvds_overlay.dts

Hi @kferrari !

Reproducible

I just checked on Torizon OS 5.7.2, 6.5.0 (monthly) and 6.6.0 (nightly) and also on Toradex Reference Multimedia 6.6.0 (monthly).

The “shift” on the gpiochip numbering is reproducible.

Not a bug

But this is not a bug. The kernel probes and generate the gpiochip numbers during runtime and the probe ordering is not deterministic.

That’s why Toradex introduced the usage of gpio-line-numbers in our device tree. You can check it on imx8-apalis-v1.1.dtsi « freescale « dts « boot « arm64 « arch - linux-toradex.git - Linux kernel for Apalis, Colibri and Verdin modules

libgpiod to the rescue!

By using libgpiod you can get the gpio lines by name (libgpiod: libgpiod public API). This is the most reliable way to not depend on the probing order of the kernel.

For the curious (like me :stuck_out_tongue: )

If, by curiosity, you would like to know what gpiochip is being probed first, you can, on your module, do something like (here I was using Toradex Reference Multimedia Image from BSP 6.6.0, but should be the same for Torizon OS :wink: ):

root@apalis-imx8-07013299:~# tdx-info

Software summary
------------------------------------------------------------
Bootloader:               U-Boot
Kernel version:           5.15.148-6.6.0-6.6.0+git.23a8e831749d #1 SMP PREEMPT Thu Feb 29 20:25:21 UTC 2024
Kernel command line:      pci=nomsi root=PARTUUID=95e92dfe-02 ro rootwait console=tty1 console=ttyLP1,115200 consoleblank=0 earlycon
Distro name:              NAME="TDX Wayland with XWayland"
Distro version:           VERSION_ID=6.6.0-build.12
Distro variant:           -
Hostname:                 apalis-imx8-07013299
------------------------------------------------------------

Hardware info
------------------------------------------------------------
HW model:                 Toradex Apalis iMX8QM V1.1 on Apalis Evaluation Board
Toradex version:          0037 V1.1C
Serial number:            07013299
Processor arch:           aarch64
------------------------------------------------------------

root@apalis-imx8-07013299:~# ls -hal /sys/bus/gpio/devices/gpiochip*
lrwxrwxrwx 1 root root 0 Apr 17 07:42 /sys/bus/gpio/devices/gpiochip0 -> ../../../devices/platform/bus@58000000/58222000.gpio/gpiochip0
lrwxrwxrwx 1 root root 0 Apr 17 07:42 /sys/bus/gpio/devices/gpiochip1 -> ../../../devices/platform/bus@58000000/58242000.gpio/gpiochip1
lrwxrwxrwx 1 root root 0 Apr 17 07:42 /sys/bus/gpio/devices/gpiochip2 -> ../../../devices/platform/bus@5d000000/5d080000.gpio/gpiochip2
lrwxrwxrwx 1 root root 0 Apr 17 07:42 /sys/bus/gpio/devices/gpiochip3 -> ../../../devices/platform/bus@5d000000/5d090000.gpio/gpiochip3
lrwxrwxrwx 1 root root 0 Apr 17 07:42 /sys/bus/gpio/devices/gpiochip4 -> ../../../devices/platform/bus@5d000000/5d0a0000.gpio/gpiochip4
lrwxrwxrwx 1 root root 0 Apr 17 07:42 /sys/bus/gpio/devices/gpiochip5 -> ../../../devices/platform/bus@5d000000/5d0b0000.gpio/gpiochip5
lrwxrwxrwx 1 root root 0 Apr 17 07:42 /sys/bus/gpio/devices/gpiochip6 -> ../../../devices/platform/bus@5d000000/5d0c0000.gpio/gpiochip6
lrwxrwxrwx 1 root root 0 Apr 17 07:42 /sys/bus/gpio/devices/gpiochip7 -> ../../../devices/platform/bus@5d000000/5d0d0000.gpio/gpiochip7
lrwxrwxrwx 1 root root 0 Apr 17 07:42 /sys/bus/gpio/devices/gpiochip8 -> ../../../devices/platform/bus@5d000000/5d0e0000.gpio/gpiochip8
lrwxrwxrwx 1 root root 0 Apr 17 07:42 /sys/bus/gpio/devices/gpiochip9 -> ../../../devices/platform/bus@5d000000/5d0f0000.gpio/gpiochip9

In my case, for example, the gpiochip0 is the 58222000.gpio, which refers to gpio0_mipi_csi0 (imx8-ss-img.dtsi « freescale « dts « boot « arm64 « arch - linux-toradex.git - Linux kernel for Apalis, Colibri and Verdin modules) on the device tree.

Let us know if this helps you and/or if you need further help :slight_smile:

Best regards,

1 Like

Dear Henrique

First of all thank you very much for the swift reply! I really appreciate that.

Your explanation makes perfect sense and I could indeed confirm that the probe ordering is different (and identical to the example you posted).

In case anyone stumbles across this post, I’d like to give a quick rundown of the steps we needed to make to overcome the issue. We need our containers to be cross-compatible, so instead of just updating the exposed devices in docker-compose.yml, I bound /dev as volume and add a device-cgroup-rule to allow access to only the gpiochip* devices.

before:

devices:
  - "/dev/gpiochip0"
  - "/dev/gpiochip4"

after:

volumes:
  - type: bind
    source: /dev
    target: /dev
device_cgroup_rules:
  - 'c 254:* rmw' # check group with ls -l /dev/gpiochip*

Using gpiofind to toggle pins by name works for both the old and new implementation:

# from https://www.acmesystems.it/gpiod
sudo gpioset `gpiofind "PA24"`=1

Again, thank you very much! I would have never checked the gpiochip mapping :sweat_smile:

All the best,
Kim

Hi @kferrari !

Good to know that it helped you! Thanks for the nice feedback and sharing how you did it!

BTW, for completeness, we have this documentation about GPIO on Torizon OS: How to Use GPIO on Torizon OS | Toradex Developer Center

Have a nice day!