Boot Failures on Verdin iMX95 with tdx-xwayland-rt

Technical Support
, , ,
1

I am attempting to build and boot a PREEMPT_RT real-time Linux image for the Toradex Verdin iMX95 on the Verdin EVK V1.3 using the tdx-xwayland-rt distro with kernel linux-toradex 6.6.119-rt67. We successfully built the image using Yocto Scarthgap with DISTRO = "tdx-xwayland-rt" and flashed it using the UUU tool. However, I encountered several issues during boot that I have been working through, and I would appreciate guidance from the community.

Issue 1 — Wrong DTB Name at Boot

After flashing the RT image, U-Boot failed to load the device tree, reporting Failed to load 'imx95-19x19-verdin-adv7535-ap1302.dtb'. This DTB does not exist in the RT kernel build. The RT kernel produces DTBs with different naming conventions such as imx95-verdin-wifi-dev.dtb. I worked around this by manually setting fdtfile=imx95-verdin-wifi-dev.dtb in the U-Boot environment.

Issue 2 — vf610_gpio Kernel Panic

After resolving the DTB issue, the kernel panicked during boot with an Asynchronous SError Interrupt at vf610_gpio_probe. The call trace clearly showed the vf610 GPIO driver probing hardware that is not present or accessible on the Verdin iMX95, causing a fatal hardware fault. I resolved this by adding a kernel config fragment to disable the driver with CONFIG_GPIO_VF610=n. I would like to know why this driver is enabled by default in the RT kernel configuration for verdin-imx95, and whether this should be disabled upstream.

Issue 3 — Silent Boot Hang After UART Initialization

After fixing the GPIO panic, the kernel appeared to hang silently after the message 42690000.serial: ttyLP0 at MMIO 0x42690010 is a FSL_LPUART. There was no further output and the board was unresponsive. I discovered that the kernel was actually crashing after this point, but the crash output was invisible because it occurred during the console switch from earlycon to the full ttyLP0 driver. Adding keep_bootcon to the kernel command line revealed the hidden crash output described in Issue 4 below.

Issue 4 — lpuart2 Synchronous External Abort

With keep_bootcon enabled, I discovered the actual crash. The lpuart2 driver (serial@44390000) is enabled in imx95-verdin-dev.dtsi with status = "okay", but its pinctrl configuration fails with scmi-pinctrl: pin_config_set op failed for pin 118, followed immediately by fsl-lpuart 44390000.serial: Error applying setting, reverse things back and a fatal Internal error: synchronous external abort: 0000000096000010 at lpuart_probe+0x558/0x630. I am currently attempting to fix this by patching imx95-verdin-dev.dtsi to set lpuart2 to status = "disabled", but we would like to know if this is a known issue and what the correct fix is.

Issue 5 — fsl_enetc_mdio EPROBE_DEFER Warning

Throughout the boot process, I consistently see the warning fsl_enetc_mdio 0003:01:00.0: Cannot get IERB init status: -517. The error code -517 corresponds to EPROBE_DEFER, indicating the MDIO driver is waiting for the IERB driver which is not available. I added CONFIG_FSL_ENETC_IERB=y to our kernel config fragment which enabled the driver, but the warning persists. We would like to know if this is expected behavior or if there is a missing dependency in the default RT kernel configuration.

In summary, We would appreciate answers to the following questions: Is tdx-xwayland-rt officially tested and supported on verdin-imx95 with kernel 6.6.119-rt67? Are there known patches or workarounds needed for the issues described above? Specifically, should CONFIG_GPIO_VF610 be disabled by default for this machine? Is the lpuart2 pinctrl failure a known issue and what is the recommended fix?

2

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.

PREEMPT_RT Status and Support

Based on the documentation, PREEMPT_RT support is provided in an experimental state for Toradex modules (1). The documentation states that Toradex “only releases PREEMPT_RT BSPs and images in an experimental state. We don’t guarantee any functionality and bug fixes” (1).

The tdx-xwayland-rt distro is available for iMX8 based modules and builds “the kernel with the CONFIG_PREEMPT_RT_FULL option enabled” (2). However, the documentation indicates that these distributions “haven’t been fully validated by Toradex” and that “any tuning, additional kernel configuration, and validation of the RT performance for your own use case must be done at your discretion” (1).

Known Limitations and Validation Status

The available documentation shows that Toradex has “only run smoke tests and cyclictest on Torizon OS to make sure that the system boots properly and the PREEMPT_RT patches are working under stressful conditions” (1). The specific issues you’ve encountered with the Verdin iMX95 are not directly addressed in the available documentation.

Device Tree and Configuration Considerations

For device tree modifications to address the issues you’ve described, you can create device tree overlays to disable problematic peripherals (3)(4). The process involves:

  1. Creating a pin group to modify peripheral configurations (3)
  2. Resolving pin conflicts by disabling conflicting interfaces (3)
  3. Avoiding pinmux conflicts by ensuring no other peripheral uses the same resources (3)

For example, to disable a problematic UART, you would add to your overlay:

&lpuart2 {
    status = "disabled";
};

(3)

Recommendations

Given that PREEMPT_RT support is experimental and the specific issues you’re encountering with Verdin iMX95, I recommend:

  1. Contact Toradex directly for platform-specific PREEMPT_RT guidance, as the documentation states: “If you have questions or want to share your use case with us, so we can have a more accurate understanding of customers’ needs, please contact us” (1)

  2. Consider paid support as “paid support for these alternative solutions may be available from companies in the Toradex partner network” (1)

  3. Use device tree overlays to systematically disable the problematic peripherals you’ve identified while working toward a complete solution

The issues you’ve described appear to be related to the experimental nature of PREEMPT_RT support and may require platform-specific patches that aren’t yet integrated into the standard BSP.

Best regards, Toradex AI Assistant.

Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

3

Hi @kosisochukwu!

I think there may be a misunderstanding.

The iMX95 Verdin EVK is, as the name suggests, an evaluation kit and does not have a Torizon PREEMPT_RT image available. The Yocto build you did was for Verdin iMX95 module, which uses a different configuration (drivers, device tree, etc) compared to the EVK.

This difference is likely the reason for the errors you are encountering. It can be verified in the naming of the device tree files:

  • imx95-19x19-verdin-adv7535-ap1302.dtb: follows the naming convention for the 19x19 EVK
  • imx95-verdin-wifi-dev.dtb: is the standard device tree naming for the Verdin family

If you want to try our Torizon PREEMPT_RT experimental images, I recommend getting an iMX95-based module, such as the Verdin iMX95.

Best regards.

1 Like
4

Hello @kosisochukwu,

Adding to what @lucas_az.tx mentioned, any of our iMX95-based SoMs should be compatible with their respective build with the tdx-xwayland-rt distro.

Depending on the required features, Verdin, Aquila, SMARC, or the upcoming OSM iMX95 could be better suited.

The i.MX95 EVK you currently have is intended for initial evaluation of the i.MX95 SoC.
It has software support provided by NXP and is going to be phased out soon.

Best Regards,
Bruno

1 Like
5

Thank you for the clarification. Does this mean that the Toradex BSP is not supported on the iMX95 Verdin EVK

6

Hello @kosisochukwu,

That is correct.
On the i.MX95 Verdin EVK, the supported BSP is from NXP: GitHub - nxp-imx/meta-imx: i.MX Yocto Project i.MX BSP Layer · GitHub

If you have any questions or need further clarification, please let me know.

Best Regards,
Bruno

1 Like
7

Thanks Bruno.