Torizon OS kernel panic when running M7 application on Verdin IMX8M Plus

jschreib · May 14, 2024, 3:30pm

I’m trying to get a HMP application running on our Verdin IMX8M Plus SoM. I’m following the instructions as outlined here: Heterogeneous Multi-core Processing (HMP) Documentation Overview | Toradex Developer Center. Thus far I’ve successfully acquired and installed the MCUXpresso SDK for Linux (SDK_2.15.000_MIMX8ML8xxxKZ) and the ARM toolchain and have compiled and loaded the “hello_world” demo application. If I run the application from Uboot (run m4boot) all seems well, I see “hello world” printed to the terminal and the characters I type are all echoed back to me. However, when I attempt to boot into the Torizon OS I get an immediate kernel panic. If I don’t run the hello_world application the OS boots just fine.

Here are the relevant details of my setup:

SOM: Verdin iMX8M Plus
Carrier: Dahlia
Torizon OS: torizon-core-docker-verdin-imx8mp-Tezi_6.6.1+build.14 (w/hello_world dt overlay as described in the instructions)

My Uboot variables are set as follows:

load_cmd=ext4load mmc 2:1
m4image=/ostree/deploy/torizon/var/usrlocal/bin/hello_world.bin
m4image_size=18472
loadm4image=ext4load mmc 2:1 0x48200000 /ostree/deploy/torizon/var/usrlocal/bin/hello_world.bin
m4boot=ext4load mmc 2:1 0x48200000 /ostree/deploy/torizon/var/usrlocal/bin/hello_world.bin; cp.b 0x48200000 0x7e0000 18472; dcache flush; bootaux 0x7e0000

The message the accompanies the panic is shown below. The first two lines I also see when the OS boots normally, the difference starts on the third line:

[ 0.892436] imx-drm display-subsystem: no available port
[ 1.012047] rtc-ds1307 0-0032: hctosys: unable to read the hardware clock
[ 1.021201] SError Interrupt on CPU1, code 0x00000000bf000002 – SError
[ 1.021209] CPU: 1 PID: 103 Comm: kworker/u8:2 Not tainted 5.15.148-6.6.1+git.23a8e831749d #1-TorizonCore
[ 1.021216] Hardware name: Toradex Verdin iMX8M Plus WB on Dahlia Board (DT)
[ 1.021220] Workqueue: events_unbound deferred_probe_work_func
[ 1.021234] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=–)
[ 1.021242] pc : of_match_node+0x28/0xb0
[ 1.021250] lr : i2c_imx_probe+0x33c/0xa10
[ 1.021261] sp : ffff80000a6e3a90
[ 1.021264] x29: ffff80000a6e3a90 x28: ffff800009411530 x27: ffff0000c43d03f4
[ 1.021278] x26: ffff0000c43d00f0 x25: ffff80000a269e28 x24: ffff8000094114f8
[ 1.021288] x23: ffff80000a26c980 x22: 000000000000002b x21: ffff0000c015e000
[ 1.021299] x20: ffff0000c015e010 x19: ffff800009411530 x18: 0000000000000001
[ 1.021312] x17: 000000040044ffff x16: 00500072b5503510 x15: 0000000000000000
[ 1.021325] x14: 0000000000000170 x13: 0000000000007400 x12: ffff0000c0515914
[ 1.021337] x11: ffff0000c0515900 x10: aaaaaaaaaaaaaaab x9 : 0000000000000060
[ 1.021348] x8 : ffff800009ea8bb0 x7 : 0000000000000000 x6 : 000000000007a120
[ 1.021360] x5 : 0000000000b71aff x4 : 0000000000000002 x3 : 0000000000000002
[ 1.021372] x2 : 000000000000000c x1 : ffff0000ff7c9838 x0 : ffff80000a26c980
[ 1.021389] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 1.021392] SMP: stopping secondary CPUs
[ 1.021699] Kernel Offset: disabled
[ 1.021700] CPU features: 0x0,00002001,20000846
[ 1.021705] Memory Limit: none

I’m at a loss trying to understand what’s happening here. Any help would be greatly appreciated.

ToradexAI · May 14, 2024, 3:31pm

Hi, I’m the Toradex AI Assistant. I’m in early beta, and I would appreciate your feedback.
The Toradex Support team will follow up with you shortly!

Did you like my answer?

Hello,

It appears that the kernel panic you’re encountering when booting Torizon OS after running the Cortex-M7 application on your Verdin iMX8M Plus SoM might be related to a conflict in resource allocation or initialization between the Cortex-A and Cortex-M cores. Given the details you’ve provided, here are a few steps and considerations that might help resolve the issue:

Review Device Tree Overlays: Ensure that the device tree overlay applied for the hello_world application correctly disables the pins or peripherals that might be shared between the Cortex-A and Cortex-M cores. This is crucial to prevent the Linux kernel from attempting to reinitialize or claim resources already in use by the M7 core.
Synchronize Resource Usage: If the Cortex-M7 application uses peripherals that are also managed by the Linux kernel (e.g., UARTs, I2C, SPI), ensure that these peripherals are not being accessed by the Linux system at the same time. This might require adjusting the device tree or using Linux kernel mechanisms to reserve these peripherals for exclusive Cortex-M7 use.
Check for Firmware Updates: Ensure that your Verdin iMX8M Plus SoM is running the latest firmware version. Firmware updates often include fixes for hardware and low-level software issues that could cause conflicts between the Cortex-A and Cortex-M cores.
Analyze the Boot Sequence: The kernel panic log indicates a problem occurring early in the boot process, possibly related to device initialization. Pay close attention to the sequence of boot operations and look for any anomalies or errors that occur before the panic. This might give clues as to which subsystem or driver is causing the conflict.
Isolate the Issue: Try running a minimal Cortex-M7 application that does not interact with any peripherals or shared resources to see if the kernel panic still occurs. This can help determine if the issue is related to specific resource usage or a more general problem with HMP configuration.
Review HMP Configuration: Double-check the Heterogeneous Multi-core Processing (HMP) configuration to ensure that it’s set up correctly for your specific use case. This includes verifying boot arguments, resource partitioning, and inter-processor communication settings.
Debugging and Logging: Increase the verbosity of kernel logging to get more detailed information about the system state and operations leading up to the kernel panic. This can be done by modifying the kernel command line parameters (e.g., adding earlyprintk and debug).

Given the complexity of HMP systems

Please, also check if these links can help you solve your problem:

How to Run a Hello World on the Cortex-M | Toradex Developer Center.

eric.tx · May 14, 2024, 5:08pm

Hey @jschreib,

Can you verify the overlay that you have included in your build? You should be able to see it while on the shelled into the module and via : ls /proc/device-tree/chosen/overlays/

UART4 (access by both Cortex-M and linux) cannot coexist. So the device tree overlay disables Linux’s access to the UART4.

-Eric

jschreib · May 14, 2024, 5:17pm

Hi @eric.tx,

Thanks for getting back to me. I can see the overlay “cortexm-4-demo.dtbo”, which is the one I added for running the hello_world application. The contents of the overlay are as follows, which I took directly form the instructions:

/dts-v1/;
/plugin/;

/ {
    compatible = "toradex,verdin-imx8mp-wifi-dahlia",
             "toradex,verdin-imx8mp-wifi",
             "toradex,verdin-imx8mp",
             "fsl,imx8mp";
};

&gpio5 {
    gpio-line-names = "SODIMM_42",
              "SODIMM_46",
              "SODIMM_187",
              "SODIMM_20",
              "SODIMM_22",
              "SODIMM_15",
              "SODIMM_196",
              "SODIMM_200",
              "SODIMM_198",
              "SODIMM_202",
              "SODIMM_164",
              "SODIMM_152",
              "SODIMM_116",
              "SODIMM_128",
              "",
              "",
              "SODIMM_55",
              "SODIMM_53",
              "SODIMM_95",
              "SODIMM_93",
              "SODIMM_14",
              "SODIMM_12",
              "SODIMM_129",
              "SODIMM_131",
              "SODIMM_137",
              "SODIMM_139",
              "SODIMM_147",
              "SODIMM_149",
              "SODIMM_151",
              "SODIMM_153";
};

/* Remove pinctrl_gpio_hog4 and add pinctrl_gpio_hog1 */
&iomuxc {
    pinctrl-names = "default";
    pinctrl-0 = <&pinctrl_gpio1>, <&pinctrl_gpio2>,
            <&pinctrl_gpio3>, <&pinctrl_gpio4>,
            <&pinctrl_gpio7>, <&pinctrl_gpio8>,
            <&pinctrl_gpio_hog1>, <&pinctrl_gpio_hog2>, <&pinctrl_gpio_hog3>,
            <&pinctrl_hdmi_hog>;
};

/* Verdin UART_4 */
/* Used by the M7 and then should not be enabled here. */
&uart4 {
    status = "disabled";
};

eric.tx · May 14, 2024, 5:54pm

Hey @jschreib,

I was able to quickly test the hello world w/ imx8mp/dahlia. The only difference is I also have the hmp_overlay enabled for rpmsg. I don’t believe this should have any affect. But I will disable and try hello world again. Also my m4image_size=19000 for a buffer on the hello_world.

Do you have any other modifications to the image? I would try and comment these out of the build and isolate just the hello world dts.

Can you also give the output of sudo tdx-info on the module?

Here’s a screen grab of the linux os booted w/ m4 running hello_world. So it should be able to work.

-Eric

jschreib · May 14, 2024, 6:23pm

Hi @eric.tx,

Ok, I modified by setup so that I only have the torizon OS image and the overlay from my previous post. Nothing else. I get the same behavior (the m7 app runs fine, but the kernel panics when I try to boot it). Note that I’m using the torizoncore-builder to customize the image that includes the overlay. The yaml file is as follows:

input:
  easy-installer:    
    local: images/torizon-core-docker-verdin-imx8mp-Tezi_6.6.1+build.14.tar
customization: 
  device-tree:    
    include-dirs:
      - linux/include/   
    custom: linux/arch/arm64/boot/dts/freescale/imx8mp-verdin-wifi-dahlia.dts   
    overlays:     
      add:      
        - device-trees/overlays/cortexm-4-demo.dts
output:  
  easy-installer:    
    local: images/torizon-core-docker-verdin-imx8mp-Tezi_6.6.1+build.14.CUSTOM

Should I try a different method? Ultimately we want a custom image that includes overlays and a few kernel modules, this was just a first step.

The output of command you requested is:

torizon@verdin-imx8mp-15007313:~$ sudo tdx-info
Password:

Software summary
------------------------------------------------------------
Bootloader:               U-Boot
Kernel version:           5.15.148-6.6.1+git.23a8e831749d #1-TorizonCore SMP PREEMPT Thu Feb 29 20:25:21 UTC 2024
Kernel command line:      root=LABEL=otaroot rootfstype=ext4 quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3 ostree=/ostree/boot.1/torizon/3fe74887c4b89cd479765fe255073fcfdd8c1086704ecdce744e1bd9690558f9/0
Distro name:              NAME="TorizonCore"
Distro version:           VERSION_ID=6.6.1-build.14
Distro variant:           VARIANT="Docker"
Hostname:                 verdin-imx8mp-15007313
------------------------------------------------------------

Hardware info
------------------------------------------------------------
HW model:                 Toradex Verdin iMX8M Plus WB on Dahlia Board
Toradex version:          0058 V1.1A
Serial number:            15007313
Processor arch:           aarch64
------------------------------------------------------------

Out of curiosity, what version of the NXP SDK are you using? I’m using 2.15.000, which I downloaded yesterday.

Thanks for the feedback.

jschreib · May 14, 2024, 6:35pm

Hi @eric.tx,

Ok, so I ADDED the verdin-imx8mp_hmp_overlay.dts to my setup and NOW the kernel boots! It seems like that overlay is required for HMP to work(?)

jschreib · May 14, 2024, 7:10pm

Looking at the panic messages it appears as though the kernel panics while probing the i2c devices (based on the values in the Program Counter[PC] and Link Register[LR]). That HMP overlay disables i2c3. Is that the issue? Why would that overlay need to disable one of the I2C devices?

eric.tx · May 14, 2024, 8:04pm

Hey @jschreib,

It is odd that it was required, I would not have expected this for just the hello world on the Cortex M. I will do some inquiry to see if I can find out more. And see if our developer guide can receive an update if applicable.

-Eric