Kernel hangs on torizon-core 5.3.0 onwards when M4 loaded via u-boot

I’ve revisited a technical issue that I first encountered here. Essentially, I can partition the AP and M4 cores in the boot container and modify the u-boot environment variables of loadaddr and ramdisk_addr_r to remove the clash between these memory regions and the ones used by the M4 cores. This will boot fine on any yoctolinux release but will only boot successfully on torizon-core 5.3.0 or older. On torizon-core 5.4.0 onwards if a M4 partitioned boot container is used then the kernel loads but seems to hang, see the output from the AP cores below. It works fine if just the AP partition is present in the boot container.

My question is: What change happened to torizon-core build for 5.4.0 onwards that could prevent the kernel from loading?

U-Boot 2020.04-06964-g33bb8e9683 (Oct 19 2022 - 16:20:23 +0000)

CPU:   NXP i.MX8QM RevB A53 at 1200 MHz

DRAM:  3.8 GiB
MMC:   FSL_SDHC: 0, FSL_SDHC: 1, FSL_SDHC: 2
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
Model: Toradex Apalis iMX8 QuadMax 4GB Wi-Fi / BT IT V1.1C, Serial# 06959235

 BuildInfo:
  - SCFW 216a2c2e, SECO-FW c9de51c0, IMX-MKIMAGE 6a315dbc, ATF
  - U-Boot 2020.04-06964-g33bb8e9683

switch to partitions #0, OK
mmc0(part 0) is current device
flash target is MMC:0
Net:   eth0: ethernet@5b040000
Fastboot: Normal
Normal Boot
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
973 bytes read in 11 ms (85.9 KiB/s)
## Executing script at 83100000
4520 bytes read in 22 ms (200.2 KiB/s)
167875 bytes read in 35 ms (4.6 MiB/s)
43 bytes read in 27 ms (1000 Bytes/s)
Applying Overlay: apalis-imx8_hdmi_overlay.dtbo
2177 bytes read in 35 ms (60.5 KiB/s)
12193586 bytes read in 297 ms (39.2 MiB/s)
Uncompressed size: 30659072 = 0x1D3D200
9179941 bytes read in 228 ms (38.4 MiB/s)
## Flattened Device Tree blob at 83000000
   Booting using the fdt blob at 0x83000000
   Loading Ramdisk to 84d87000, end 85648325 ... OK
   Loading Device Tree to 0000000083fb3000, end 0000000083ffffff ... OK

Starting kernel ...

Hi @EvoFrank,

Sorry for the delay.

Could you please explain to me better how you partitioned the cores inside the boot container and what modifications you did to your bootloader? I would like to run these modifications on my side to check what might be happening with your image. Thanks!

Best Regards,
Hiago.

Thanks for the response,

So, to incorporate the M4 code into the boot container of my iMX8QM I originally followed the instructions on creating a boot container here:
https://developer.toradex.com/linux-bsp/in-depth/build-u-boot-and-linux-kernel-from-source-code/#additional-mandatory-steps-for-specific-socs
these instructions don’t specifically indicate how to incorporate the M4 binary, however, looking at the following page:
https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/i-MX8-Boot-process-and-creating-a-bootable-image/ta-p/1101253
I could see that I could command mkimage with the following:

mkimage_imx8 -soc QM -rev B0 -append mx8qmb0-ahab-container.img -c -flags 0x00200000 -scfw scfw_tcm.bin -ap u-boot-atf.bin a53 0x80000000  -p3 -m4 ~/build/m40_image.bin 0 0x34FE0000 -out flash.bin

Where ~/build/m40_image.bin is the path to the M4 binary for core 0.

! This all requires that the board.c file of the system controller firmware (SCFW) has been updated to correctly partition the board resources and that a new scfw_tcm.bin is generated and passed to mkimage. My particular M4 binary requires CAN and SPI so my partitioning in board.c looks like this:

/* Create M4 0 partition */
        if (rm_is_resource_avail(SC_R_M4_0_PID0) != SC_FALSE)
        {
            sc_rm_mr_t mr;

            /* List of resources */
            static const sc_rsrc_t rsrc_list[9U] =
            {
                SC_R_SYSTEM,
                SC_R_IRQSTR_M4_0,
                SC_R_MU_5B,
                SC_R_MU_7A,
                SC_R_MU_8B,
                SC_R_GPT_4,
                SC_R_SECO_MU_4,
                SC_R_SPI_0,
                SC_R_CAN_0

            };

            /* List of pads */
            static const sc_pad_t pad_list[6U] =
            {
                RM_RANGE(SC_P_M40_I2C0_SCL, SC_P_M40_GPIO0_01),
                RM_RANGE(SC_P_FLEXCAN0_RX, SC_P_FLEXCAN0_TX),
                RM_RANGE(SC_P_SPI0_SCK, SC_P_SPI0_SDI)
            };

Now, this will create the boot container as flash.bin this should then be renamed as imx-boot and can replace the same file in the TEZI archive or what I prefer to do is use the UUU utility and just flash the container directly onto my system.

Now, without modifying the u-boot enviroment variables this will fail on boot. The reason can be seen by looking at the file apalis-imx8.h in the u-boot-toradex source (Current version here) and looking at how the memory is mapped, up until and including 5.3.0 it looked like this (taken from apalis-imx8.h):

/**
 * SYS_SDRAM_BASE	0x80000000	0.125MiB
 * SYS_TEXT_BASE	0x80020000	2.375MiB
 * kernel_addr_r	0x80280000	45.5MiB
 * fdt_addr_r		0x83000000	1MiB
 * scriptaddr		0x83100000	15MiB
 * __RESERVED__		0x84000000	48MiB
 * loadaddr		0x870000000	48MiB
 * ramdisk_addr_r	0x8A000000	288MiB (to hdp_addr)
 * SYS_MEMTEST_START	0x90000000
 * hdp_addr		0x9c000000
 * SYS_MEMTEST_END	0xC0000000
 */

Now the M4’s (i.e. the SCFW) is assigned memory between 0x8800000 to 0x9000000 and then RPMSG and a few other devices use up to 0x96000000. So there is a conflict with loadaddr and the ramdisk.

So the solution I found here: https://community.toradex.com/t/i-mx8qm-u-boot-is-conflicting-with-m4-0-m4-1-ddr-aliasing-memory-region/14087, that is to shift the load address to 0xa000000 and the ramdisk to 0xa3000000. On 5.4.0 onwards therre is an update to apalis-imx8.h and the memory map looks like:

/**
 * SYS_TEXT_BASE        0x80020000      47.9MiB
 * fdt_addr_r           0x83000000      1MiB
 * scriptaddr           0x83100000      15MiB
 * decoder_boot         0x84000000      4MiB
 * encoder_boot         0x86000000      4MiB
 * loadaddr             0x87000000      48MiB
 * Tezi DTB             0x87000000      48MiB
 * Tezi overlays        0x870F0000      48MiB
 * M4 (FreeRTOS)        0x88000000      128MiB
 * ramdisk_addr_r       0x8a000000      96MiB
 * SYS_MEMTEST_START    0x90000000
 * RPMSG/IPU/DSP        0x90000000      96MiB
 * kernel_addr_r        0x96000000      64MiB
 * hdp_addr             0x9c000000      64MiB
 * SYS_MEMTEST_END      0xC0000000
 */

Which still has clashing memory regions so again I shift the loadaddr and ramdisk to 0xa0000000 and 0xa30000000 respectivly.

So by doing this I can get the kernel to boot fine on all versions of torizoncore and yoctolinux if I don’t add a M4 partition to the boot container. If I do add M4 code and partition the boot container for the M4 code It will boot fine on all versions of the tdx-reference-minimal image up to the current 5.7.0 and on torizoncore up to the 5.3.0 release.

However, on the first 5.4.0 development release of torizoncore the kernel doesn’t start, It gives Starting kernel … and sits there.

I think this points to some change in the torizoncore layer between 5.3.0 to 5.4.0-devel-202108 that prevents the kernel from loading correctly.

I would really love to get this solved so I can update to 5.7.0 and get the benifits of a LTS release.

Thanks for your time.

Hi @EvoFrank,

Thanks for all the good and detailed explanations you gave.

I’ll reproduce it on my side and will update you.

Best Regards,
Hiago.

Hi @EvoFrank,

I was able to reproduce the issue with your instructions and I’ll escalate this ticket internally for a discussion with our development team since it’s not a simple thing that we can solve it here (at least from what I saw here).

I’ll keep you updated if anything changes and, meanwhile, I’ll still try to create a workaround for TorizonCore 5.7.

Thanks for your patience.

Best Regards,
Hiago.

Thanks Hiago, I really appreciate it.

~WRD3381.jpg

image001.png

image004.png

image005.png

image006.png

image007.png

image008.png

Hi @EvoFrank,

I have some news about this issue. I talked with our development team and this way of booting the Cortex-M is not intended to be supported right now. Let me explain:

This is broken due to the memory layout defined inside the u-boot configuration file, as you kindly showed. However, supporting this mode right would require a change in the memory layout inside that would cause some compatibility issues with other boards/BSP versions. And every release from now on would require patches to solve this issue, which is not something we want for our products. I hope you understand this part.

However, I would like to help you with this issue for your specific case. My first question is, why do you need to boot the M4 core with the Core A? Can you wait for the Core A to boot and then u-boot will start the M4 core?

This would cause a delay in the boot process of cortex M4 for sure, however, I believe this boot process is very fast since we’re talking about u-boot. Can you explain me better this part?

I believe that booting from u-boot would be simpler too. Here is the reference guide: How to load compiled binaries into Cortex-M | Toradex Developer Center

Let me know so we can discuss it a little further and understand how we can deal with your scenario in the best way possible.

Thanks for your patience.

Best Regards,
Hiago.

Hi @hfranco.tx

Thank you for investigating this.

There seems to be some confusion over the M4 cores. On the first page of the Apalis-iMX8 datasheet the M4 cores are described thus:

Additional to the main CPU complex, all i.MX 8 family members feature two Arm Cortex-M4
processors, which peak up to 266MHz. These two processors are independent of each other and
feature their own dedicated interfaces while they can also access the regular interfaces. This
heterogeneous multi-core system allows for running additional real-time operating systems on the
M4 cores for time- and security-critical tasks. The i.MX 8 features a System Controller Unit (SCU),
which runs on an independent Cortex-M processor. A major task of this controller is resource
management with proper access and permission control to make sure the different M4 cores and
main CPU complex are isolated from each other. This massively increases the safety of the
heterogeneous multi-core system in comparison with older SoC.

My application for the M4 is real-time and safety critical. So in order to achieve this, I need to allow the SCU to correctly partition pins and resources between the M4 and A cores. In addition, for my M4 core to act in real-time it requires priority to the peripherals as they all share the same bus. This is something the SCU can also facilitate.

In order to use this regime on an iMX8, the M4 code along with custom code for the SCU and additionally u-boot (A core code) need to be bundled into a boot container. More information can be found in the NXP post I referenced in the original post.

If I were instead to load the M4 code using u-boot as you suggest. I wouldn’t get the benefits of having the SCU partition pins and resources or give priority to peripherals. This isn’t just a hypothetical situation, the code I currently run on the M4 does not perform in real-time if I load the M4 code via u-boot.

Do you follow this reasoning?

Regards,

EvoFrank

Hi @EvoFrank,

Thanks for giving me this information.

I follow your requisites. I would say it’s also possible to partition everything using the device tree since you can disable the peripherals from linux and use them only in the M4 core. I understand your scenario in this case.

Unfortunately, I can’t guarantee you that our development team will make sure that this feature works on the next releases or that it will be fixed on TorizonCore 5.3+. This is something that will have to be patched by you on every image to allow u-boot to work with the boot container (and then create a new TorizonCore image with these patches).

Despite that, I will take a look at the differences between TorizonCore 5.3 and 5.7 recipes to check whether I can find the root of this problem or not and if I can patch it to create a boot container.

Thanks for your patience. I’ll update you when I have something.

Best Regards,
Hiago.