Boot MCU from U-Boot on AM62X

cillex · October 10, 2023, 1:17pm

Hello Team,
I am currently trying to get the M4 on AM62 running.
After compiling an example from TI I don’t get it to run on the SOM.

Setting up Yocto on A53 worked, Easy Installer as well and entering U-Boot via USB (verdin development board) was no issue.

I used the SDK provided by TI:
mcu_plus_sdk_am62x_09_00_00_19)
and compiled it according to their instructions using make (automatically
uses ti_cgt_armllvm_2.1.3.LTS, gcc-arm-9.2.2029.12-x86_64_aarch64-none-elf and gcc-arm-none-eabi-7-2017-q4-major).

The output contains a file “gpio_led_blink.release.out” which is in fact just a *.elf file (objdump -x FILE says it is elf32-little)
gpio_led_blink.release.out (305.2 KB)
I tried to copy this file via tftp to loadaddr and use "bootelf ${loadaddr} to run it.

The boot process interrupts with following error message:
“Error” handler, esr 0xbf000000
elr: 0000000080897258 lr : 000000008089959c (reloc)
elr: 000000009bf98258 lr : 000000009bf9a59c
x0 : 0000000000008200 x1 : 0000000088200180
x2 : 0000000000004fe0 x3 : 0000000000000040
x4 : e7ef0a01f10a3401 x5 : 0000000000000000
x6 : 000000009bfa7439 x7 : 0000000000000044
x8 : 0000000000000010 x9 : 0000000000000008
x10: 0000000000000000 x11: 000000000000000a
x12: 0000000000000004 x13: 0000000000000200
x14: 000000000000000e x15: 0000000000000002
x16: 000000009bf17308 x17: 0000000000000000
x18: 0000000099efed80 x19: 000000008824c064
x20: 0000000000000002 x21: 0000000000000028
x22: 0000000088200000 x23: 0000000000000003
x24: 000000009bfe146c x25: 0000000000000000
x26: 0000000000000000 x27: 0000000000000000
x28: 0000000099f20810 x29: 0000000099eec8a0

Code: eb03005f 540001e1 d65f03c0 f8636824 (f8236804)
Resetting CPU …

Do you have any solution or better / different approach to flashing the M4?
I don’t mind using a different method.

Thank you and best regards,
Clemens

lucas_a.tx · October 11, 2023, 1:45pm

Hi @cillex ,

Can you try following the TI instructions from this link :

AM62x MCU+ SDK: Getting Started

See if that works.

Best regards,
Lucas Akira

cillex · October 11, 2023, 2:11pm

Hello @lucas_a.tx ,

I downloaded all files for SDK 08_06 (the one you linked), but I still have the same issues.

I tried three different approaches:

CCS->Compile->launch configuration:

here it tells me “no device found”
maybe due to different development board and no Boot-Mode setup in EVM setup

push file.out as file.elf to u-boot->loadaddr and start with bootelf ${loadaddr}:

gives same error as mentioned before

push am62-mcu-m4f0_0-fw to linux and boot with remoteproc:

shows booting on startup:
[ 6.781123] k3-m4-rproc 5000000.m4fss: configured M4 for remoteproc mode
[ 6.852795] remoteproc remoteproc0: 5000000.m4fss is available
[ 6.939738] remoteproc remoteproc0: powering up 5000000.m4fss
[ 6.946606] remoteproc remoteproc0: Booting fw image am62-mcu-m4f0_0-fw, size 24812
[ 7.822933] remoteproc remoteproc1: 78000000.r5f is available
[ 7.823088] remoteproc remoteproc1: attaching to 78000000.r5f
[ 7.878822] remoteproc remoteproc1: remote processor 78000000.r5f is now attached
[ 10.716467] remoteproc remoteproc2: 30074000.pru is available
[ 10.728239] remoteproc remoteproc3: 30078000.pru is available
but when started manually with “echo start > /sys/class/remoteproc/remoteproc0/state”
[ 54.053618] remoteproc remoteproc0: powering up 5000000.m4fss
[ 54.060613] remoteproc remoteproc0: Booting fw image am62-mcu-m4f0_0-fw, size 24812
[ 54.069503] remoteproc remoteproc0: Boot failed: -22
could this be a device tree issue?

lucas_a.tx · October 17, 2023, 2:36pm

Hi @cillex ,

From the startup logs you provided apparently the image (am62-mcu-m4f0_0-fw) started correctly. Just to be sure, when you tried starting it manually after the boot, did you stop the current execution? The M4 can’t start an image if it already is executing something else.

Best regards,
Lucas Akira

cillex · October 19, 2023, 12:19pm

HI @lucas_a.tx ,

I am still trying to make it work.

After booting linux I still get the same positive remoteproc entries in dmesg, but no signals on any assigned Pins (neither UART@MCU nor GPIO). Is this maybe because of the default Devicetree? If so, is there any easy solution for disabling the specified Pins only?

When stop-/starting it from within linux I always get:

root@verdin-am62-15133540:~# echo stop > /sys/class/remoteproc/remoteproc0/state
sh: write error: Invalid argument
root@verdin-am62-15133540:~# echo start > /sys/class/remoteproc/remoteproc0/state
[ 3898.027936] remoteproc remoteproc0: powering up 5000000.m4fss
[ 3898.034977] remoteproc remoteproc0: Booting fw image am62-mcu-m4f0_0-fw, size 24668
[ 3898.043883] remoteproc remoteproc0: Boot failed: -22
sh: write error: Invalid argument

so it looks like there is something wrong with the commands, but at the same time the start command is executed.

And I have an additional question regarding the same issue:

Is there any way to boot the m4 directly from u-boot?

Best Regards,
Clemens

lucas_a.tx · October 20, 2023, 8:11pm

Hi @cillex ,

Is the binary file you’re trying to run on the Cortex-M actually named am62-mcu-m4f0_0-fw and in /lib/firmware/?

I ask this because the invalid argument errors might be caused by an invalid path i.e. it cannot find the binary file.

Is there any way to boot the m4 directly from u-boot?

Yes, it is possible, but we currently don’t have documentation on how to do this specifically for the AM62 just yet. For now you can try to take a look at our articles for other SoMs and use that as a basis:

Heterogeneous Multi-core Processing (HMP) Documentation Overview | Toradex Developer Center

We’re currently working on updating these articles for the AM62 as well.

Best regards,
Lucas Akira

cillex · October 23, 2023, 1:20pm

Hi @lucas_a.tx ,

I double checked the file system and tested both directly placed, as well as via symlink:

root@verdin-am62-15133540:~# ls -l /lib/firmware/
lrwxrwxrwx 1 root root 36 Oct 23 13:11 am62-mcu-m4f0_0-fw → /lib/firmware/mcu/am62-mcu-m4f0_0-fw

I still get the same error with both tested versions.
Do you have something like an easy “UART+LED_BLINK” example, so I could maybe test the setup?

The Auto-boot from linux option (lib/firmware/am…) doesn’t work on any of the images:
I also checked for differences on Multimedia/Minimal image and Torizon. The first two both show me the same errors, on Torizon it is completely impossible due to “read only file systems”

The provided Documentation uses u-boot “bootaux” command to run the file, but this is missing on the am62x and therefore not functional at the moment.

lucas_a.tx · October 23, 2023, 9:14pm

Hi @cillex ,

I double checked the file system and tested both directly placed, as well as via symlink

Right, you seemed to do everything correctly.

Do you have something like an easy “UART+LED_BLINK” example, so I could maybe test the setup?

We have separate examples for UART and LED, but none specifically for the AM62 right now.

We were able to reproduce the remoteproc remoteproc0: Boot failed: -22 issue when echoing start to /sys/class/remoteproc/remoteproc0/state. It could be that recent versions of the TI SDK or toolchain break something, or maybe the it is related to the kernel version we’re using.

I’ll report this internally for further investigation.

How critical is this issue for you? Is your development currently blocked by it?

Best regards,
Lucas Akira

cillex · October 24, 2023, 2:05pm

Hi @lucas_a.tx ,

Sadly the usage of the m4 is mandatory to our project, without it we need to consider using a different system.

For now every possible method of flashing m4 would be fine for development.
If you have any other ideas on how to flash the m4 I would like to try them.
Until then we can not proceed with the project.

I would be interested in trying to flash your UART example. If you could share it I would appreciate it.

We also tested the newest Kernel version, but also without positive results:
Linux verdin-am62-15133540 6.1.46-6.4.0-devel+git.5ed54483177b #1 SMP PREEMPT Wed Oct 4 10:01:54 UTC 2023 aarch64 GNU/Linux

Thank you very much.
Best Regards,

Clemens

lucas_a.tx · October 25, 2023, 7:35pm

Hi @cillex ,

I would be interested in trying to flash your UART example. If you could share it I would appreciate it.

I meant these examples are included in our article on setting up the MCUXpresso SDK for the NXP modules, so they probably are not applicable to the AM62.

We’ll discuss your problem with the relevant teams and prioritize it according to our available resources.

In the meantime, can you try building the examples you tried before, but using older TI SDKs to see if it works?

Best regards,
Lucas Akira

cillex · October 31, 2023, 10:31am

Hi @lucas_a.tx ,

I checked a lot of different TI SDKs from the oldest to the newest and they all show the same issues as mentioned before (Errors on booting from u-boot and from linux).

Please provide me updates as soon as you have any new methods to test powering and flashing the M4.
If you know any other options, even complicated once, I would be happy to try them.

Until then we have to hold the project.

Thank you for all your help,

Regards,
Clemens

henrique.tx · November 7, 2023, 12:48pm

Hi @cillex !

We did some investigation and seems like we found the culprit

After adding some printk to remoteproc stuff in kernel/drivers, we found out that remoteproc is failing due to lack of resource table in the binary built for M4. You can see it here:

https://git.toradex.com/cgit/linux-toradex.git/tree/drivers/remoteproc/remoteproc_elf_loader.c?h=toradex_ti-linux-6.1.y#n338

Indeed checking the binary for the hello-world from the TI SDK, there is no resource table.

This thread in TI forum is related to the issue: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1114668/am6252-load-m4-firmware-generate-from-mcu-rtos-sdk

We tried to manually “inject” a resource table into the linker script of the hello-world, but it didn’t work: the -22 error occurred.

After, we compiled some IPC demo from TI SDK(according to the TI thread thread referenced above, it will have a resource table) and ran it. The -22 didn’t happen, which is a win (but it failed differently).

We specifically compiled the examples/drivers/ipc/ipc_rpmsg_echo_linux/am62x-sk/m4fss0-0_freertos. Here is the output (the rbeims stuff was added by @rafael.tx: it prints the function and the line in the source code):

root@verdin-am62-15133532:~# echo start > /sys/class/remoteproc/remoteproc0/state
[  832.694530] remoteproc remoteproc0: powering up 5000000.m4fss
[  832.701236] remoteproc remoteproc0: Booting fw image am62-mcu-m4f0_0-fw, size 54860
[  832.709141] rbeims: rproc_fw_boot:1393
[  832.713102] rbeims: rproc_fw_boot:1399 0
[  832.717451] rbeims: rproc_fw_boot:1407 0
[  832.721686] rbeims: rproc_fw_boot:1417 0
[  832.728117] rbeims: rproc_fw_boot:1429 0
[  832.732220] rbeims: rproc_fw_boot:1437 0
[  832.736362] rbeims: rproc_start:1271
[  832.745347] rbeims: rproc_start:1279
[  832.749578] rbeims: rproc_start:1295
[  832.754771] rbeims: k3_m4_rproc_request_mbox:111 ret: 0
[  832.760212] rbeims: k3_m4_rproc_start:254 ret: 0
[  832.765329] rbeims: k3_rproc_release:95 ret: 0
[  832.769848] rbeims: k3_m4_rproc_start:259 ret: 0
[  832.774597] rbeims: rproc_start:1304
[  832.778338] rproc-virtio rproc-virtio.6.auto: assigned reserved memory node m4f-dma-memory@90
[  832.789448] virtio_rpmsg_bus virtio1: rpmsg host is online
[  832.795309] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
[  832.803240] rproc-virtio rproc-virtio.6.auto: registered virtio1 (type 7)
[  832.817340] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
[  832.828598] rbeims: rproc_start:1312
[  832.834959] rbeims: rproc_start:1321
[  832.840045] remoteproc remoteproc0: remote processor 5000000.m4fss is now up
[  832.848086] rbeims: rproc_fw_boot:1445 0

This is the part that it fails:

[  850.280024] ti-sci 44043000.system-controller: Mbox timedout in)
[  850.290150] ti-sci 44043000.system-controller: Mbox send fail -110
[  850.296379] ti-sci-clk 44043000.system-controller:clock-controller: determine-rate failed fo0
[  850.307179] cpu cpu0: _opp_config_clk_single: failed to set clock rate: -22
[  850.314200] cpufreq: __target_index: Failed to change cpu frequency: -22
[  851.336010] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: sci_clk_determi)
[  851.346133] ti-sci 44043000.system-controller: Mbox send fail -110
[  851.352362] ti-sci-clk 44043000.system-controller:clock-controller: determine-rate failed fo0
[  852.392006] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: sci_clk_determi)
[  852.402140] ti-sci 44043000.system-controller: Mbox send fail -110
[  852.408372] ti-sci-clk 44043000.system-controller:clock-controller: determine-rate failed fo0
[  853.448006] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: sci_clk_determi)
[  853.458138] ti-sci 44043000.system-controller: Mbox send fail -110
[  853.464373] ti-sci-clk 44043000.system-controller:clock-controller: determine-rate failed fo0
[  853.475169] cpu cpu0: _opp_config_clk_single: failed to set clock rate: -22
[  853.482188] cpufreq: __target_index: Failed to change cpu frequency: -22
[  854.503999] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: sci_clk_determi)
[  854.514115] ti-sci 44043000.system-controller: Mbox send fail -110
[  854.520341] ti-sci-clk 44043000.system-controller:clock-controller: determine-rate failed fo0
[  855.559997] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: sci_clk_determi)
[  855.570122] ti-sci 44043000.system-controller: Mbox send fail -110
[  855.576370] ti-sci-clk 44043000.system-controller:clock-controller: determine-rate failed fo0
[  856.616004] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: sci_clk_determi)
[  856.626131] ti-sci 44043000.system-controller: Mbox send fail -110
[  856.632366] ti-sci-clk 44043000.system-controller:clock-controller: determine-rate failed fo0
[  856.643160] cpu cpu0: _opp_config_clk_single: failed to set clock rate: -22
[  856.650176] cpufreq: __target_index: Failed to change cpu frequency: -22

As you can see from the logs above, it is a progress at least

We intend to continue investigating this latter this week.

Hopefully, this helps you and you can try something in parallel.

Have a nice day!

Best regards,

cillex · November 7, 2023, 2:43pm

Hi @henrique.tx ,

Thank you for your detailed update.
I am going to take a look at it and try to reproduce it myself as well.

I am looking forward to hear from you again.

Best Regards,
Clemens

cillex · November 9, 2023, 4:04pm

Hi @henrique.tx

I tried to reproduce your findings and confirm the same issue on my side.

I also dug into the TI forum and asked for advice on that matter and they forwarded following link:
https://dev.ti.com/tirex/explore/node?node=A__AVn3JGT9fqm0PbS.pegO-g__AM62-ACADEMY__uiYMDcq__LATEST

With this manual I was able to get

[ 7.231552] remoteproc remoteproc0: remote processor 5000000.m4fss is now up
and: root@verdin-am62-15133540:~# cat /sys/class/remoteproc/remoteproc0/state → running

Sadly even though everything looks like it is working there is no communication on the UART (ttyUSBX), nor /sys/kernel/debug and also nothing visible on the pins (when using the hello_world example).
There is also no GPIO blink, but this could be because of missing devicetree alteration (gpio_blink).

I hope you have something new to share as soon as possible.

Best Regards,
Clemens

henrique.tx · November 13, 2023, 9:24pm

Hi @cillex!

Thanks for your update!

We could run the RPMsg demo with the following modification on the device tree:

diff --git a/arch/arm64/boot/dts/ti/k3-am62-verdin.dtsi b/arch/arm64/boot/dts/ti/k3-am62-verdin.dtsi
index 0f62d9cea1f8..1a94c0885f4c 100644
--- a/arch/arm64/boot/dts/ti/k3-am62-verdin.dtsi
+++ b/arch/arm64/boot/dts/ti/k3-am62-verdin.dtsi
@@ -160,6 +160,11 @@ reserved-memory {
                #address-cells = <2>;
                #size-cells = <2>;
                ranges;
+               rtos_ipc_memory_region: ipc-memories@9c800000 {
+                       compatible = "shared-dma-pool";
+                       reg = <0x00 0x9c800000 0x00 0x00300000>;
+                       no-map;
+               };

@rafael.tx found it here: k3-am62x-sk-common.dtsi « ti « dts « boot « arm64 « arch - linux-toradex.git - Linux kernel for Apalis, Colibri and Verdin modules

After applying this modification and running the RPMsg example, seems like the Cortex-M was up and running, but we got several problems reported by Journald.

So the investigation is still ongoing.

Best regards,

cillex · November 16, 2023, 2:27pm

hi @henrique.tx

I have great news on the MCU:
After trying to add a fix provided by TI, that still was unfinished I finally managed to get the MCU running with UART and GPIO usage:

In addition to the mentioned changes (see previously provided Link) to the RPMsg config there is a need to change some memory regions:

Change and copy all entries in “RAT” (now 8 at 256MB instead of 4 at 512MB, also change addresses)
Change and copy all entries in “MPU ARMv7” (now 3 instead of 2)

With these changes it is possible to compile and run MCU firmware via /lib/firmware/am… on startup.

I still have a lot of problems on SPI but startup and running MCU is fixed with this.

Thank you and best Regards,
Clemens

henrique.tx · November 17, 2023, 11:59am

Hi @cillex !

Thanks a lot for sharing this! It is great for our Community!

It will be certainly useful for more people!

Best regards,