Use remoteproc to start M7 core on Verdin iMX8M-Plus

Technical Support
, , , ,
1

In the past few months I’ve been able to start M7 coreof Verdin iMX8M-Plus (or Mini) loading the firmware and running it from u-boot.
The following topics:

Last week I talked to @stefano.tx and he told me that now (November 2023), remoteproc approach should work.
And so I tried.
I followed the steps here and this paragraph in particular.
But it doesn’t work.
Here is what happens

# echo start > /sys/class/remoteproc/remoteproc0/state
[   644.366723] remoteproc remoteproc0: powering up imx-rproc                                                                              [   644.366844] remoteproc remoteproc0: Direct firmware load for firmware_imxcm7.elf failed with error -2
[   644.366857] remoteproc remoteproc0: request_firmware failed: -2
[   644.372849] remoteproc remoteproc0: Boot failed: -2

What could be the issue?

2

Hi @vix !

I just answered a related question here:

As you can see, it worked for me.

Could you please follow each step of the article How to Use RemoteProc | Toradex Developer Center and share the output here?

Also, please share the output of tdx-info.

Best regards,

3

Hi @henrique.tx
I think I didn’t explain well what happens.
If I follow the Toradex steps (that you described in your post) everything works.
No problem.
The method to load the firmware is EXT4LOAD Loading Method. This means from U-Boot.

Some months ago, loading the firmware using remoteproc kernel module

$ sudo -s
$ cd /sys/class/remotreproc/remoteproc0/
$ echo stop > state
$ echo rpmsg_lite_str_echo_rtos_imxcm4.elf > firmware
$ echo start > state

was not 100% reliable.
With some firmware it worked, with some other it didn’t work.
If U-Boot has already loaded and started a firmware, sometimes it works.
If U-Booot hasn’t loaded and started a firmware, it doesn’t work.
Or it seems so.

As I wrote, when I talk to @stefano.tx a couple of weeks ago, he told me that this method should work now. And he asked me to try, and to report any issue.

This is the reason of this topic.

4

Hi @vix !

Thanks for the explanation.

I just tried it on Verdin iMX8MP Q 4GB WB IT V1.0B with Torizon OS 6.5.0-devel-202311+build.14.

Indeed when previously starting Cortex-M7 with the firmware on U-Boot, seems like remoteproc works more reliably.

If the Cortex-M7 is not started on U-Boot, remoteproc fails.

I will bring this up internally.

Thanks for reporting!

Best regards,

5

Hi @vix !

By the way, is this currently blocking you in any way?

Always starting the Cortex-M7 from U-Boot seems to be a good workaround to make remoteproc work reliably. Is this workaround suitable for you at the moment?

Best regards,

7

Hi @henrique.tx

if U-Boot is used to start the firmware, the .bin must be included in the Torizon OS build.
And so every firmware update requires a Torizon OS update.
For this reason, this an acceptable short-term solution.
For the long term we would prefer using remoteproc so that the firmware can be deployed and updates in its own container.

8

Hi @vix !

This is indeed a way of doing it, but please keep in mind that you can use the Torizon Subsystem Update to avoid performing an OS update. Please take a look at this article about Subsystem Updates Overview | Toradex Developer Center

An article with an actual example of how to use it is currently in the works and should be available soon.

Best regards,

9

Do you have any news on this issue @henrique.tx?
I’ve been creating an automated pipeline to build and deploy TorizonCore + my containers.
And I don’t find a way to automate the deploying of a u-boot firmware.

Now this is a blocking task for my project.

10

Hi @vix !

Thanks for letting us know how this affects your project.

In my last message, I told you about Torizon Subsystem Update, so the Cortex-M firmware can be loaded during boot, therefore remoteproc is not needed to start Cortex-M. Not to mention that Cortex-M will start earlier, which can be advantageous.

Doesn’t it help you?

11

Hi @henrique.tx
it can be, but I’m not sure if I will use Torizon Cloud in the project (it’s under investigattion at the moment).
The task is setup the pipeline(s) to produce a TEZI image for my project.
Do you have an idea on how a pipleine can pull a specific binary (produced by another pipeline)?
I use Azure DevOps Péipelines, but if you can provide an example for GitHub or Gitlab I can take inspiration.

12

Hi @vix !

Could you explicitly point out where/what is the issue?

I am not experienced with CI/CD pipelines, but since you are using this approach (very nice! :slight_smile: ) I expect you to have:

  1. a repository with the code for your Cortex-M and a pipeline built in there which can build the Cortex-M binary and create releases, and
  2. a repository to build your customized Torizon OS (using TorizonCore Builder), which would get (wget, curl, etc) the binary released from the Cortex-M repository.
    2.1. the binary should be placed in the right path in your tcbuild.yaml by using the filesystem from customization (see more about it in TorizonCore Builder Tool “build” command | Toradex Developer Center).

Since we are talking about repositories and CI/CD, please also check the tips in the section
Version Controlling of the tab Detailed manual of the link above.

After asking internally for some tips, I was pointed to Job artifacts | GitLab documentation.

Here is the idea of a possible implementation (not tested, of course) that was shared with me.

# project that creates the M4 binary
# the project ID is 1234556
build-m4-fw:
  script: my-script.sh -o path/to/artifact/m4.bin
  artifacts:
    paths:
      - path/to/artifact/m4.bin

----------------------------------------------

# project that builds the custom image with TCB
build-tcb:
  scripts: |
             wget https://gitlab.com/api/v4/projects/1234556/jobs/artifacts/main/download?job=build-m4-fw
             torizoncore-builder build

Let us know if this helps you.

Best regards,

13

Hello @vix,
We discovered that to make remoteproc work you need to add the argument clk-imx8mp.mcore_booted=1 to the kernel command line.

Could you try to add this parameter and check if remoteproc starts to work in your case as well?

Best regards,
Rafael

14

Hi @rafael.tx
can you explain how I can add the argument clk-imx8mp.mcore_booted=1 to the kernel command line?
Should I do this with u-boot customization?
Or with TorizonCore Builder?

I add the following lines to tcbuild.yaml

customization:
  # Customization section items.
  kernel:
    arguments:
      # prevents Linux from disabling the root clock of the Cortex M7
      - clk-imx8mp.mcore_booted=1

and if I run sudo tdx-info I get

Kernel command line:      root=LABEL=otaroot rootfstype=ext4 quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3 ostree=/ostree/boot.1/torizon/277551d6531e8143bfac04e4f70fecf5f84b7499d333e44b48a9617b3a635802/0 clk-imx8mp.mcore_booted=1

so I image the parameter is passed in the proper way.

But when I try to run M7 I get

[ 6849.243922] remoteproc remoteproc0: powering up imx-rproc
[ 6849.245325] remoteproc remoteproc0: Booting fw image firmware_imxcm7.elf, size 1251068
[ 6849.245428] imx-rproc imx8mp-cm7: map memory: 00000000b756f985+100000
[ 6849.245466] imx-rproc imx8mp-cm7: map memory: 000000005015e6c2+8000
[ 6849.245476] imx-rproc imx8mp-cm7: map memory: 00000000ad4b4323+8000
[ 6849.245484] imx-rproc imx8mp-cm7: map memory: 000000009895b7cc+1000000
[ 6849.245501] remoteproc remoteproc0: da = 0x80000000 len = 0x2a8 va = 0x00000000d40f8f5d
[ 6849.245509] remoteproc remoteproc0: da = 0x80000400 len = 0x134e8 va = 0x000000000076d005
[ 6849.245586] remoteproc remoteproc0: da = 0x800138e8 len = 0x29c88 va = 0x00000000bcbb84d2
[ 6849.245628] remoteproc remoteproc0: da = 0x80200000 len = 0x100000 va = 0x000000008633db6d
[ 6849.758963]  remoteproc0#vdev0buffer: assigned reserved memory node vdevbuffer@55400000
[ 6849.869243] imx-rproc imx8mp-cm7: imx_rproc_kick: failed (0, err:-62)
[ 6849.875703] virtio_rpmsg_bus virtio0: rpmsg host is online
[ 6849.875785]  remoteproc0#vdev0buffer: registered virtio0 (type 7)
[ 6849.875794] remoteproc remoteproc0: remote processor imx-rproc is now up

and I see no output from M7 debug UART (as it should be).
I see the message

[ 6849.869243] imx-rproc imx8mp-cm7: imx_rproc_kick: failed (0, err:-62)

and it seems strange to me.

If I try

# cat /sys/class/remoteproc/remoteproc0/state

I get

running

but this is strange, since I got no output from UART.

I can add one thing: my CM7 firmware donesn’t run from TCM, but from DDR.
I changed the linker of the firmware in the proper way.
Amnd so my question is: how can remoteproc know where is the right value for the Program Counter PC?
I think that maybe this PC is not set and so it has a default value of 0x7e0000 which is inside TCM.
And so remoteproc thinks it’s running, but from the wrong PC.

I found this topic on NXP and it’s similar to what I see.

Any suggestion?

15

@vix replying here - if you have moved the location of where the CM7 firmware runs in memory, I’m almost positive you are going to have to modify the HMP overlay here:

https://git.toradex.com/cgit/device-tree-overlays.git/tree/overlays/verdin-imx8mp_hmp_overlay.dts?h=toradex_5.15-2.1.x-imx

My understanding of the Remote Proc driver is that it locates the CM7 code in RAM and inspects it expecting certain information to appear there that allows it to complete the hook ups for RPMsg.

16

@davidkhess I don’t think so.
You can see that the default HMP overlay has the following memory regions:

&{/} {
	imx8mp-cm7 {
		...
		memory-region = <&vdevbuffer>, <&vdev0vring0>, <&vdev0vring1>, <&rsc_table>, <&m7_reserved>;
		...
	};
};

...


&resmem {
	...

	m7_reserved: m7@0x80000000 {
		no-map;
		reg = <0 0x80000000 0 0x1000000>;
	};

	vdev0vring0: vdev0vring0@55000000 {
		no-map;
		reg = <0 0x55000000 0 0x8000>;
	};

	vdev0vring1: vdev0vring1@55008000 {
		no-map;
		reg = <0 0x55008000 0 0x8000>;
	};

	vdevbuffer: vdevbuffer@55400000 {
		compatible = "shared-dma-pool";
		no-map;
		reg = <0 0x55400000 0 0x100000>;
	};

	rsc_table: rsc_table@550ff000 {
		no-map;
		reg = <0 0x550ff000 0 0x1000>;
	};
};

and so you can see that m7_reserved is in DDR region and it matches the rfegion of the NXP linker file.
I think that the problem is how to specify the Program Coiunter value from remoteproc.
From u-boot, there is the following command

m4boot=load mmc 0:1 ${loadaddr} 0x80000000; cp.b ${loadaddr} 0x80000000 10000; bootaux 0x80000000;

and you can see that the DDR address is used in two contexts:

  • loadaddr i.e., where to load the firmware
  • bootaux i.e., where to set the PC so that it can boot
    or I imagine so.
    In this example they’re the same because the interrupt vector table is at the beginning of the firmware region, but the vector table could be relocated to a different region (if needed). amnd in this case, load address is different from boot address.

Or, this is what I imagine from my experience of standalone Cortex-M devices.

17

Hi @vix !

Two questions here:

  • Did you try to use TCM instead? If you do so, does remoteproc work?
  • Could you please share what you specifically did to build/link it for DDR?
18

My firmware is too large and doesn’t fit into TCM.
This is the reason why I can’t use TCM only.

I used the linker file provided in NXP examples for DDR. As an example this one.
The examples from NXP provide different linker files for the several memory regions:

  • TCM
  • external QSPI flash
  • external DDR
19

I see. But if you try with something that fits inside the TCM (like a hello world or alike) and compile it with the TCM linker script, does remoteproc works as expected?

It is known that MCUXpresso has linker scripts for different memory regions. I wanted to know specifically what you used. From this I understood that you used the linker script for DDR from MCUXpresso without modifying it. Is that right?

Best regards,

20

I didn’t try, because I trust Toradex and I think this is exactly the solution tested by Toradex.

Only one minor change with an additonal section to store a buffer

  /* The named section ram_shmem goes first into m_data */
  .ram_shmem 0x80200000 :
  {
    KEEP(*(.ram_shmem)) /* keep my variable even if not referenced */
  } > m_data
21

Hello @henrique.tx
I can add one more issue :worried:
I don’t know kif this is the reason, but now the size of my .elf file is 1.2 MB (and the .bin is larger than 3MB).
I have the firmware properly loaded and executed from u-boot.
And so I should be able to use remoteproc to stop M7 core, load the same fimrware (.elf) and start it again.
If I do the above steps I got the following errors:

[  169.645816] rcu: INFO: rcu_preempt self-detected stall on CPU
[  169.651580] rcu:     0-...!: (2100 ticks this GP) idle=b8b/1/0x4000000000000008 softirq=7470/7470 fqs=0 
[  169.660809] rcu: rcu_preempt kthread timer wakeup didn't happen for 2100 jiffies! g6333 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
[  169.672112] rcu:     Possible timer handling issue on cpu=0 timer-softirq=1199
[  169.679071] rcu: rcu_preempt kthread starved for 2103 jiffies! g6333 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
[  169.689419] rcu:     Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
[  169.698547] rcu: RCU grace-period kthread stack dump:
[  169.703673] rcu: Stack dump where RCU GP kthread last ran:
[  196.005817] watchdog: BUG: soft lockup - CPU#0 stuck for 45s! [swapper/0:0]
[  196.013168] Kernel panic - not syncing: softlockup: hung tasks
[  196.019002] SMP: stopping secondary CPUs
[  196.023216] Kernel Offset: disabled
[  196.026703] CPU features: 0x0,00002001,20000846
[  196.031232] Memory Limit: none
[  196.034288] Rebooting in 5 seconds..

Do you have any idea on why this happens?
The size of the firmware?