RPMSG on iMX8M-Plus having error "imx_rproc_kick: failed"

[aditya.yantra]

Hello,

I am having an issue with RPMSG on Verdin iMX8M-Plus.

• Yocto Version: Kirkstone
• BSP: 6.0.0 Quaterly Release
• RPMSG Device Tree Customization is based on following article:
  RPMSG does not work on IMX8M-Plus - Technical Support - Toradex Community

While running the Cortex-M7 binary, I am getting this error: imx-rproc imx8mp-cm7: imx_rproc_kick: failed (1, err:-62)
Timing of this error is very random, so I am unable to debug this error.

Can anyone please help me with identifying the reason of this error and find a solution?

Thanks,
Aditya

[hfranco.tx]

Hi @aditya.yantra,

For BSP 6, please check my answer to this post:

How do I access both M7 and A53 consoles simultaneously on Verdin plus module?

Hi @mhwm, I believe I replied to you by email with all the information. But I will also write the answer here so others can benefit from this information. I know why the kernel is panicking at this point, let me try to explain here, and let me know if you have any questions. For BSP 6, the following code was added to u-boot downstream source: u-boot-toradex.git - U-Boot bootloader for Apalis and Colibri modules Although this code was committed by Toradex, this was copied from NXP u-boot dow…

The overlay there should work for the demo you’re trying to run. Please let me know if you have any questions.

Best Regards,
Hiago.

[aditya.yantra]

@hfranco.tx

Thanks for the reply. I will check your device tree overlay and update you soon.

In my current device tree, I2C3 node is already disabled.
But we are using I2C3 in Cortex-M7 firmware. Might this be a cause of imx_rproc error?

Reference Note:
We are facing another issue. I think it is related to this problem. So, I am sharing the details in this thread:
I2C3 sometimes struck in the cortex-m7 firmware while sending data. We have tested this firmware by pausing boot process in uboot (without starting linux), but we are facing the issue.
May be it is the same problem.

[hfranco.tx]

Hi @aditya.yantra,

Please check this thread too: Support of rpmsg in TorizonCore on Verdin iMX8MP - #5 by hfranco.tx. I’ve added some information about the overlay with the M Plus there too.

@aditya.yantra

In my current device tree, I2C3 node is already disabled.
But we are using I2C3 in Cortex-M7 firmware. Might this be a cause of imx_rproc error?

We’re still investigating the imx_rproc problem, but I would say it shouldn’t be a problem for you since this is already disabled on the device tree.

Best Regards,
Hiago.

[aditya.yantra]

Hi @hfranco.tx,

I have compared your device tree overlay with my current custom device tree. It has no difference.
So, I guess that wouldn’t be issue.

imx8mp-verdin-yantra.dts (29.0 KB)

I2C3 is disabled in device tree and also I have changed it to I2C4 in Cortex-M7 firmware.
But I am still facing the issue of “imx_rproc_kick: failed".
So, there might be some other issue with remoteproc.

Secondly, I am not using remoteproc to run Cortex-M7 binary from Linux. I am using only J-Link or U-Boot to load the binary and run. In this case, remoteproc is not used.

(1) I have noticed that remoteproc is connected somehow with Resource Table (which contains data of RPMSG buffers and memory).

Following function is used in Cortex-M7 firmware:
copyResourceTable();

Will there be a problem ?

(2) I have also studied terminal output of linux. There I found the reference of remoteproc.

So, I guess remoteproc is necessary for RPMSG to work.

If you have any solution regarding how to use RPMSG without remoteproc, please share it with me.
Or do we have to wait till the remoteproc problem is solved?

Please also find attached Linux Terminal outputs for the reference:

• In BOOTAUX, I have loaded the binary using bootaux command of U-Boot.
• In JLINK, I have loaded the binary with the help of Seggar J-Link.

BOOTAUX.txt (41.3 KB)
JLINK.txt (40.6 KB)

Thanks,
Aditya

[hfranco.tx]

Hi @aditya.yantra,

Please check my overlay that I uploadeed here

Support of rpmsg in TorizonCore on Verdin iMX8MP

Hi @MadDoc, Sorry for the delay, it took me a while to debug what was going on and I have some points to share with you. Please let me know if you have any questions. This is my setup, it’s using TorizonCore 6.1.0-devel-202212-build.4 as you mentioned: torizon@verdin-imx8mp-07174398:~$ sudo tdx-info Password: Software summary ------------------------------------------------------------ Bootloader: U-Boot Kernel version: 5.15.77-6.1.0-devel+git.a8d2c55c6ae7 #1-TorizonC…

and check if that works in your module, without your device tree. Then, after checking if it works, you can enable your device tree and compare.

Here is the steps that I’ve followed:

1. Loaded the rpmsg_lite_str_echo_rtos.bin:

Verdin iMX8MP # ext4load mmc 2:2 ${loadaddr} /home/root/rpmsg_lite_str_echo_rtos.bin
19868 bytes read in 1 ms (18.9 MiB/s)
Verdin iMX8MP # cp.b ${loadaddr} 0x7e0000 20000
Verdin iMX8MP # dcache flush
Verdin iMX8MP # bootaux 0x7e0000
## Starting auxiliary core stack = 0x20020000, pc = 0x000004DD...

The binary is loaded and waiting. When you boot the Linux kernel, you can see the message appearing “Nameservice sent, ready for incoming messages…”:

$ picocom -b 115200 /dev/ttyUSB2
picocom v3.1

port is        : /dev/ttyUSB2
flowcontrol    : none
baudrate is    : 115200
parity is      : none
databits are   : 8
stopbits are   : 1
escape is      : C-a
local echo is  : no
noinit is      : no
noreset is     : no
hangup is      : no
nolock is      : no
send_cmd is    : sz -vv
receive_cmd is : rz -vv -E
imap is        :
omap is        :
emap is        : crcrlf,delbs,
logfile is     : none
initstring     : none
exit_after is  : not set
exit is        : no

Type [C-a] [C-h] to see available commands
Terminal ready

RPMSG String Echo FreeRTOS RTOS API Demo...

Nameservice sent, ready for incoming messages...

After that, I loaded the kernel module:

root@verdin-imx8mp-07174398:~# modprobe imx_rpmsg_tty
[   72.367176] imx_rpmsg_tty virtio0.rpmsg-virtual-tty-channel-1.-1.30: new channel: 0x400 -> 0x1e!
[   72.376179] Install rpmsg tty driver!

Then I can see the hello world showing up. The RPMsg is working.

RPMSG String Echo FreeRTOS RTOS API Demo...

Nameservice sent, ready for incoming messages...
Get Message From Master Side : "hello world!" [len : 12]

Please let me know if you can reproduce the same steps.

@aditya.yantra

(1) I have noticed that remoteproc is connected somehow with Resource Table (which contains data of RPMSG buffers and memory).

Following function is used in Cortex-M7 firmware:
copyResourceTable();

Will there be a problem ?

You’re correct, but in this case we’re already accessing the resource table in the device, so this shouldn’t be a problem.

@aditya.yantra

So, I guess remoteproc is necessary for RPMSG to work.

You’re seeing the remoteproc because the remoteproc driver is already enabled by default in our Kernel, however this is not required for RPMsg to work.

Please, test this overlay with the default device tree and let me know if it works.

Best Regards,
Hiago.

[aditya.yantra]

Hi @hfranco.tx ,

Thanks for the reply. I will test your overlay and update you soon.

Please also note the following important details:

I haven’t described my issue and test cases in detail. So, I am sharing now:

(1) Understanding the Problem:

First, I have customized Device Tree with the help of this article:
RPMSG does not work on IMX8M-Plus - Technical Support - Toradex Community
Then, I have tested RPMSG on BSP 6.0.0. It works very well.

Here is the screenshot of working “RPMSG string echo example”.
(Sorry that the code of M7 is bit customized. I have this only screenshot available for now.)
Left side (Linux Terminal) is the log of messages which is received from Cortex -M7 and on the right side (M7 Terminal), there is a log of messages which is received from Linux.

It was working fine, and I have tested it by sending hundreds of messages in both directions.

Secondly, we are using Qt based software on Linux side for our application.
So, we have attached the /dev/ttyRPMSG30 port to QtSerial.
For this application, we have tested RPMSG by sending message from Qt software to Cortex-M7 firmware and it was working.

Now, problem came when we were testing our Qt software with M7 firmware.
RPMSG was initializing and working well. But after some time, we were getting error of:
imx-rproc imx8mp-cm7: imx_rproc_kick: failed (1, err:-62)

So, I have to power off and restart the board. Again, RPMSG works for sometime and it stops working after transferring a few messages.

Here is the screenshot of the error.
Left side terminal (COM6) is the debug output of M7 firmware. You can see that nameservice is enabled and after that we have transferred a lot of messages between both M7 and Linux.
Right side (COM7) is Linux debug output where we can see the rproc error which we have got after some time. (Note: “modprobe imx_rpmsg_tty” is run from Qt software. So, this command is not showing in debug output.)

(2) Test Cases:

• First, I thought it was conflicting with I2C3. So, I have tested it many ways as mentioned in previous posts. As the result, I found that this might not be the problem as I2C3 is disabled from device tree.

• From the NXP SDK’s M7 RPMSG example, I have tested RPMSG by transferring hundreds of strings with the delay of 1 millisecond. It was working there. But as soon as I try Qt software with whole M7 firmware, it shows error after some time.
  I am only guessing that maybe is it conflicting with other peripherals or so? I have attached many peripherals in my M7 firmware, and it is a large code. However, probability of this type of conflict is near to zero from my point of view.

• I am not using remoteproc to load the firmware but only using UBOOT or JLINK. You can find BOOTAUX.txt & JLINK.txt log files of Linux in previous post where you can also see that RPMSG is initializing and working fine. So, this cause is also ruled out.

• I will also test your overlay and update you. But there is the problem that, I have to test it with my firmware and hardware mapping. So, again I will end up with customized device tree.

(3) Conclusion:

RPMSG is working fine usually. But problem comes after some time. It automatically disables and we are struck with the error of imx-rproc imx8mp-cm7: imx_rproc_kick: failed (1, err:-62) .

Timing of the error is very random. So, we cannot easily debug or relate it with any error from our code side.

Due to the weird timing pattern of this error, I am guessing that anyhow remoteproc is crasing by unknown reasons and that is why RPMSG is stopped working.

(4) Solution Finding Process:

• One way to solve this error is by understanding reason of this error. I am unable to find data of this error over the internet. Can you describe me what this error is suggesting or what this error code -62 is refering to? Maybe that way, I can debug from my side the cause of this problem.

• If you have any other idea or test cases, please share it with me.

Thanks,
Aditya

[hfranco.tx]

Hi @aditya.yantra,

I apologize for the delay, I was out of the office last week. I would like to thank you for the very detailed answer!

First things first, from my research, the 62 error code that you are seeing means “timer expired”:

#define	ETIME		62	/* Timer expired */

The error numbers from the Linux kernel are defined under include/uapi/asm-generic/errno.h. Please note that the negative value that you’re seeing is probably from a return -ETIME. Searching for this line inside imx_rproc.c, I couldn’t find anything, so it might come from a different file.

I couldn’t reproduce the error you’re seeing on my side. Can you please share with me how you’re compiling your code? Also, have you had the chance to test the device tree overlay that I mentioned?

One to debug it would be by enabling the kernel dynamic debugging, by running

# echo "file imx_rproc.c +p" > /sys/kernel/debug/dynamic_debug/control

inside your module. This should print some debug messages that you will be able to see in your dmesg:

# dmesg | grep -i -E "(rproc|rpmsg)"

Or you can also check for system calls by using strace:

# strace modprobe imx_rpmsg_tty

as an example.

Also, I would like to mention that we uploaded the verdin-imx8mm_hmp_overlay.dts to our repository (please note this is for mini, we are still working on the plus). This might help you with some insights into your overlays.

Best Regards,
Hiago.