RPMSG on TorizonCore 6 with Verdin iMX8M-Mini

Hi @hfranco.tx
sorry for the late answer.

Now the overlay is correct and I was able to load the firmware on Cortex-M from Linux (remoteproc). bootaux approach is a little bit confusing to me (I don’t understand ext4ls commands) and I’ve never been able to use it.

When the firmware is started with echo start > state I see that ttyRPMSG30 is created.
Then the output from Cortex-M debug UART shows

RPMSG String Echo FreeRTOS RTOS API Demo...

Nameservice sent, ready for incoming messages...
Get Message From Master Side : "hello world!" [len : 12]

First qauestion: who sends “hello world!” to ttyRPMSG30 from the Linux side?
More or less after 20 seconds doing nothing, I see that the iMX8M-Mini reboots (I don’t know why).
Do you have anhy idea of the reason?

Hi @vix,

Good to know that now it works! The bootaux command to launch the Cortex M4 through u-boot was added to our latest version. If you want, you can test it on nightly builds, but only for testing. The best approach is to wait for a more stable release with this feature enabled. For more information about the ext4 commands, you can check this guide: High performance, low power Embedded Computing Systems | Toradex Developer Center.

The TTY driver from NXP sends the “hello world” message, when you load it with the modprobe command, for example. As soon as the driver is loaded and it detected the RPMSG channel, the Linux driver sends this message. You can check the source code here.

...
/* this needs to be less then (RPMSG_BUF_SIZE - sizeof(struct rpmsg_hdr)) */
#define RPMSG_MAX_SIZE		256
#define MSG		"hello world!"
...

I haven’t seen this behavior. As you’re using TorizonCore 6, can you send me the output of these two commands?

# sudo tdx-info
# sudo tdx-info -dt

This will output information about your module, which will help me to check the versions and try to reproduce it on my side. Thanks!

One more thing I would like to mention, after you opened this thread we added the overlay to our repository, so in our nightly builds the HMP overlay will be inside our image by default, from now on. As I said before, you can test it, but it’s better to wait for a more stable build. You can check the source code here: verdin-imx8mm_hmp_overlay.dts « overlays - device-tree-overlays.git - Sources for Device Tree Overlays

One thing to mention, the

&mu {
	status = "okay";
};

node was removed since the mailbox is already enabled. But this won’t be a problem for you, enabling again won’t cause any issues.

Best Regards,
Hiago.

Hi @hfranco.tx

here is the output from sudo tdx-info

Software summary
------------------------------------------------------------
Bootloader:               U-Boot
Kernel version:           5.15.77-6.1.0-devel+git.a8d2c55c6ae7 #1-TorizonCore SMP PREEMPT Wed Nov 30 05:37:42 -03 2022
Kernel command line:      root=LABEL=otaroot rootfstype=ext4 quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3 ostree=/ostree/boot.1/torizon/d499c845f87aefa8a06a05cbaf558465d77b88a4a65745a7bfd33db07acf1218/0
Distro name:              NAME="TorizonCore"
Distro version:           VERSION_ID=6.1.0-devel-20221222151753-build.0
Hostname:                 verdin-imx8mm-06827409
------------------------------------------------------------

Hardware info
------------------------------------------------------------
HW model:                 Toradex Verdin iMX8M Mini on Verdin Development Board
Toradex version:          0057 V1.1A
Serial number:            06827409
Processor arch:           aarch64
------------------------------------------------------------

and here is the output from sudo tdx-info -dt

Device tree
------------------------------------------------------------
Device tree enabled:      imx8mm-verdin-nonwifi-dev.dtb
Compatible string:        toradex,verdin-imx8mm-nonwifi-devtoradex,verdin-imx8mm-nonwifitoradex,verdin-imx8mmfsl,imx8mm
Device trees available:   
                          imx8mm-verdin-nonwifi-dahlia.dtb
                          imx8mm-verdin-nonwifi-dev.dtb
                          imx8mm-verdin-wifi-dahlia.dtb
                          imx8mm-verdin-wifi-dev.dtb
------------------------------------------------------------

Device tree overlays
------------------------------------------------------------
Overlays enabled:         fdt_overlays=verdin-imx8mm_dsi-to-hdmi_overlay.dtbo verdin-imx8mm_spidev_overlay.dtbo verdin-imx8mm_hmp_overlay.dtbo
Overlays available:
                          display-dpi-lt170410_overlay.dtbo
                          display-edt5.7_overlay.dtbo
                          display-edt7_overlay.dtbo
                          display-fullhd_overlay.dtbo
                          display-lt161010_overlay.dtbo
                          display-lt170410_overlay.dtbo
                          display-vga_overlay.dtbo
                          verdin-imx8mm_20mhz_can1.dtbo
                          verdin-imx8mm_disable_can1.dtbo
                          verdin-imx8mm_dsi-to-hdmi_overlay.dtbo
                          verdin-imx8mm_dsi-to-lvds_panel-cap-touch-10inch-lvds_overlay.dtbo
                          verdin-imx8mm_hmp_overlay.dtbo
                          verdin-imx8mm_ov5640_overlay.dtbo
                          verdin-imx8mm_spidev_overlay.dtbo
------------------------------------------------------------
1 Like

Hi @vix,

Thank you, I will get back to you once I have something.

Best Regards,
Hiago.

Hi @vix,

Unfortunately, I couldn’t reproduce the reboot issue. Here, the code is running for 2 hours almost and nothing happened. Here are the commands that I ran:

$ sudo mount -o remount,rw /usr/
$ sudo mv rpmsg_lite_str_echo_rtos_imxcm4.elf /lib/firmware/
$ cd /sys/class/remoteproc/remoteproc0
$ sudo -s
# echo rpmsg_lite_str_echo_rtos_imxcm4.elf > firmware
# echo start > state
# modprobe imx_rpmsg_tty
# exit
exit
$ echo test > /dev/ttyRPMSG30

First I had to remount the /usr/ to copy my firmware since TorizonCore is read-only inside /usr/. Next, copied the firmware and had to log in as root to access the remoteproc framework. Everything worked as expected:

RPMSG String Echo FreeRTOS RTOS API Demo...

Nameservice sent, ready for incoming messages...
Get Message From Master Side : "hello world!" [len : 12]
Get Message From Master Side : "test" [len : 4]
Get New Line From Master Side
Get Message From Master Side : "test" [len : 4]
Get New Line From Master Side
Get Message From Master Side : "test" [len : 4]
Get New Line From Master Side
Get Message From Master Side : "test" [len : 4]
Get New Line From Master Side

Did you do something different? Or there is something enabled in your image that might be causing this issue?

I would recommend you test in a nightly build, to check if the problem still occurs, like this one: Toradex Download Links (Torizon, Linux BSP, WinCE and Partner Demos) | Toradex Developer Center

Can you also send me your dmesg log?

Let me know if there is something different that I can test on my side.

Best Regards,
Hiago.

Hello @hfranco.tx
I upgraded TorizonCore to torizon-core-docker-verdin-imx8mm-Tezi_6.1.0-devel-20230117+build.150.tar but now something is broken with the tty driver.
When I start the firmware on the Cortex-M, dmesg | grep -i -E "(rpmsg|rproc)" gives

root@verdin-imx8mm-06827409:/sys/class/remoteproc/remoteproc0# dmesg | grep -i -E "(rpmsg|rproc)"
[    0.030073] imx rpmsg driver is registered.
[    1.279191] remoteproc remoteproc0: imx-rproc is available
[  954.228159] remoteproc remoteproc0: powering up imx-rproc
[  954.234613] remoteproc remoteproc0: Booting fw image rpmsg_lite_str_echo_rtos_imxcm4.elf, size 428152
[  954.750557] virtio_rpmsg_bus virtio0: rpmsg host is online
[  954.750621] remoteproc remoteproc0: remote processor imx-rproc is now up
[  954.753843] virtio_rpmsg_bus virtio0: creating channel rpmsg-virtual-tty-channel-1 addr 0x1e

Compared to what happened with older images (as in your messgae above), all the part relayted to tty is missing.
As a matter of fact:

  • /dev/ttyRPMSG30 is not created
  • debug from Cortex-M UART gives
RPMSG String Echo FreeRTOS RTOS API Demo...

Nameservice sent, ready for incoming messages...

but no more

Get Message From Master Side : "hello world!" [len : 12]

and you explained that "hello world" comes from TY driver from NXP.

Can you double check, please?

Hello @hfranco.tx
my fault.
I forgot to load the kernel module with

sudo modprobe imx_rpmsg_tty

So, the example works (even if its behavior is not consistent: sometimes it reboots, sometimes it hangs, …)
Need investigation from my side

Hi @vix,

Sorry for the delay. I’m glad it works now. Unfortunately, I couldn’t reproduce the issue on my side, let me know if you find something or how I can help you debug this issue.

Best Regards,
Hiago.

Hi @hfranco.tx
one thing that could be useful to me is undesrtanding which is your “test setup”.
I have:

  • on my Linux development machine, screen attached to dev/ttyUSB2 so that I can se the debug messages from Cortex-M firmware
  • on my Linux development machine, screen attached to dev/ttyUSB3 so that I can se the debug messages from Cortex-A Linux
  • on the Verdin, echo xxxxx > dev/ttyRPMSG30 to send strings to Cortex-M (I can see messages from Cortex-M debug UART dev/ttuUSB2)

I cannot see in any way the messages echoed back from Cortex-M to dev/ttyRPMSG30 (Torizon doesn’t have neither screen, nor minicom, nor picocom, …).
How do you send strings to Cortex-M (from Cortex-A) and see what it sends you back?

Hi @vix,

On my side, I’m using the Dahlia carrier board (sometimes I also use the development board, but I don’t think it matters here) with the Verdin Mini and the serial cable connected to my computer (X18 USB type C on the Dahlia carrier board), so I can see both linux terminal and the UART from the M4.

I use picocom with a baud rate of 115200 to /dev/ttyUSB3 and /dev/ttyUSB2 to check the messages.

picocom -b 115200 /dev/ttyUSB3
picocom -b 115200 /dev/ttyUSB2

Same here, I use the “echo” command to test the RPMSG. This week I’ve also tested a container with python to write to /dev/ttyRPMSG30, if you want to check:

After running the echo command on linux, can’t you see any messages on the /dev/ttyUSB2?

Best Regards,
Hiago.

Hi @hfranco.tx

After running the echo command on linux, can’t you see any messages on the /dev/ttyUSB2?

Yes, I see on /dev/ttyUSB2 that Cortex-M receives the message
Get Message From Master Side : ...
But, then, Cortex-M echoes back the same message to /dev/ttyRPMSG30 and I’m not able to see it from the Linux side.
If I run cat /dev/ttyRPMSG30 I see a lot of empty lines and some messages, and I’m not sure this is ok.

EDIT:
I tried to deploy a python container as you did, but I use ApolloX extension and it fails for the same reason as the node.js sample.
Is there any document describing how to build and deploy python container with command line?

Hi @vix,

I could reproduce this issue. I think is caused by the “cat” itself and how it read the bytes from the buffer. I believe that “cat” reads one byte at a time and causes the buffer to be refreshed each time, causing an infinite loop.

Instead, I created a python script that writes to the buffer, waits a bit (I chose 1 second to make things easier), and then reads the echo message and it works as expected.

#!python3

import serial
from time import sleep

def main():

    for i in range(10):
        try:
            with serial.Serial("/dev/ttyRPMSG30", 115200, timeout=1) as ser:
                ser.write("Toradex Test! {}".format(i).encode())
                sleep(1)
                print("Bytes waiting in buffer: " + str(ser.in_waiting))
                read = ser.read(ser.in_waiting)
                print("Msg received: " + str(read))
                print()
        except Exception as err:
            print(err)
    
if __name__ == "__main__":
    main()

We can see all the message on the M4:

Get Message From Master Side : "Toradex Test!" [len : 13]
Get Message From Master Side : "Toradex Test!" [len : 13]
Get Message From Master Side : "Toradex Test! 0" [len : 15]
Get Message From Master Side : "Toradex Test! 1" [len : 15]
Get Message From Master Side : "Toradex Test! 2" [len : 15]
Get Message From Master Side : "Toradex Test! 3" [len : 15]
Get Message From Master Side : "Toradex Test! 4" [len : 15]
Get Message From Master Side : "Toradex Test! 5" [len : 15]
Get Message From Master Side : "Toradex Test! 6" [len : 15]
Get Message From Master Side : "Toradex Test! 7" [len : 15]
Get Message From Master Side : "Toradex Test! 8" [len : 15]
Get Message From Master Side : "Toradex Test! 9" [len : 15]

And then it echos back the message:

Bytes waiting in buffer: 15
Msg received: b'Toradex Test! 7'

Bytes waiting in buffer: 15
Msg received: b'Toradex Test! 8'

Bytes waiting in buffer: 15
Msg received: b'Toradex Test! 9'

Note on my code that I first check how many bytes are waiting in the buffer and then reads all the bytes at the same time, so the whole buffer gets consumed.

I’m also using ApolloX here. I just create a python sample from the template and added my code to main.py. added pyserial to the requirements file and shared the device under the docker compose file:

version: "3.9"
services:
  hmp-debug:
    build:
      context: .
      dockerfile: Dockerfile.debug
    image: ${LOCAL_REGISTRY}:5002/hmp-debug:${TAG}
    ports:
      - 6502:6502
      - 6512:6512
    devices:
      - "/dev/ttyRPMSG30:/dev/ttyRPMSG30"
    
  hmp:
    build:
      context: .
      dockerfile: Dockerfile
    image: ${DOCKER_LOGIN}/hmp:${TAG}

I’m using the debug version here. Next, I deployed my container by pressing F5 on the debug tab.

Let me know if that works for you as well.

Best Regards,
Hiago.

@vix,

one thing that I forgot to mention, I’m using the verdin-imx8mm_hmp_overlay.dts.

This a new overlay that was added into our repository a couple of weeks ago, so in order to get this automatically you will have to install TorizonCore 6.2 from the nonstable releases.

If you want to use it with the 6.1, you can also copy the source code and compile it yourself. This should work as well, as 6.2 is not stable yet.

Hi @hfranco.tx
I’m going to test the container after this issue is solved.
I’m going to upgrade TorizonCore to 6.2 since I want to stay aligned in this stage of my development.
You wrote about a file named verdin-imx8mp_hmp_overlay.dts (and usually imx8mp stays for iMX8M-Plus), but the link points to a file which is named verdin-imx8mm_hmp_overlay.dts (and usually imx8mm stays for iMX8M-Mini).
Is the file the same for Mini and Plus variants of the Verdin?

Hi @vix,

Sorry, it was a typo from my side, I meant verdin-imx8mm. We currently don’t have an overlay for the Plus.

Ok, let me know if you need anything.

Best Regards,
Hiago.

Hi @hfranco.tx
in the next few weeks I’m going to move my focus on the Plus (I’m in touch with sales office to evaluate the best solution), because the new requirements from the management involve two ethernet interfaces (and the HDMI).
Do you see any issue in having remoteproc/rpmsg working on Torizon for the Plus?
Let me know, because this is essential for the project.
Thanks

Hi @vix,

We have an overlay that is under review right now for the Plus, it should be merged on the device-tree repository in a few weeks. I can share it with you if you need it.

RPMSG works with the Verdin Plus, but to see the messages we need to disable the UART4 because the cortex M7 uses these pins. However, on the Plus, UART4 is also used for WiFi/BT control, as stated on its datasheet:

The Wi-Fi module is connected over a 4-bit SDIO interface with the i.MX 8M Plus SoC. It uses the
USDHC1 instance of the SoC. For Bluetooth Audio, an additional I2S interface is available between
the SoC and the Wi-Fi module, which uses the SAI5 instance of the SoC. The Bluetooth UART
signals (RX, TX, RTS, CTS) signals are connected to an alternate location of the UART4 of the SoC.
This interface can only be used if the Verdin standard UART_4 (M7 debug port) is not in use. It is
either possible to use the M7 debug UART or the Bluetooth UART simultaneously

So by using the debug port of the M7, you will lose this functionality. However, this port is usually only used for debugging, once you have your code running inside the M7 you can disable this debugging port or you can route the debug port of the M7 to a different UART that is available (although I never tried this option).

Now a note about the remote proc, currently it’s not working with our BSP 6/TC 6, only BSP 5/TC 5. We already have a ticket internally to investigate this issue, although I don’t have any updates right now. This doesn’t exclude the option of using the u-boot to load the binary, of course, but if you need the remote proc feature this can be an issue.

Please check this ticket for more info: Zephyr Remoteproc (verdin-imx8mp on verdin-dev) - #6 by hfranco.tx

Also, there is this ticket open with NXP: https://community.nxp.com/t5/i-MX-Processors/Linux-Remoteproc-on-i-MX8MP/m-p/1587608

Let me know if you have any questions.

Best Regards,
Hiago.

Hio @hfranco.tx
can you confirm that the overlay for the Plus has been merged in the last available nightly 6.2.0-devel-20230406+build.232.container at the moment?

Do you mean that we need to disable UART4 on the TorizonCore side? Or something different?
Is UART4 disabled in the standard device tree included with TorizonCore?
If not, how can I disable it?

Hi @vix,

The overlay for the Plus was not merged yet, only for the Mini. I’ll keep you updated and can help with your overlay if needed.

In order to see the messages coming from Verdin Cortex-M7, this peripheral needs to be disabled from Linux. Can you share which hardware version are you using? Is it Wifi BT or the version without wifi?

Best Regards,
Hiago.

I use PN: 0058 (with WiFi).
Since this topic is for the Mini, I’m going to open a new topic to discuss about the overlay for the Plus.