/dev/ttyRPMSG30 not created in Torizon OS 7 on Colibri iMX8QXP

Hi @rudhi.tx ,

As far as the UART goes, I changed the patch to use UART B, which works. So I am convinced that the M4 binary is being loaded and run correctly. I don’t think it’s worth looking into this any further - the important thing is that the M4 code is running.

Then, without touching the U-Boot settings, I’ve replaced the bin file with yours. Just in case there’s any problem I’ve added the colibri-imx8x_disable-cm40-uart_overlay.dtbo.

However, it’s still not working :frowning:

torizon@colibri-imx8x-06996015:~$ dmesg | grep -i rpms
[    0.046741] imx rpmsg driver is registered.
torizon@colibri-imx8x-06996015:~$ sudo modprobe imx_rpmsg_tty
Password:
torizon@colibri-imx8x-06996015:~$ ls /dev/ | grep -i rpms

U-Boot settings :-

torizon@colibri-imx8x-06996015:~$ sudo fw_printenv bootcmd
bootcmd=run m4boot_0; run distro_bootcmd;
torizon@colibri-imx8x-06996015:~$ sudo fw_printenv m4boot_0
m4boot_0=run loadm4image_0; dcache flush; bootaux ${loadaddr} 0
torizon@colibri-imx8x-06996015:~$ sudo fw_printenv loadm4image_0
loadm4image_0=load mmc ${mmcdev}:${mmcpart} ${loadaddr} ${m4_0_image}
torizon@colibri-imx8x-06996015:~$ sudo fw_printenv m4_0_image
m4_0_image=/ostree/deploy/torizon/var/rootdirs/home/torizon/rpmsg_lite_str_echo_rtos.bin

Overlays :-

torizon@colibri-imx8x-06996015:~$ ls /proc/device-tree/chosen/overlays
colibri-imx8x_disable-cm40-uart_overlay.dtbo  colibri-imx8x_hmp_overlay.dtbo  colibri-imx8x_vga-640x480_overlay.dtbo  name
torizon@colibri-imx8x-06996015:~$ cat /proc/device-tree/chosen/overlays/colibri-imx8x_disable-cm40-uart_overlay.dtbo
0torizon@colibri-imx8x-06996015:~$ cat /proc/device-tree/chosen/overlays/colibri-imx8x_hmp_overlay.dtbo
0torizon@colibri-imx8x-06996015:~$ cat /proc/device-tree/chosen/overlays/colibri-imx8x_vga-640x480_overlay.dtbo
0torizon@colibri-imx8x-06996015:~$

modprobe :-

0torizon@colibri-imx8x-06996015:~$ lsmod | grep -i rpmsg
imx_rpmsg_tty          16384  0

So I’m pretty convinced that the problem is on my Linux side. We must have something different.

My tdx-info is the same as above (post #36).

I suppose the next step is to try and diagnose modprobe.

Tim

I didn’t get anywhere useful with modprobe. It doesn’t appear to generate any error message with -v and nothing new appears in dmesg.

Also, I dug out a dirty colibri evaluation board from the back of a cupboard, and tried the same module in there. But with the same result (well, the text transmitted on the serial port was garbled, but maybe this is due to 3.3V ?). Seeing the output means that the application is running on the M4, and that’s all we need to know.

I would be very surprised if the Iris board is the difference which is causing the RPMsg to fail.

Can you double-check your OS details ?

Tim

Please, @rudhi.tx , we need this working or we’ll have to continue using the ugly Yocto instead of the shining Torizon.

We have already spend a significant effort on this and feel we’re close to the solution, but we need your support.

Thanks in advance.

Hi @Timo and @vic,
I’m seeing some differences between what you get on your OS and mine. I’m working on an iMX8DX with a Yocto based on the minimal Toradex Linux image so there is quite a few differences but just in case one of these differences can steer you in the right direction, here goes :
Just like you, I’m using UART B and managed to make that rpmsg_lite_str_echo_rtos example work:

dmesg :

//With M4 firmware running
root@colibri-imx8x-15508241:~# dmesg | grep -i rpms
[    0.276825] imx rpmsg driver is registered.
[    2.041545] imx-rpmsg 90000000.rpmsg: assigned reserved memory node vdevbuffer@90400000
[    2.053353] virtio_rpmsg_bus virtio0: rpmsg host is online
[    2.060457] virtio_rpmsg_bus virtio0: creating channel rpmsg-i2c-channel addr 0x1
[    2.070979] virtio_rpmsg_bus virtio1: rpmsg host is online
[    2.071074] i2c-rpmsg virtio0.rpmsg-i2c-channel.-1.1: new channel: 0x400 -> 0x1!
[    2.091617] virtio_rpmsg_bus virtio1: creating channel rpmsg-openamp-demo-channel addr 0x1e
[    4.036023] imx_rpmsg_tty virtio1.rpmsg-openamp-demo-channel.-1.30: new channel: 0x400 -> 0x1e!
[    4.063485] Install rpmsg tty driver!

//Without the M4 running
root@colibri-imx8x-15508241:~# dmesg | grep -i rpms
[    0.276816] imx rpmsg driver is registered.
[    2.028123] imx-rpmsg 90000000.rpmsg: assigned reserved memory node vdevbuffer@90400000
[    2.060492] virtio_rpmsg_bus virtio0: rpmsg host is online
[    2.092757] virtio_rpmsg_bus virtio1: rpmsg host is online

root@colibri-imx8x-15508241:~# ls /dev | grep -i rpms
rpmsg_ctrl0
rpmsg_ctrl1

U-boot is basically the same for me but modprobe also counts the ctrl and char modules

root@colibri-imx8x-15508241:~# lsmod | grep rpms
rpmsg_ctrl             16384  0
rpmsg_char             16384  1 rpmsg_ctrl
imx_rpmsg_tty          20480  0

From what I can see, your trace misses the “imx-rpmsg 90000000.rpmsg: assigned reserved memory node” part, which would make me wonder if your device tree and overlays have a problem. Maybe try and dump your device tree to see if the hmp overlay is setup correctly ?

The absence of the ctrl and char modules also hint at a different kernel config. So at the very least, if you want to compare with mine :

root@colibri-imx8x-15508241:~# zcat /proc/config.gz | grep RPMSG
CONFIG_KEYBOARD_RPMSG=y
CONFIG_GPIO_IMX_RPMSG=y
# CONFIG_REGULATOR_PF1550_RPMSG is not set
CONFIG_SND_SOC_FSL_RPMSG=m
CONFIG_SND_SOC_IMX_AUDIO_RPMSG=m
CONFIG_SND_SOC_IMX_PCM_RPMSG=m
CONFIG_SND_SOC_IMX_RPMSG=m
CONFIG_SND_SOC_RPMSG_WM8960=m
CONFIG_SND_SOC_RPMSG_WM8960_I2C=m
CONFIG_SND_SOC_RPMSG_AK4497=m
CONFIG_RTC_DRV_IMX_RPMSG=y
# CONFIG_CROS_EC_RPMSG is not set
CONFIG_RPMSG=y
CONFIG_RPMSG_CHAR=m
CONFIG_RPMSG_CTRL=m
CONFIG_RPMSG_NS=y
CONFIG_RPMSG_QCOM_GLINK=y
CONFIG_RPMSG_QCOM_GLINK_RPM=y
CONFIG_RPMSG_VIRTIO=y
CONFIG_HAVE_IMX_RPMSG=y
CONFIG_IMX_RPMSG_PINGPONG=m
CONFIG_IMX_RPMSG_TTY=m
CONFIG_RPMSG_LIFE_CYCLE=y
CONFIG_RPMSG_IIO_PEDOMETER=m

If nothing here has anything to do with your actual problem, the last thing I would think of is to check if the M4 is actually running once Linux boots. I’ve had problems with some BSP version where the M4 would just reset at some point in the booting process. Which means it had time to power up, initialize, send some messages on my UART and then silently die (not really, it actually resets the board, but it’s not evident in the middle of a boot sequence) while I wondered why my rpmsg_tty driver did not link with the firmware.

Edit : Although, your Uboot is already setup to power up the M4 automatically, because of which the issue explained just above would have resulted in a chain of reset instead of an actual complete boot, so maybe scrap that idea if you are actually able to reach Linux.

Good luck !
Tom

Hey @Tom,

Thanks for the hints - I will try and figure out what modules are being loaded, or not. Certainly I am not seeing the assigned reserved memory message in this latest TorizonOS. I did with an ealier version :thinking:.

Is your Yocto build based on a TorizonOS minimal build? We may go that route, although my understanding is that the TorizonOS container version is based on the minimal build anyway.

Cheers,

Tim

Hi @Timo,
No, our Yocto is based on a Toradex minimal Linux distribution:

require recipes-images/images/tdx-reference-minimal-image.bb

It’s also my understanding that Torizon is based on this, with quite a few more tweaks.

Cheers,
Tom

Hi @Timo,

Very sorry about the delay. I have been trying to make it work on an Iris board and I run into the problem of accessing the UART for M4 core. You mentioned you have a working patch to use UARTB. Could you please share that with me?

I think I might be closer to a solution with that.

Hi @rudhi.tx !

Good to have you back :slight_smile:

For now the UART output is useful to show us the m4 binary is running. But I think we’re pretty convinced about that.

The RPMsg TTY should work in any case. Does it not work in your case on the Iris board ? One issue could be the device tree for the Iris, which could include the pins used for the M4 UART. But there is an overlay to disable this which you can add. This should prevent any issue with the linux side, which could interfere perhaps with the code running on the M4.

I will look out my code hack in the meantime. Although it outputs on UARTB correctly, there is a slight concern that there could be interference with the linux side without an overlay to disable linux.

Did you check the OS version you have on your module against mine (above) ?

Many thanks,

Tim

Hi @Timo,

Right now, I have a setup here on my desk with Colibri iMX8X and Iris carrier board. It is the same module I used on the evaluation board - therefore, the same firmware binary is loaded. Here I have ttyRPMSG device being listed:

torizon@colibri-imx8x-14740785:~$ sudo fw_printenv fdtfile
fdtfile=imx8qxp-colibri-iris-v2.dtb

torizon@colibri-imx8x-14740785:~$ ls /proc/device-tree/chosen/overlays
colibri-imx8x_hmp_overlay.dtbo  name

torizon@colibri-imx8x-14740785:~$ ls /dev/ | grep -i rpmsg
rpmsg_ctrl0
rpmsg_ctrl1
ttyRPMSG30

Regarding the Torizon OS version, I have 6.8.2 on my side:
torizon@colibri-imx8x-14740785:~$ sudo tdx-info

Software summary
------------------------------------------------------------
Bootloader:               U-Boot
Kernel version:           5.15.177-6.8.2+git.5a6602cc6c32 #1-TorizonCore SMP PREEMPT Fri Mar 21 16:22:58 UTC 2025
Kernel command line:      root=LABEL=otaroot rootfstype=ext4 quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3 ostree=/0
Distro name:              NAME="TorizonCore"
Distro version:           VERSION_ID=6.8.2-build.30
Distro variant:           VARIANT="Docker"
Hostname:                 colibri-imx8x-14740785
------------------------------------------------------------

Hardware info
------------------------------------------------------------
HW model:                 Toradex Colibri iMX8QXP on Colibri Iris V2 Board
Toradex version:          0038 V1.0E
Serial number:            14740785
Processor arch:           aarch64
------------------------------------------------------------

I can test with 7.3.0 and let you know what I see. Could you please test the binary I sent you with version 6.8.2 as well?

Hi @rudhi.tx,

I will test with 6.8.2 and let you know. The OS was failing to update from Torizon Cloud when I was trying it before - maybe it refuses to downgrade ? Anyway, I should find time today.

Let me know how it goes with 7.3.0 on your side :wink:

Tim

Hi @rudhi.tx again,

Well, with

Software summary
------------------------------------------------------------
Bootloader:               U-Boot
Kernel version:           5.15.177-6.8.2+git.5a6602cc6c32 #1-TorizonCore SMP PREEMPT Fri Mar 21 16:22:58 UTC 2025
Kernel command line:      root=LABEL=otaroot rootfstype=ext4 quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3 ostree=/ostree/boot.1/torizon/0814a081cba6e63f5b6527f0aa65e44ff60eb4474f6ac61583e487765097ef41/0
Distro name:              NAME="TorizonCore"
Distro version:           VERSION_ID=6.8.2-build.30
Distro variant:           VARIANT="Docker"
Hostname:                 colibri-imx8x-06996015
------------------------------------------------------------

Hardware info
------------------------------------------------------------
HW model:                 Toradex Colibri iMX8QXP on Colibri Iris V2 Board
Toradex version:          0038 V1.0D
Serial number:            06996015
Processor arch:           aarch64
------------------------------------------------------------

it is working :-

torizon@colibri-imx8x-06996015:~$ dmesg | grep -i rpms
[    0.064470] imx rpmsg driver is registered.
[    2.634093] imx-rpmsg 90000000.rpmsg: assigned reserved memory node vdevbuffer@90400000
[    2.634772] virtio_rpmsg_bus virtio0: rpmsg host is online
[    2.635372] virtio_rpmsg_bus virtio1: rpmsg host is online
[    2.635407] virtio_rpmsg_bus virtio1: creating channel rpmsg-openamp-demo-channel addr 0x1e
[    2.636239] virtio_rpmsg_bus virtio0: creating channel rpmsg-i2c-channel addr 0x1
[    2.636355] i2c-rpmsg virtio0.rpmsg-i2c-channel.-1.1: new channel: 0x400 -> 0x1!
torizon@colibri-imx8x-06996015:~$ ls /dev | grep -i rpms
rpmsg_ctrl0
rpmsg_ctrl1
torizon@colibri-imx8x-06996015:~$ sudo modprobe imx_rpmsg_tty
Password:
torizon@colibri-imx8x-06996015:~$ ls /dev | grep -i rpms
rpmsg_ctrl0
rpmsg_ctrl1
ttyRPMSG30
torizon@colibri-imx8x-06996015:~$

Glory be praised!

So, sounds as if something got left out of Torizon OS 7 !

Unfortunately iirc there’s a reason why V7 is important. Or maybe I’m thinking about the fact that a while ago hw rendering for Qt libs was not supported yet in v7, when it was on v6 - but that is supported now on v7. Maybe it was something else - I would have to check my notes from last month :-D.

I think we need to log this as a bug in v7, and hope it will be fixed sooner rather than later.

Also, I feel I should re-iterate what was said here, that this should be added to the automated test suite. If we go down the path of container-based OTA updates, then we will be relying on Torizon being tested by Toradex !

Hopefully the fix will be as easy as it was for the Verdin case - and perhaps it could be applied at a common level ?

TIA

Tim

Hello @Timo,

Thanks for sharing your test results.

In the meantime, I also tested on Torizon OS 7.3.0 and could reproduce your issue. After spending so much time on it and investigating, I’m at the conclusion that something is off with the addresses used in the HMP overlay on kernel 6.6.y. Therefore, indeed, this is an issue on our side with the device tree overlay. I am already in touch with our R&D team to find a fix. I will keep working on it and should be able to send you a solution by the end of this week.

Regarding adding an automated test for RPMsg on Torizon - I will also discuss with our test team and let you know of the conclusion.

Hi @rudhi.tx ,

Excellent - great to hear that it’s something consistent, and not some weird sporadic case (I had been getting a bit desperate).

I look forward to hearing about a fix and test :smiley:

Tim

Hi @Timo,

I hope you are doing well. As I mentioned earlier, I escalated the issue to our Torizon R&D team and one of my colleagues got the rpmsg demo working on Torizon OS 7. I haven’t had the time to reproduce it and test again. However, I want to share the FW binary with you, which you can test on Torizon OS 7:
rpmsg_lite_str_echo_rtos.bin (64.7 KB)

Could you please give it a try and let me know how that goes?

He told me the only thing he did are the following to get it working:

  • Used the 14.3 toolchain by modifying fsl_sbrk.c and adding typedef void* caddr_t; at the start of the code due to a compilation issue.
  • Followed our developer pages article on building and loading the binary

Here is his setup:

torizon@colibri-imx8x-06750825:~$ cat /etc/os-release
ID=torizon
NAME="Torizon OS"
VERSION="7.3.0+build.18 (scarthgap)"
VERSION_ID=7.3.0-build.18
VERSION_CODENAME="scarthgap"
PRETTY_NAME="Torizon OS 7.3.0+build.18 (scarthgap)"
CPE_NAME="cpe:/o:openembedded:torizon:7.3.0+build.18"
BUILD_ID="18"
ANSI_COLOR="1;34"
VARIANT="Docker"
torizon@colibri-imx8x-06750825:~$ lsmod | grep -i rpmsg
imx_rpmsg_tty          16384  0
rpmsg_ctrl             12288  0
rpmsg_char             20480  1 rpmsg_ctrl
torizon@colibri-imx8x-06750825:~$ dmesg |grep "rpmsg\|rproc\|virtio"
[    0.047734] imx rpmsg driver is registered.
[    1.576029] remoteproc remoteproc0: imx-rproc is available
[    1.576096] remoteproc remoteproc0: attaching to imx-rproc
[    1.576790] rproc-virtio rproc-virtio.0.auto: assigned reserved memory node vdevbuffer@90400000
[    1.577311] virtio_rpmsg_bus virtio0: rpmsg host is online
[    1.577342] rproc-virtio rproc-virtio.0.auto: registered virtio0 (type 7)
[    1.577360] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node vdevbuffer@90400000
[    1.577831] virtio_rpmsg_bus virtio1: rpmsg host is online
[    1.577892] virtio_rpmsg_bus virtio1: creating channel rpmsg-openamp-demo-channel addr 0x1e
[    1.578023] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
[    1.578036] remoteproc remoteproc0: remote processor imx-rproc is now attached
[    1.582555] virtio_rpmsg_bus virtio0: creating channel rpmsg-i2c-channel addr 0x1
[   74.686000] imx_rpmsg_tty virtio1.rpmsg-openamp-demo-channel.-1.30: new channel: 0x400 -> 0x1e!
[   74.686315] Install rpmsg tty driver!
[   74.787446] imx-rproc imx8x-cm4: imx_rproc_kick: failed (3, err:-62)
torizon@colibri-imx8x-06750825:~$ ls /dev/ | grep -i rpmsg
rpmsg_ctrl0
rpmsg_ctrl1
ttyRPMSG30
torizon@colibri-imx8x-06750825:~$ cat /boot/ostree/torizon-6dc6a4a6e35b2270fd8def445ebf7fe5f3bdd37981f5c5f1715335bca152f655/dtb/overlays.txt
fdt_overlays=colibri-imx8x_disable-cm40-uart_overlay.dtbo colibri-imx8x_hmp_overlay.dtbo

Hi @rudhi.tx !

Sorry for the delay - I was away on holiday, and had to get stuck into other things when I got back.

Anyway, good news! I can confirm it’s working* with this OS version, and also 7.4.0b12 from the (current) monthly releases.

However, the TCloud OS update causes a problem, since it changes the overlays.txt files for both OS versions - that was pretty disconcerting ! I will open a new ticket…

I have also tested an M4 binary built with an earlier toolchain, and can confirm that this does not work after a power cycle (even though it does work after a #reboot). A rpmsg_lite_str_echo_rtos.bin which I built myself with a new toolchain does work correctly.

-* when I say it “works” what I’m testing is the presence of /dev/ttyRPMSG30

Many thanks for the support,

Tim

Hello @Timo,

I am jumping in here as Rudhi is now on vacation.

Thanks for the update.
I saw your other thread and it looks like the response from the AI was enough to solve the matter.
The response is indeed correct and accurately explains the problem, but if you continue to have issues with that, please send another reply there so we can take it over.

Best Regards,
Bruno