I made a change to the overlay removing a specific node that I had added before to address an issue that seems solved by the kernel patch. It looks like removing this node avoids the kernel panic, according to my tests.
I’ve attached both the overlay source and the compiled blob. Would you be able to test this, please?
verdin-imx8mp_rpmsg_overlay.dtbo (2.8 KB)
verdin-imx8mp_rpmsg_overlay.dts (1.7 KB)
Hi, @gustavo.tx san, @alvaro.tx san.
Thank you for the update information.
I tried to applying new verdin-imx8mp_rpmsg_overlay.dtbo.
I found “virtio_rpmsg_bus … error -12” in boot log. What should I review ?
boot.log (42.4 KB)
[ 0.276825] 000: virtio_rpmsg_bus: probe of virtio0 failed with error -12
Best regards, Takeo.
Good morning @gustavo.tx @alvaro.tx
I can again confirm the findings of @deepimpact
I use the same board as @gustavo.tx (Verdin iMX8MP Q 2GB). As a base image I used again commit 35cb96f1e8834a1a4dea2a3ee24b2bc0595438ba (HEAD → default, origin/dunfell-5.x.y,
m/dunfell-5.x.y) with the second CAAM patch from @alvaro.tx
root@verdin-test:~# dmesg | grep rpmsg
[ 0.063090] imx rpmsg driver is registered.
[ 1.827807] imx_rpmsg_find_vqs+0xd4/0x268
[ 1.831907] rpmsg_probe+0xdc/0x338
[ 1.875103] imx_rpmsg_probe+0x1f4/0x430
[ 1.934589] virtio_rpmsg_bus: probe of virtio0 failed with error -12
Hi @gustavo.tx
I need to get this working urgently. You use the same module as we are, so I guess it is really only a problem of using the right manifest/kernel.
Can you please state explicitly which commit of toradex-manifest you are using, and what patch you apply?
Hi @dpenning!
Another customer was able to make this work with the latest overlay I posted. IIRC, they’re using our default image. To clarify, I’ll set up everything from scratch and test this all out.
About the manifest, we’re using everything default here. The patch to be applied is attached. Apply this to kernel branch toradex_5.4-2.3.x-imx.
I’ll start working on reproducing and documenting this right now and keep you up-to-date.
caam-fix.patch.diff (1021 Bytes)
In the meantime, I’ve uploaded the kernel image, the device tree overlay and the kernel modules that are running on my testing module right now.
5.4.129-37347-g511c08dfa42b-dirty.tar.xz (3.7 MB)
verdin-imx8mp_rpmsg_overlay.dtbo (2.9 KB)
Image.gz
Hi @gustavo.tx
Can you please also post your version of imx8mp-verdin-nonwifi-dev.dtb ?
Thanks for reproducing on a clean machine, I am very much looking forward to your results!
Sure!
Here you go:
imx8mp-verdin-nonwifi-dev.dtb (83.2 KB)
@gustavo.tx Thanks! I naively tried to patch only parts of my running system with your files. Kernel/Modules/DT but once I enable your overlay, it crashes during boot with a kernel panic. I will wait for your result of reproducing the build.
In the process of reproducing the issue from scratch I stumbled upon the same kernel panic. Turns out this latest overlay I had uploaded still wasn’t updated. Now everything’s working here for me. Let me describe all the steps you have to take.
fdt_high U-Boot variable either via U-Boot itself (setenv fdt_high 0xffffffffffffff then saveenv and boot) or via Linux userspace ( fw_setenv fdt_high 0xffffffffffffff, then reboot)overlays.txt to load said overlay. Just to be on the safe side, I removed the other overlays (display related) and loaded only the rpmsg one. overlays.txt should look like:root@verdin-imx8mp:/boot# cat overlays.txt
fdt_overlays=verdin-imx8mp_rpmsg_overlay.dtbo
modprobe imx_rpmsg_tty. In the case of the rpmsg_str_echo demo, you should see communication happening.Good luck and please report back so I can help with anything you might need!
verdin-imx8mp_rpmsg_overlay.dtbo (2.8 KB)
Hi @gustavo.tx Thanks for reproducing. May you post the new contents of the overlay as well?
I will integrate everything in a clean yocto build and let you know about the results.
Sure! It looks like the content is the same, I might have done something wrong while compiling the previous version. But here is the source for this latest blob:
verdin-imx8mp_rpmsg_overlay.dts (1.7 KB)
Hi @gustavo.tx
I tried two ways of doing this, both failed with the same result (already observed earlier above by @deepimpact and me).
[ 1.932476] virtio_rpmsg_bus: probe of virtio0 failed with error -12
First approach:
I downloaded the minimal image you mentioned (https://artifacts.toradex.com/artifactory/tdxref-oe-prod-frankfurt/dunfell-5.x.y/release/10/verdin-imx8mp/tdx-xwayland/tdx-reference-minimal-image/oedeploy/Verdin-iMX8MP_Reference-Minimal-Image-Tezi_5.3.0+build.10.tar)
After boot, I replaced the Image and the modules with the ones you provided in your previous post.
I followed all other instructions you mentioned. The result is the above failure during boot.
boot.log (34.3 KB)
As seen in the boot log I use these commands to start the RPMSG demo on the M7.
setenv m4addr 0x7e0000
setenv fdt_high 0xffffffffffffff
saveenv
fatload mmc 2 0x48000000 rpmsg_demo.bin && cp.b 0x48000000 ${m4addr} 20000 && bootaux ${m4addr}
The M7 UART then properly displays the beginning of the demo:
RPMSG String Echo FreeRTOS RTOS API Demo...
I then use boot command to proceed booting linux.
I would expect that
modprobe imx_rpmsg_ttySecond approach
I used toradex manifest 1687afae4359efc5ced34e6ceb701527462ff7c1 (5.3.0), patched the CAAM driver and and built everything from scratch using yocto. Exactly the same result.
I tried this because formerly I based everything on manifest 5.4.0.
What next?
Could you possible upload a complete installable image (Torizon Installer) that is working on your Verdin IMX8MP Q 2GB? Just to make sure there is no hardware issue?
Edit: I tested on the Verdin Development Board and also using our own custom board.
This is very strange! A Linux error -12 usually indicates ENOMEM (out of memory), which doesn’t make sense in this context.
Can you confirm which SDK you’re using for the M7? Also, which demo are you compiling? Did you make any modifications to the demo?
It’s particularly strange because a customer followed the exact same steps and got it to work just like it does in my setup.
I forgot to ask you a relevant question: which version of the module and carrier board are you using?
Hi @gustavo.tx , just tried this and like the fellow @deepimpact and @dpenning, it boots but I get the same:
root@verdin-imx8mp:~# dmesg | grep rpm
[ 0.077909] imx rpmsg driver is registered.
[ 2.033505] imx_rpmsg_find_vqs+0xd4/0x268
[ 2.037607] rpmsg_probe+0xdc/0x338
[ 2.080805] imx_rpmsg_probe+0x1f4/0x430
[ 2.140286] virtio_rpmsg_bus: probe of virtio0 failed with error -12
[ 2.699272] mmcblk2rpmb: mmc2:0001 S0J56X partition 3 4.00 MiB, chardev (237:0)
The kernel changes are as always, the caam patch. And I directly use your latest DTBO. Also, when modprobbing imx_rpmsg_tty I get no response…
The demo I’m using is rpmsg_lite_str_echo_rtos.bin, as-is, but you can send your binary and we can check if that is the issue (Shouldn’t be). I get the RPMSG String Echo FreeRTOS RTOS API Demo... message.
BTW, what are the changes in the device tree? I just diff this and saw no difference.
I second @dpenning: If possible, please create a Toradex Easy Installer image and see if we can get it working that way, at least as a binary… So that the customers are not stuck and they can continue the development at least partially. You and I can figure out later the exact changes and document it.
Thanks, Alvaro.
Hi @gustavo.tx , some more thoughts: The demo is loaded to the eMMC through:
fatload mmc 2 0x48000000 rpmsg_demo.bin && cp.b 0x48000000 ${m4addr} 20000 && bootaux ${m4addr}
Could it be that you have the demo stored somewhere else and then directly loaded to RAM? Can you share your commands on loading the demo? So far, I think the commands where only shared here (and we are all having the same issue): RPMSG does not work on IMX8M-Plus - #52 by alvaro.tx
This could explain the ENOMEM error if there is some collision between where the .bin is loaded and other reserved memory.
Hi @alvaro.tx
Thats an interesting guess. @gustavo.tx can you please share your commands on loading the demo, and the binary demo itself?
Hi @dpenning and @alvaro.tx!
I’ve used virtually the same command (which is on the Getting Started documentation that comes with the SDK), except that I use TFTP for loading instead of an SD card, but I also tried it and it worked:
tftp 0x48000000 rpmsg_lite_str_echo_rtos.bin; cp.b 0x48000000 0x7e0000 20000; bootaux 0x7e0000
Since you defined $m4addr as 0x7e0000, the command is the same. I’ve attached the latest binary I tried. Please give it a try. This is the “debug” release.
rpmsg_lite_str_echo_rtos.bin (28.5 KB)
Hi @gustavo.tx
Thanks for the quick response. I tried your binary, unfortunately the same result - nothing changed. I also booted from a TFTP server, so this has no effect.
Can you please provide an installable Toradex Easy Installer, so that we can make sure it is no hardware issue? Maybe @alvaro.tx has an idea to make otherwise sure we have the same conditions?
A build reproduced on yocto would be the another possible strategy I think?
I think we need to guarentee that all environment conditions (uboot etc) are absolutely the same. A yocto build based on your distribution will guarentee that.
Edit: After the M7 is set up, I just use the boot command to start the linux boot. I hope this is right.
Hey @dpenning!
This is rather strange. Another customer was able to make this work, so we cannot discard this being a hardware issue.
Can you please provide the version of your module and carrier board?
I’ll work on providing you an installable image from what I got on my target.