I am trying to use Linux remoteproc with a Verdin IMX8MP on a Verdin Dev board. I am building YP with the following layers/versions:
meta-toradex-nxp = "kirkstone-6.x.y:364341c23ea49fb875e84f980b208699311aaf9c"
meta-freescale = "kirkstone:1c7f17f6063d0b747d94a17059b176f3ebdb3e3e"
meta-freescale-3rdparty = "kirkstone:e1ec96f0b1d89adab9ec1f224e7d6dcb0ef565c0"
meta-toradex-bsp-common = "kirkstone-6.x.y:6ee1533ec892e81a3f5937315ab132aa46fac3c7"
meta-oe
meta-filesystems
meta-gnome
meta-xfce
meta-networking
meta-multimedia
meta-python
meta-initramfs = "kirkstone:a9c25bef8882a69fd35ea7dbac3d978ce440fe06"
meta-freescale-distro = "kirkstone:d5bbb487b2816dfc74984a78b67f7361ce404253"
meta-qt6 = "dev:cfbcd5f4f1121f1d222ff2bdd7161c06b7809a2d"
meta-toradex-distro = "kirkstone-6.x.y:2712e46aadd2ec74711d69503b61dba3548e0aa4"
meta-poky
meta = "kirkstone:d3e378397395542179a8497fbff4b610f6f7e802"
I am using this device tree include to enable remoteproc/rpmsg:
#include <dt-bindings/clock/imx8mp-clock.h>
#include <dt-bindings/gpio/gpio.h>
/ {
aliases {
i2c0 = &i2c1;
i2c1 = &i2c2;
i2c2 = &i2c_rpbus_3;
};
reserved-memory {
#address-cells = <2>;
#size-cells = <2>;
ranges;
/delete-node/ linux,cma;
m4_reserved: m4@0x80000000 {
no-map;
reg = <0 0x80000000 0 0x1000000>;
};
m7_itcm: m4@0x7E0000 {
no-map;
reg = <0 0x7E0000 0 0x20000>;
};
m7_dtcm: m4@0x800000 {
no-map;
reg = <0 0x800000 0 0x20000>;
};
vdev0vring0: vdev0vring0@55000000 {
reg = <0 0x55000000 0 0x8000>;
no-map;
};
vdev0vring1: vdev0vring1@55008000 {
reg = <0 0x55008000 0 0x8000>;
no-map;
};
vdevbuffer: vdevbuffer@55400000 {
compatible = "shared-dma-pool";
reg = <0 0x55400000 0 0x100000>;
no-map;
};
rsc_table: rsc_table@550ff000 {
reg = <0 0x550ff000 0 0x1000>;
no-map;
};
};
imx8mp-cm7 {
compatible = "fsl,imx8mn-cm7";
rsc-da = <0x55000000>;
clocks = <&clk IMX8MP_CLK_M7_DIV>;
mbox-names = "tx", "rx", "rxdb";
mboxes = <&mu 0 1
&mu 1 1
&mu 3 1>;
memory-region = <&vdevbuffer>, <&vdev0vring0>, <&vdev0vring1>, <&rsc_table>, <&m4_reserved>, <&m7_itcm>, <&m7_dtcm>;
status = "okay";
};
};
/*
* ATTENTION: M7 may use IPs like below
* ECSPI0/ECSPI2, FLEXCAN, GPIO1/GPIO5, GPT1, I2C3, I2S3, UART4,
* PWM4, SDMA1/SDMA2
*/
&ecspi2 {
status = "disabled";
};
&flexcan1 {
status = "disabled";
};
&flexspi {
status = "disabled";
};
/delete-node/ &i2c3;
&pwm4{
status = "disabled";
};
&sai3 {
status = "disabled";
};
&sdma3{
status = "disabled";
};
&uart4 {
status = "disabled";
};
I am building a Zephyr image with
west build -p always -b mimx8mp_evk_ddr zephyr/samples/philosophers
for the Philosophers test application. I prefer that application as it continuously outputs the philosopher status to the serial console of the M core. I also build the application for itcm to test both.
Starting either application for ddr or itcm from the u-boot prompt works fine. Starting Linux after shows the running M core attached to remoteproc:
# cat /sys/class/remoteproc/remoteproc0/state
attached
Stopping the M core also works fine by
# echo stop >/sys/class/remoteproc/remoteproc0/state
However, starting the M core again either with the itcm or ddr elf causes a page fault in Linux:
For ddr:
# echo start >/sys/class/remoteproc/remoteproc0/state
[ 120.544522] remoteproc remoteproc0: powering up imx-rproc
[ 120.550578] remoteproc remoteproc0: Firmware is an elf32 file
[ 120.550596] remoteproc remoteproc0: Booting fw image rproc-imx-rproc-fw, size 676512
[ 120.558483] imx-rproc imx8mp-cm7: iommu not present
[ 120.558516] remoteproc remoteproc0: No resource table in elf
[ 120.564240] imx-rproc imx8mp-cm7: map memory: 0000000091bca557+100000
[ 120.564283] imx-rproc imx8mp-cm7: map memory: 00000000d2321245+8000
[ 120.564295] imx-rproc imx8mp-cm7: map memory: 00000000c7b96d40+8000
[ 120.564305] imx-rproc imx8mp-cm7: map memory: 00000000c8f03ea6+1000000
[ 120.564316] imx-rproc imx8mp-cm7: map memory: 00000000bc20cfac+20000
[ 120.564327] imx-rproc imx8mp-cm7: map memory: 00000000ea51ea99+20000
[ 120.564339] remoteproc remoteproc0: phdr: type 1 da 0x80000000 memsz 0x6e9c filesz 0x6e9c
[ 120.564349] remoteproc remoteproc0: da = 0x80000000 len = 0x6e9c va = 0x00000000b21df214
[ 120.564382] Unable to handle kernel paging request at virtual address ffff80000c006e5c
[ 120.572359] Mem abort info:
[ 120.575171] ESR = 0x96000061
[ 120.578363] EC = 0x25: DABT (current EL), IL = 32 bits
[ 120.583692] SET = 0, FnV = 0
[ 120.586759] EA = 0, S1PTW = 0
[ 120.589926] FSC = 0x21: alignment fault
[ 120.593949] Data abort info:
[ 120.596841] ISV = 0, ISS = 0x00000061
[ 120.600703] CM = 0, WnR = 1
[ 120.603671] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000049888000
[ 120.610384] [ffff80000c006e5c] pgd=100000013ffff003, p4d=100000013ffff003, pud=100000013fffe003, pmd=0068000080000711
[ 120.621023] Internal error: Oops: 96000061 [#1] PREEMPT SMP
[ 120.626602] Modules linked in: cfg80211 fsl_jr_uio caam_jr caamkeyblob_desc caamhash_desc caamalg_desc crypto_engine rng_core authenc libdes bluetooth rfkill hid_multitouch crct10dif_ce snd_soc_imx_hdmi sec_mipi_dsim_imx sec_dsim dw_hdmi_cec snd_soc_fsl_sai snd_soc_nau8822 snd_e
[ 120.626672] CPU: 3 PID: 874 Comm: sh Tainted: G O 5.15.77-6.1.0-devel+git.349786b46e61 #1
[ 120.626679] Hardware name: Toradex Verdin iMX8M Plus on Verdin Development Board (DT)
[ 120.626682] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 120.626689] pc : __memcpy+0x188/0x250
[ 120.626699] lr : rproc_elf_load_segments+0x184/0x260
[ 120.626707] sp : ffff80000b8bbb20
[ 120.626709] x29: ffff80000b8bbb20 x28: 0000000000006e9c x27: 0000000000006e9c
[ 120.626717] x26: 0000000080000000 x25: 00000000000000d4 x24: 0000000000000001
[ 120.626725] x23: ffff0000c11e7000 x22: 0000000000000020 x21: 0000000000000005
[ 120.626732] x20: 0000000000000001 x19: ffff80000bad5054 x18: ffffffffffffffff
[ 120.626740] x17: 303030307830203d x16: 2061762063396536 x15: 72796870655a2067
[ 120.626747] x14: 6e69746f6f42202a x13: 623738663038672d x12: 303635332d302e32
[ 120.626755] x11: 000a2a2a2a206236 x10: 3038343962373866 x9 : 3038672d30363533
[ 120.626763] x8 : 2d302e322e33762d x7 : 72796870657a2064 x6 : 6c69756220534f20
[ 120.626770] x5 : ffff80000c006e9c x4 : ffff80000badbf70 x3 : ffff80000c006e40
[ 120.626778] x2 : ffffffffffffffcc x1 : ffff80000badbf54 x0 : ffff80000c000000
[ 120.626786] Call trace:
[ 120.626789] __memcpy+0x188/0x250
[ 120.626794] imx_rproc_elf_load_segments+0x20/0x40
[ 120.779785] rproc_start+0x30/0x168
[ 120.779795] rproc_boot+0x344/0x5f0
[ 120.779800] state_store+0x44/0x104
[ 120.779805] dev_attr_store+0x18/0x30
[ 120.779812] sysfs_kf_write+0x44/0x54
[ 120.779819] kernfs_fop_write_iter+0x118/0x1ac
[ 120.779825] new_sync_write+0xe8/0x184
[ 120.779831] vfs_write+0x22c/0x290
[ 120.779835] ksys_write+0x68/0xf4
[ 120.779839] __arm64_sys_write+0x1c/0x2c
[ 120.779844] invoke_syscall+0x48/0x114
[ 120.779851] el0_svc_common.constprop.0+0xd4/0xfc
[ 120.779856] do_el0_svc+0x28/0x90
[ 120.779861] el0_svc+0x28/0x80
[ 120.779867] el0t_64_sync_handler+0xa4/0x130
[ 120.779872] el0t_64_sync+0x1a0/0x1a4
[ 120.779883] Code: a97e2488 a9032c6a a97f2c8a a904346c (a93c3cae)
[ 120.779888] ---[ end trace 6d13cbdfead451d0 ]---
For itcm:
# echo start >/sys/class/remoteproc/remoteproc0/state
[ 86.857238] remoteproc remoteproc0: powering up imx-rproc
[ 86.881319] remoteproc remoteproc0: Firmware is an elf32 file
[ 86.881339] remoteproc remoteproc0: Booting fw image rproc-imx-rproc-fw, size 676496
[ 86.889187] imx-rproc imx8mp-cm7: iommu not present
[ 86.889223] remoteproc remoteproc0: No resource table in elf
[ 86.894994] imx-rproc imx8mp-cm7: map memory: 0000000048d67ebf+100000
[ 86.895044] imx-rproc imx8mp-cm7: map memory: 00000000b777ff91+8000
[ 86.895065] imx-rproc imx8mp-cm7: map memory: 00000000a047c2fb+8000
[ 86.895077] imx-rproc imx8mp-cm7: map memory: 000000001459a6a0+1000000
[ 86.895201] imx-rproc imx8mp-cm7: map memory: 00000000b2e789f0+20000
[ 86.895212] imx-rproc imx8mp-cm7: map memory: 00000000637ae398+20000
[ 86.895222] remoteproc remoteproc0: phdr: type 1 da 0x0 memsz 0x6e84 filesz 0x6e84
[ 86.895230] remoteproc remoteproc0: da = 0x0 len = 0x6e84 va = 0x00000000db7dc377
[ 86.895298] remoteproc remoteproc0: phdr: type 1 da 0x6e84 memsz 0x12 filesz 0x12
[ 86.895304] remoteproc remoteproc0: da = 0x6e84 len = 0x12 va = 0x00000000bdcd6411
[ 86.895309] remoteproc remoteproc0: phdr: type 1 da 0x6e96 memsz 0x4 filesz 0x4
[ 86.895315] remoteproc remoteproc0: da = 0x6e96 len = 0x4 va = 0x000000007e8d31ef
[ 86.895321] remoteproc remoteproc0: phdr: type 1 da 0x20000018 memsz 0x43c0 filesz 0x0
[ 86.895327] remoteproc remoteproc0: da = 0x20000018 len = 0x43c0 va = 0x0000000037d3ce0e
[ 86.895380] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[ 86.904206] Mem abort info:
[ 86.907027] ESR = 0x96000004
[ 86.910101] EC = 0x25: DABT (current EL), IL = 32 bits
[ 86.915449] SET = 0, FnV = 0
[ 86.918519] EA = 0, S1PTW = 0
[ 86.921672] FSC = 0x04: level 0 translation fault
[ 86.926575] Data abort info:
[ 86.929469] ISV = 0, ISS = 0x00000004
[ 86.933318] CM = 0, WnR = 0
[ 86.936287] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000106616000
[ 86.942751] [0000000000000000] pgd=0000000000000000, p4d=0000000000000000
[ 86.949560] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[ 86.955136] Modules linked in: cfg80211 fsl_jr_uio caam_jr caamkeyblob_desc caamhash_desc caamalg_desc crypto_engine rng_core authenc libdes bluetooth rfkill hid_multitouch crct10dif_ce snd_soc_imx_hdmi snd_soc_nau8822 dw_hdmi_cec lm75 ina2xx lontium_lt8912b flexcan snd_soc_fsle
[ 86.955205] CPU: 3 PID: 880 Comm: sh Tainted: G O 5.15.77-6.1.0-devel+git.349786b46e61 #1
[ 86.955211] Hardware name: Toradex Verdin iMX8M Plus on Verdin Development Board (DT)
[ 87.007503] pstate: a0000005 (NzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 87.007509] pc : __memcpy+0x110/0x250
[ 87.007519] lr : rproc_start+0x84/0x168
[ 87.007526] sp : ffff80000db5bbd0
[ 87.007528] x29: ffff80000db5bbd0 x28: ffff0000c175b800 x27: 0000000000000000
[ 87.007536] x26: 0000000000000000 x25: ffff0000c13d8c80 x24: ffff0000c1783f80
[ 87.007544] x23: ffff0000c1048340 x22: ffff0000c1783f80 x21: ffff0000c1048038
[ 87.007551] x20: ffff800009e1d000 x19: ffff0000c1048000 x18: ffffffffffffffff
[ 87.007559] x17: 303030307830203d x16: 2061762030633334 x15: 7830203d206e656c
[ 87.007567] x14: ffff800009e1d000 x13: ffff8000099e2500 x12: 0000000000000666
[ 87.007575] x11: 0000000000000222 x10: ffff8000099e2500 x9 : ffff8000099e2500
[ 87.007582] x8 : 00000000ffffefff x7 : ffff800009a3a500 x6 : ffff800009a3a500
[ 87.007590] x5 : ffff800009e1d400 x4 : 0000000000000400 x3 : 0000000000000000
[ 87.007597] x2 : 0000000000000400 x1 : 0000000000000000 x0 : ffff800009e1d000
[ 87.007605] Call trace:
[ 87.007608] __memcpy+0x110/0x250
[ 87.007613] rproc_boot+0x344/0x5f0
[ 87.007619] state_store+0x44/0x104
[ 87.007624] dev_attr_store+0x18/0x30
[ 87.007631] sysfs_kf_write+0x44/0x54
[ 87.007639] kernfs_fop_write_iter+0x118/0x1ac
[ 87.007645] new_sync_write+0xe8/0x184
[ 87.007651] vfs_write+0x22c/0x290
[ 87.007655] ksys_write+0x68/0xf4
[ 87.007660] __arm64_sys_write+0x1c/0x2c
[ 87.007664] invoke_syscall+0x48/0x114
[ 87.007671] el0_svc_common.constprop.0+0xd4/0xfc
[ 87.007676] do_el0_svc+0x28/0x90
[ 87.007681] el0_svc+0x28/0x80
[ 87.007687] el0t_64_sync_handler+0xa4/0x130
[ 87.007693] el0t_64_sync+0x1a0/0x1a4
[ 87.007701] Code: cb01000e b4fffc2e eb0201df 540004a3 (a940342c)
[ 87.007706] ---[ end trace d0ace529e7c99988 ]---
I have read and studied pretty much all information I was able to find on the web:
- NXP Application Note AN5317
- Marcel Zisweiler’s Embedded Linux Conference Presentation
- IMX8MMEVK: Remoteproc to reboot M4 from Linux
It is entirely possible that the elf files are not created correctly by the Zephyr build process.
This is the readelf -S dump for the ddr image:
# readelf -S zephyr.philosophers.mimx8mp_evk_ddr.elf
There are 30 section headers, starting at offset 0xa4df0:
Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] rom_start PROGBITS 80000000 0000d4 0002bc 00 WAX 0 0 4
[ 2] text PROGBITS 800002bc 000390 0051fc 00 AX 0 0 4
[ 3] .ARM.exidx ARM_EXIDX 800054b8 00558c 000008 00 AL 2 0 4
[ 4] initlevel PROGBITS 800054c0 005594 000038 00 A 0 0 4
[ 5] devices PROGBITS 800054f8 0055cc 000048 00 A 0 0 4
[ 6] sw_isr_table PROGBITS 80005540 005614 0004f8 00 WA 0 0 4
[ 7] device_handles PROGBITS 80005a38 005b0c 000016 00 A 0 0 2
[ 8] log_const_se[...] PROGBITS 80005a50 005b24 000020 00 A 0 0 4
[ 9] zephyr_dbg_info PROGBITS 80005a70 005b44 000040 00 WA 0 0 4
[10] rodata PROGBITS 80005ab0 005b84 0013ec 00 A 0 0 4
[11] .ramfunc PROGBITS 80200000 006f86 000000 00 W 0 0 1
[12] datas PROGBITS 80200000 006f70 00000c 00 WA 0 0 4
[13] device_states PROGBITS 8020000c 006f7c 000006 00 WA 0 0 1
[14] bss NOBITS 80200018 006f88 00067a 00 WA 0 0 8
[15] noinit NOBITS 80200698 006f88 003d40 00 WA 0 0 8
[16] .comment PROGBITS 00000000 006f86 000020 01 MS 0 0 1
[17] .debug_aranges PROGBITS 00000000 006fa8 000fd8 00 0 0 8
[18] .debug_info PROGBITS 00000000 007f80 04a890 00 0 0 1
[19] .debug_abbrev PROGBITS 00000000 052810 008b90 00 0 0 1
[20] .debug_line PROGBITS 00000000 05b3a0 01792f 00 0 0 1
[21] .debug_frame PROGBITS 00000000 072cd0 002464 00 0 0 4
[22] .debug_str PROGBITS 00000000 075134 00be6b 01 MS 0 0 1
[23] .debug_loc PROGBITS 00000000 080f9f 016fa6 00 0 0 1
[24] .debug_ranges PROGBITS 00000000 097f48 003c08 00 0 0 8
[25] .ARM.attributes ARM_ATTRIBUTES 00000000 09bb50 00002e 00 0 0 1
[26] .last_section PROGBITS 80006eae 006f82 000004 00 A 0 0 1
[27] .symtab SYMTAB 00000000 09bb80 004f90 10 28 659 4
[28] .strtab STRTAB 00000000 0a0b10 00419d 00 0 0 1
[29] .shstrtab STRTAB 00000000 0a4cad 000142 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
D (mbind), y (purecode), p (processor specific)
The start address is 0x8000.0000 which seems correct to me. However, the DDR maps into the A53 address space and into the M7 address space at the same start address albeit with different address space sizes.
Where I am lost is at the address mapping:
[ 120.564349] remoteproc remoteproc0: da = 0x80000000 len = 0x6e9c va = 0x00000000b21df214
[ 120.564382] Unable to handle kernel paging request at virtual address ffff80000c006e5c
If the virtual address after mapping is va = 0x00000000b21df214 why would the paging request occur at ffff80000c006e5c?
And this one is for the itcm elf:
# readelf -S zephyr.philosophers.mimx8mp_evk_itcm.elf
There are 30 section headers, starting at offset 0xa4de0:
Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] rom_start PROGBITS 00000000 0000d4 0002bc 00 WAX 0 0 4
[ 2] text PROGBITS 000002bc 000390 0051fc 00 AX 0 0 4
[ 3] .ARM.exidx ARM_EXIDX 000054b8 00558c 000008 00 AL 2 0 4
[ 4] initlevel PROGBITS 000054c0 005594 000038 00 A 0 0 4
[ 5] devices PROGBITS 000054f8 0055cc 000048 00 A 0 0 4
[ 6] sw_isr_table PROGBITS 00005540 005614 0004f8 00 WA 0 0 4
[ 7] device_handles PROGBITS 00005a38 005b0c 000016 00 A 0 0 2
[ 8] log_const_se[...] PROGBITS 00005a50 005b24 000020 00 A 0 0 4
[ 9] zephyr_dbg_info PROGBITS 00005a70 005b44 000040 00 WA 0 0 4
[10] rodata PROGBITS 00005ab0 005b84 0013d4 00 A 0 0 4
[11] .ramfunc PROGBITS 20000000 006f6e 000000 00 W 0 0 1
[12] datas PROGBITS 20000000 006f58 00000c 00 WA 0 0 4
[13] device_states PROGBITS 2000000c 006f64 000006 00 WA 0 0 1
[14] bss NOBITS 20000018 006f70 00067a 00 WA 0 0 8
[15] noinit NOBITS 20000698 006f70 003d40 00 WA 0 0 8
[16] .comment PROGBITS 00000000 006f6e 000020 01 MS 0 0 1
[17] .debug_aranges PROGBITS 00000000 006f90 000fd8 00 0 0 8
[18] .debug_info PROGBITS 00000000 007f68 04a890 00 0 0 1
[19] .debug_abbrev PROGBITS 00000000 0527f8 008b90 00 0 0 1
[20] .debug_line PROGBITS 00000000 05b388 01792f 00 0 0 1
[21] .debug_frame PROGBITS 00000000 072cb8 002464 00 0 0 4
[22] .debug_str PROGBITS 00000000 07511c 00be6b 01 MS 0 0 1
[23] .debug_loc PROGBITS 00000000 080f87 016fa6 00 0 0 1
[24] .debug_ranges PROGBITS 00000000 097f30 003c08 00 0 0 8
[25] .ARM.attributes ARM_ATTRIBUTES 00000000 09bb38 00002e 00 0 0 1
[26] .last_section PROGBITS 00006e96 006f6a 000004 00 A 0 0 1
[27] .symtab SYMTAB 00000000 09bb68 004f90 10 28 659 4
[28] .strtab STRTAB 00000000 0a0af8 0041a4 00 0 0 1
[29] .shstrtab STRTAB 00000000 0a4c9c 000142 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
D (mbind), y (purecode), p (processor specific)
Here the start address is 0x0000.0000 which is correct for the ITCM on the M7 but on the A53 it should be 0x007e.0000.
Since @marcel.tx was apparently able to make this work for the 8M Plus according to the presentation, I would appreciate some suggestions.
However, since the kernel is a Toradex fork from the NXP kernel there might be some code that has not been ported yet. I didn’t do that analysis yet but you probably know better about the Toradex kernel tree ancestry.