iMX7D 5.4.x kernel. Launching M4 FW using remoteproc

@gustavo.tx,

Just restored to default kernel config. The same. M4 working properly after launching with ./m4fwloader fw-tcm.bin 0x7F8000, but no /dev/ttyRPMSG0 or virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x0 in dmesg. I even tried kicking MU with m4fwloader. Kicks are recognised by M4, but no openmap in dmesg.

dmesg config

Hi,

kernel cloned from here . Xwayland, minimal.

Initial device tree doesn’t provide working remoteproc settings for imx7d, I mean compatible “fsl,imx7d-cm4”, so that it would work with both imx_rproc and imx_rpmsg drivers. Whatever I do, in case M4 is started from remoteproc/imx_rproc, imx_rpmsg doesn’t see M4 talking. I mean missing virtio_rpmsg_bus strings like this

[ 1855.725824] virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x0
[ 1855.734294] virtio_rpmsg_bus virtio0: rpmsg host is online
[ 1855.741298] imx rpmsg driver is registered.

I see only this:

[ 1855.741298] imx rpmsg driver is registered.

@Edward,

I’m gonna try to reproduce this and let you know.

@Edward,

Can you please share your changes to the device tree? I tried something based on your first post but couldn’t make it work as an overlay.

Hi @gustavo.tx,

Actually with m4fwloader we can eliminate remoteproc from puzzle and concentrate on why it matters for rpmsg whether M4 or Linux boots first.
I already modified tree and don’t have original one. Here’s what I have now, nombox = "true"; is from my attempt to cut all MU stuff from remoteproc driver. I thought it could matter somehow, but m4fwloader just confirms that something is not right either in RPMSG.

/dts-v1/;
#include "imx7d-colibri.dtsi"
#include "imx7-colibri-eval-v3.dtsi"

/ {
	model = "Toradex Colibri iMX7D on Colibri Evaluation Board V3";
	compatible = "toradex,colibri-imx7d-eval-v3",
		     "toradex,colibri-imx7d",
		     "fsl,imx7d";

/*	memory@80000000 {
		device_type = "memory";
		reg = <0x80000000 0x0fff0000>,
		      <0x90000000 0x10000000>;
	};
*/

	reserved-memory {
		#address-cells = <1>;
		#size-cells = <1>;
		ranges;

		vrings0@0x8fff0000 {
			compatible = "shared-dma-pool";
			reg = <0x8fff0000 0x10000>;
			no-map;
		};



		cm4sramcode: cm4sramcode@0x900000 {
			compatible = "shared-dma-pool";
			reg = <0x00900000 0x40000>;
			no-map;
		};

		cm4tcmcode: cm4tcmcode@0x7f8000 {
			compatible = "shared-dma-pool";
			reg = <0x007f8000 0x8000>;
			no-map;
		};
/*
		_vdev0vring0: _vdev0vring0@8fff0000 {
			compatible = "shared-dma-pool";
			reg = <0x8fff0000 0x8000>;
			no-map;
		};
		_vdev0vring1: _vdev0vring1@8fff8000 {
			compatible = "shared-dma-pool";
			reg = <0x8fff8000 0x8000>;
			no-map;
		};
*/
	};


/*
	imx7d-cm4 {
		compatible = "fsl,imx7d-cm4", "syscon";
//		ipc-only;
//		rsc-da=<0x8fff0000>;
		nombox = "true";
		mbox-names = "tx", "rx", "rxdb", "txdb";
		syscon = <&src>;
		mboxes = <&mu 0 1
			  &mu 1 1
			  &mu 3 1
			  &mu 2 1>;
//		memory-region = <&vdev0vring0>, <&vdev0vring1>;
		memory-region = <&cm4sramcode>, <&cm4tcmcode>;
	};
*/
};

@Edward,

This really looks like something going on with the RPMsg implementation. Did you check if the kernel modules loaded correctly?

Would you be able to inquire NXP in their community regarding this, since their implementation is not upstream? Meanwhile I’m checking internally if we’re able to further investigate this. What exactly is your use case? Do you have an application that should launch the firmware using remoteproc? Is launching the firmware from Linux an absolute requirement?

Hi @gustavo.tx,

Yes, modules load fine.

  • Would you be able to inquire NXP in their community regarding this, since their implementation is not upstream?

To request them I need to use their kernel. Do they have one that could run on Colibri? Any directions I could try their kernel? Actually I’m not familiar with what is up and down-stream.

  • What exactly is your use case? Do you have an application that should launch the firmware using remoteproc? Is launching the firmware from Linux an absolute requirement?

M4 is must have due to real time requirements not achievable with Linux only. Not an absolute requirement, but I already upgraded all required VF61 drivers for 5.4 kernel, including rpmsg and remoteproc. remoteproc + rpmsg is working. It would be nice if the same M4 approach could work as well on iMX7D.

  1. Unlike remoteproc for VF61, remoteproc for iMX7D allows M4 code reload, which would greatly speed up M4 debugging!
  2. Services or whatever it is called feature sounds interesting. If it worked, I’d try to use different “rpc” calls
  3. Bootauxing M4 code from u-boot makes u-boot and its environment variables more critical part, which I would like to avoid.
  4. Upgrading application code could be limited to rootfs only…

Since you mention upstream, I diffed toradex_5.4.y/drivers/rpmsg with toradex_5.4-2.3.x-imx/drivers/rpmsg. virtio_rpmsg_bus.c is different even when CONFIG_RPMSG_VIRTIO_CHAR is not defined… I think I should try that driver, unfortunately no time for it by now.

Regards,
Edward

@Edward,

The downstream kernel we’re using is actually based on the NXP release. Just make sure you’re using the toradex_5.4-2.3.x-imx branch. The toradex_5.4.y branch contains our mainline-based kernel, as you noticed.

Hi @gustavo.tx

Thanks for your reply. I tried those versions, no go. For some reason rpmsg doesn’t start if M4 is started from Linux. I’ll look for differences between VF61 and iMX7D code when time permits. Of course iMX7D carrier board with JTAG could help, perhaps we will buy one.

@Edward,

Let me know of any developments on your side. I’ll also make sure to update this topic and the developer documentation in case of any new findings.

Hi @gustavo.tx,

Thank you. No, I had to switch to different taks, u-boot currently launches M4. If nothing breaks plans, I’m going to play with it soon.
Looking for something else I found This and This . I was trying to modularize rpmsg and reload modules, perhaps I modularized some different part of it, I should check. Aster and JTAG debugger are ready, so I hope it will be solved.

Hi @gustavo.tx,

I found something weird. 4.14 kernel or 5.x - the same issue. Modifying 4.14 Kconfig files I made imx_rpmsg driver loadable, put printk’s to trace MU driver and imx_rpmsg calls. Issue is somewhat fuzzy behaving device tree. I don’t know, but if cat /sys/firmware/devicetree/../rpmsg/status gives disabled, then even don’t try launching M4 from Linux, RPMSG won’t work. If status is OK, then starting M4 by m4fwloader, JTAG debugger, perhaps remoteproc as well should make RPMSG working, no matter is imx_rpmsg loaded before launching M4 or after.

The problem is something flips rpmsg DT status in /sys/firmware from okay to disabled.

Without doing any changes to DT, using the same *.dtb file, one time /sys status matches DT and is okay, and another time it doesn’t match DT and is disabled:

  1. Loading M4 FW and then issuing bootaux - makes imx_rpmsg /sys status okay.

  2. Not loading M4 FW after reset and just issuing to uninitialized memory bootuax 80800000 - makes imx_rpmsg /sys status okay again :-).

  3. Not touching bootaux, using JTAG debugger without any cool Linux kernel support or something, setting HW breakpoint at /init/main.c start_kernel() routine, reaching that breakpoint (so it stops for a while), then continuing executing … makes imx_rpmsg /sys status okay again. :slight_smile:

  4. Not using bootaux or JTAG renders /sys/firmware/devicetree/../rpmsg/status disabled. Since disabled, driver isn’t probed, which gives no chance to get RPMSG working launching M4 from Linux.

I’m picking main hair why it is so. Made many different searches though kernel sources to find what could render imx_rpmsg status disabled, no results yet.

Any advice how to figure why status could flip from okay in dtb to disabled in /sys? p4 above suggests something like missing cache flush, missing sync barrier, broken RAM, etc.

Loading dtb file in U-Boot and inspecting DT settings using fdt list and fdt print confirms okay. But Linux sometimes doesn’t agree with this.

Edward

Hi @gustavo.tx,

Well, I have workaround for oddities mentioned in my previous message. In short, for some odd reasons Linux may flip status of first rpmsg device from okay to disabled. My workaround is to disable first rpmsg instance and create new one. Renaming first instance didn’t work, mystery. Let me give you some details, which may help understanding better or reproducing.

  1. Teziying latest stable: TDX Wayland with XWayland 5.1.0+build.4 (dunfell) colibri-imx7 ttymxc0 Colibri-iMX7_Reference-Minimal-Image
  2. Not touching everything and not bootaux’ing M4 FW, original imx7d-colibri-eval-v3.dtb
    [upload|HCD2O/e2RUj0udSJslkLetK+B90=]
  3. Check what’s inside dtb:
    [upload|9bwYANbs7HrkTCByAz0Diw1qimk=]
  4. Disabling first rpmsg, adding second rpmsg, reserving memory for vring:
    diff --git a/imx7d-colibri-eval-v3.dts~ b/imx7d-colibri-eval-v3.dts
    index 3d2b48e..9af58c9 100644
    --- a/imx7d-colibri-eval-v3.dts~
    +++ b/imx7d-colibri-eval-v3.dts
    @@ -49,8 +49,48 @@
            compatible = "toradex,colibri-imx7d-eval-v3",
                         "toradex,colibri-imx7d",
                         "fsl,imx7d";
    +
    +       reserved-memory {
    +               #address-cells = <1>;
    +               #size-cells = <1>;
    +               ranges;
    +
    +               rpmsg-vrings@0x8fff0000 {
    +                       compatible = "shared-dma-pool";
    +                       reg = <0x8fff0000 0x10000>;
    +                       no-map;
    +               };
    +       };
    +       soc {
    +               rpmsg1 {
    +                       vdev-nums = <1>;
    +                       reg = <0x8fff0000 0x10000>;
    +                       status = "okay";
    +       
    +                       compatible = "fsl,imx7d-rpmsg";
    +                       /* up to now, the following channels are used in imx rpmsg
    +                       * - tx1/rx1: messages channel.
    +                       * - general interrupt1: remote proc finish re-init rpmsg stack
    +                       *   when A core is partition reset.
    +                       */
    +                       mbox-names = "tx", "rx", "rxdb";
    +                       mboxes = <&mu 0 1
    +                               &mu 1 1
    +                               &mu 3 1>;
    +               };
    +       };
    +};
    +
    +
    +/* defined in imx7s.dtsi, disabled. Reenabled and specified VRING address
    +   in imx7-colibri.dtsi.
    +    Disabled it and redefine new instance
    +*/
    +&rpmsg {
    +       status = "disabled";
     };
     
    +
     &usbotg2 {
            vbus-supply = <&reg_usbh_vbus>;
            status = "okay";
    
  1. Check dtb:
    [upload|bCWroN70x4aYXd2TRSRjKjgJ8JI=]
  2. Result, 2nd rpmsg instance visible to Linux:
    [upload|kBvcBdbKcEI3719ro6Ybztm+x0Q=]

Since status is OK, imx_rpmsg driver can be successfully probed. I was able to launch M4 from Linux and got RPMSG working. Remoteproc should work as well, will check later

Hi @gustavo.tx,

OK, it turns out that guilty for these issues is U-Boot. It checks if M4 was started or just attempted to start and disables in DT “fsl,imx7d-rpmsg”.

Here’s excerpt from u-boot-2020.07/u-boot-toradex/board/toradex/colibri_imx7/colibri_imx7.c:

#if defined(CONFIG_OF_LIBFDT) && defined(CONFIG_OF_BOARD_SETUP)
int ft_board_setup(void *blob, bd_t *bd)
{
#if defined(CONFIG_IMX_BOOTAUX) && defined(CONFIG_ARCH_FIXUP_FDT_MEMORY)
	int up;

	up = arch_auxiliary_core_check_up(0);
	if (up) {
		int ret;
		int areas = 1;
		u64 start[2], size[2];

		/*
		 * Reserve 1MB of memory for M4 (1MiB is also the minimum
		 * alignment for Linux due to MMU section size restrictions).
		 */
		start[0] = gd->bd->bi_dram[0].start;
		size[0] = SZ_256M - SZ_1M;

		/* If needed, create a second entry for memory beyond 256M */
		if (gd->bd->bi_dram[0].size > SZ_256M) {
			start[1] = gd->bd->bi_dram[0].start + SZ_256M;
			size[1] = gd->bd->bi_dram[0].size - SZ_256M;
			areas = 2;
		}

		ret = fdt_set_usable_memory(blob, start, size, areas);
		if (ret) {
			eprintf("Cannot set usable memory\n");
			return ret;
		}
	} else {
		int off;

		off = fdt_node_offset_by_compatible(blob, -1,
						    "fsl,imx7d-rpmsg");
		if (off > 0)
			fdt_status_disabled(blob, off);
	}

Please remove removal of imx7d-rpmsg device from DT!

Edward

Hi @Edward

Perfect that you found the solution. Thanks for the feedback.

Best regards,
Jaski

Hi @Edward

Did you get the rpmsg and rproc to work together? Can you share the entrees in the DT?

Thanks
CMR

Current U-Boot 2020.07, just checked it in git.toradex.com, still disables RPMSG unless M4 is started from U-Boot. See fdt_status_disabled(blob, off) call in my May 5 answer. You need to fix it first and only then expect RPMSG working starting M4 from Linux using imx_m4fwloader or rproc. iMX7 rproc as I understand got broken after FSL or someone else added support for iMX8. Yes, I made it working, but code is too dirty hack to share.

Thank you for the update. We are planning on running a small initialization app on M4 from U-Boot and loading the main app on m4 from Linux. But I will probably use your work around to prevent that from being an issue anyway.

@CMR,

see attached two diffs. One fixes rproc loader crash loading uninitialized elf segments or something like that. Another one makes MU unit mailboxes optional for rproc, like it was in older kernel. Without this patch I don’t know how to make both rpmsg and rproc happy sharing MU. Presence of mailbox-names property determines will rproc use MU or not. With patch applied you need something like that in DTB to make it working:

/ {

...

	reserved-memory {
		#address-cells = <1>;
		#size-cells = <1>;
		ranges;

		rpmsg_vrings: vrings0@0x8ff00000 {
//			compatible = "shared-dma-pool";
			reg = <0x8ff00000 0x100000>;
			no-map;
		};

		cm4sramcode: cm4sramcode@0x900000 {
			compatible = "shared-dma-pool";
			reg = <0x00900000 0x40000>;
			no-map;
		};

		cm4tcmcode: cm4tcmcode@0x7f8000 {
			compatible = "shared-dma-pool";
			reg = <0x007f8000 0x8000>;
			no-map;
		};

	};

/* I'm not sure regarding syscon, perhaps not required, can't retest by now */
	imx7d-cm4 {
		compatible = "fsl,imx7d-cm4", "syscon";
		clocks =  <&clks IMX7D_ARM_M4_ROOT_CLK>;

IMX7D_PLL_SYS_MAIN_240M_CLK>;
		fsl,rproc-fw-name="m4fw.bin";

		syscon = <&src>;

/*		mbox-names = "tx", "rx", "rxdb", "txdb";
		mboxes = <&mu 0 1
			  &mu 1 1
			  &mu 3 1
			  &mu 2 1>;
*/
		memory-region = <&cm4sramcode>, <&cm4tcmcode>;
	};
};

&rpmsg {
	memory-region = <&rpmsg_vrings>;
};

Hope I didn’t miss required change.

remoteproc_elf_loader.diff
imx_rproc.diff

@Edward the mailboxes and vrings are the two issues I have been struggling.

Can you re-upload the files, they are showing as missing when I try to download them.

Thanks