Verdin Mini RPMSG-Lite Increase Buffer Size

Hi,
This is a follow up question to this question where I managed to adapt the rpmsg driver for two channels. Now I want to increase the buffer size, since 512-byte payloads are not fast enough (I need around 200kBytes/s, looks like rpmsg can’t follow this speed with default payload size).

Looking at the driver on the Linux side (rpmsg « drivers - linux-toradex.git - Linux kernel for Apalis, Colibri and Verdin modules) there are several places where the size can be changed. Inside imx_rpmsg.c :

#define RPMSG_NUM_BUFS		(512)
#define RPMSG_BUF_SIZE		(512)
#define RPMSG_BUFS_SPACE	(RPMSG_NUM_BUFS * RPMSG_BUF_SIZE)
#define RPMSG_VRING_ALIGN	(4096)
#define RPMSG_RING_SIZE	((DIV_ROUND_UP(vring_size(RPMSG_NUM_BUFS / 2, \
				RPMSG_VRING_ALIGN), PAGE_SIZE)) * PAGE_SIZE)

Inside imx_rpmsg_tty.c :

/* this needs to be less then (RPMSG_BUF_SIZE - sizeof(struct rpmsg_hdr)) */
#define RPMSG_MAX_SIZE		256

and inside virtio_rpmsg_bus.c :

/*
 * We're allocating buffers of 512 bytes each for communications. The
 * number of buffers will be computed from the number of buffers supported
 * by the vring, upto a maximum of 512 buffers (256 in each direction).
 *
 * Each buffer will have 16 bytes for the msg header and 496 bytes for
 * the payload.
 *
 * This will utilize a maximum total space of 256KB for the buffers.
 *
 * We might also want to add support for user-provided buffers in time.
 * This will allow bigger buffer size flexibility, and can also be used
 * to achieve zero-copy messaging.
 *
 * Note that these numbers are purely a decision of this driver - we
 * can change this without changing anything in the firmware of the remote
 * processor.
 */
#define MAX_RPMSG_NUM_BUFS	(512)
#define MAX_RPMSG_BUF_SIZE	(512)

On the M4 side, there is rpmsg_config.h :

//! @def RL_BUFFER_PAYLOAD_SIZE
//!
//! Size of the buffer payload, it must be equal to (240, 496, 1008, ...)
//! [2^n - 16]. Ensure the same value is defined on both sides of rpmsg
//! communication. The default value is 496U.
#define RL_BUFFER_PAYLOAD_SIZE (496U)

//! @def RL_BUFFER_COUNT
//!
//! Number of the buffers, it must be power of two (2, 4, ...).
//! The default value is 2U.
//! Note this value defines the buffer count for one direction of the rpmsg
//! communication only, i.e. if the default value of 2 is used
//! in rpmsg_config.h files for the master and the remote side, 4 buffers
//! in total are created in the shared memory.
#define RL_BUFFER_COUNT (256)

Needless to say I find the naming system confusing, since buffer count sometimes reffers to whole ring buffer, and sometimes to one half (tx or rx half). Since I allocate 128kb of memory for the ring buffers in my device tree, if I double the buffer size I also divide the buffer count in half (512x512 to 1024x256 etc.).

My first tests show: changing the linux side does nothing, M4 still receives 496
bytes hello world message. As soon as I change the rpmsg_config.h on the M4 side, even just halving the number of buffers to 128, hello world message is not received anymore and the driver isn’t working.

Thanks in advance!

Hi @swiss,

Unfortunately, I haven’t tried that myself yet. Let me investigate the code a little further to check if I have some insights that might help you. Meanwhile, I found some NXP threads about this issue, please check them below:

Hope this helps to start the development.

Best Regards,
Hiago.

Hi @hfranco.tx ,

Thanks for the threads, I already saw them all before starting this thread. Unfortunatelly they are mostly refering to some older versions of rpmsg that are slightly different.
The one that actually looks similiar is this one where it is left unaswered, I commented there that I’m having a similiar issue in hope that maybe someone solved in the meantime.

I will also post a new question concerning this on the nxp forum.

Hi

This ^^ comment in virtio_rpmsg_bus is misleading. If you look at define from imx-rpmsg.c:

, which for 512 * 512B buffers … fits in 64k. It can be so only if amount of buffers applies to minimum message length, which would consist of just RPMSG header. So two rings * 16 * 512 = 16k. Add here buffers head, tail and other information, then it still may fit 64k. So RPMSG buffer size is not about real buffer size but about max message size.

Yes, M4 side with RL_BUFFER_COUNT define for some reason talks not about total buffers count like in Linux driver, but about buffers count in one direction (rx or tx).

Buffer size and amount of buffer has to match Linux vs M4. This is why:

Since you are already familiar with what’s where in Linux and M4, you miss last place where imx-rpmsg.c driver is further hardcoded. See lines 307-308 here

			rpdev->ivdev[i]->vring[0] = start;
			rpdev->ivdev[i]->vring[1] = start + 0x8000;

0x8000 is hardcoded one direction vring size. One ring at start, another ring at start + 0x8000. i is virtual device index.

Hope this helps

Edward

1 Like

Hi Edward,
Thanks for your answer!

Wait if I understand you correctly 512x512 is 256kB but the vring size in one direction is only 0x8000 as seen here:

So the RPMSG_RING_SIZE is not refering to the payload size but rather overhead information?

But let’s go back to the concrete example: If I want to go from 512x512 to 1028x256, should I change the hardcoded 0x8000 ring size? I expected it to work without change because its still 256kB as before.

EDIT: I just changed one print output for debugging (imx_rpmsg.c line 670 “imx rpmsg driver is registered”) but nothing changed, looks like the driver changes are not applied. I might be making some torizoncore builder mistake, I’m looking into it

256kB in comment in virtio_rpmsg_bus.c is totally misleading. Vring size isn’t multiple num_bufs * buf_size.
RPMSG_RING_SIZE is what fits vring size.

“Payload size”, if you meant RPMSG_BUF_SIZE, must using several tiny buffers from total amount of those tiny RPMSG_NUM_BUFS. Else 512*512 couldn’t fit 64k.

Yes, of course you have to change 0x8000. Please forget about misleading wrong number 256kB. It’s

Taking your own settings RPMSG_NUM_BUFS and RPMSG_BUF_SIZE, calculated RPMSG_RING_SIZE has to fit 20x8000=64k. And of course you new settings have to agree with device tree and M4 side settings for VRING starting addresses and VRING sizes.
For more or bigger buffers most likely you need to increase that magic 0x8000. On M4 this magic number is in VRING_SIZE define. Accordingly your adjacent M4 VDENx_VRING_BASE defines should differ by 2
VRING_SIZE.

1 Like

All clear, in order to avoid changing the magic number :slight_smile: if I use 1k buffers I can use 64 of them per direction to stay inside 0x8000, otherwise change the hardcoded size. Thank you!

Now I need @hfranco.tx 's help on building of the kernel module, it can be that my makefile or the kconfig file needs additional changes, I am trying to find similiar thread but no luck so far. Also very little experience with them. This is my makefile:

SRC := $(shell pwd)

all:
	$(MAKE) -C $(KERNEL_SRC) M=$(SRC) modules

clean:
	$(MAKE) -C $(KERNEL_SRC) M=$(SRC) clean

obj-$(CONFIG_RPMSG)		+= rpmsg_core.o
obj-$(CONFIG_RPMSG_CHAR)	+= rpmsg_char.o
obj-$(CONFIG_RPMSG_QCOM_GLINK_RPM) += qcom_glink_rpm.o
obj-$(CONFIG_RPMSG_QCOM_GLINK_NATIVE) += qcom_glink_native.o
obj-$(CONFIG_RPMSG_QCOM_GLINK_SMEM) += qcom_glink_smem.o
obj-$(CONFIG_RPMSG_QCOM_SMD)	+= qcom_smd.o
obj-$(CONFIG_RPMSG_VIRTIO)	+= virtio_rpmsg_bus.o
obj-$(CONFIG_HAVE_IMX_RPMSG)	+= imx_rpmsg.o
obj-$(CONFIG_IMX_RPMSG_PINGPONG)	+= imx_rpmsg_pingpong.o
obj-$(CONFIG_IMX_RPMSG_TTY)	+= imx_rpmsg_tty.o

And the build output from the torizoncore builder:

=> Building module located at 'rpmsg/'
make: Entering directory '/workdir/rpmsg'
make -C /storage/linux M=/workdir/rpmsg modules
make[1]: Entering directory '/storage/linux'
  CC [M]  /workdir/rpmsg/imx_rpmsg_pingpong.o
  CC [M]  /workdir/rpmsg/imx_rpmsg_tty.o
  Building modules, stage 2.
  MODPOST 2 modules
  CC [M]  /workdir/rpmsg/imx_rpmsg_pingpong.mod.o
  LD [M]  /workdir/rpmsg/imx_rpmsg_pingpong.ko
  CC [M]  /workdir/rpmsg/imx_rpmsg_tty.mod.o
  LD [M]  /workdir/rpmsg/imx_rpmsg_tty.ko
make[1]: Leaving directory '/storage/linux'
make: Leaving directory '/workdir/rpmsg'

Kernel module(s) successfully built and ready to deploy.
rpmsg/imx_rpmsg_tty.ko is set to be autoloaded on boot.
rpmsg/imx_rpmsg_pingpong.ko is set to be autoloaded on boot.
All kernel module(s) have been built and prepared.

EDIT: I can see the changes applied to the imx_rpmsg_tty.c file, but not to the imx_rpmsg.c file

Hi @swiss,

Can you give more details about the steps you are taking to modify these files?

Are you checking the modification inside the kernel source code?

Best Regards,
Hiago.

Hi @hfranco.tx ,
I am changing the kernel prints. For example, inside imx_rpmsg_tty.c line 17:

#define MSG		"hello world!2"

changes the output in the kernel when the message is received.
But when changing the imx_rpmsg.c line 670:

pr_info("imx rpmsg driver is registered2.\n");

or virtio_rpmsg_bus.c line 1298:

dev_info(&vdev->dev, "rpmsg host is online2\n");

the changes don’t come up in the kernel output.

Furthermore I as you see from the builder output here

using “modules” command in the Makefile doesn’t even compile other .c files, there are only two object (.o) files created. I then changed the makefile to:

obj-$(CONFIG_RPMSG)	 += rpmsg_core.o
obj-$(CONFIG_RPMSG_CHAR)+= rpmsg_char.o
obj-$(CONFIG_RPMSG_QCOM_GLINK_NATIVE)+= qcom_glink_native.o
obj-$(CONFIG_RPMSG_QCOM_GLINK_RPM)+= qcom_glink_rpm.o
obj-$(CONFIG_RPMSG_QCOM_GLINK_SMEM)+= qcom_glink_smem.o
obj-$(CONFIG_RPMSG_QCOM_SMD)+= qcom_smd.o
obj-$(RPMSG_VIRTIO) += virtio_rpmsg_bus.o
obj-$(CONFIG_HAVE_IMX_RPMSG) += imx_rpmsg.o
obj-$(CONFIG_IMX_RPMSG_PINGPONG)+= imx_rpmsg_pingpong.o
obj-$(CONFIG_IMX_RPMSG_TTY)+= imx_rpmsg_tty.o

SRC := $(shell pwd)

all:
	$(MAKE) -C $(KERNEL_SRC) M=$(SRC)
	$(MAKE) -C $(KERNEL_SRC) M=$(SRC) modules

clean:
	$(MAKE) -C $(KERNEL_SRC) M=$(SRC) clean

This additional line before “modules” actually compiles the other files but they are probably still not linked. The new torizoncore builder output looks like this:

=> Building module located at 'rpmsg/'
make: Entering directory '/workdir/rpmsg'
make -C /storage/linux M=/workdir/rpmsg
make[1]: Entering directory '/storage/linux'
  CC      /workdir/rpmsg/rpmsg_core.o
  CC      /workdir/rpmsg/qcom_glink_native.o
  CC      /workdir/rpmsg/qcom_glink_rpm.o
  CC      /workdir/rpmsg/imx_rpmsg.o
In file included from ./include/linux/printk.h:7,
                 from ./include/linux/kernel.h:15,
                 from ./include/linux/delay.h:22,
                 from /workdir/rpmsg/imx_rpmsg.c:7:
/workdir/rpmsg/imx_rpmsg.c: In function ‘imx_rpmsg_init’:
./include/linux/kern_levels.h:5:18: warning: too many arguments for format [-Wformat-extra-args]
    5 | #define KERN_SOH "\001"  /* ASCII Start Of Header */
      |                  ^~~~~~
./include/linux/kern_levels.h:14:19: note: in expansion of macro ‘KERN_SOH’
   14 | #define KERN_INFO KERN_SOH "6" /* informational */
      |                   ^~~~~~~~
./include/linux/printk.h:306:9: note: in expansion of macro ‘KERN_INFO’
  306 |  printk(KERN_INFO pr_fmt(fmt), ##__VA_ARGS__)
      |         ^~~~~~~~~
/workdir/rpmsg/imx_rpmsg.c:670:3: note: in expansion of macro ‘pr_info’
  670 |   pr_info("imx rpmsg driver is registered2\n");
      |   ^~~~~~~
  AR      /workdir/rpmsg/built-in.a
  CC [M]  /workdir/rpmsg/imx_rpmsg_pingpong.o
  CC [M]  /workdir/rpmsg/imx_rpmsg_tty.o
  Building modules, stage 2.
  MODPOST 2 modules
  CC [M]  /workdir/rpmsg/imx_rpmsg_pingpong.mod.o
  LD [M]  /workdir/rpmsg/imx_rpmsg_pingpong.ko
  CC [M]  /workdir/rpmsg/imx_rpmsg_tty.mod.o
  LD [M]  /workdir/rpmsg/imx_rpmsg_tty.ko
make[1]: Leaving directory '/storage/linux'
make -C /storage/linux M=/workdir/rpmsg modules
make[1]: Entering directory '/storage/linux'
  Building modules, stage 2.
  MODPOST 2 modules
make[1]: Leaving directory '/storage/linux'
make: Leaving directory '/workdir/rpmsg'

Kernel module(s) successfully built and ready to deploy.
rpmsg/imx_rpmsg_tty.ko is set to be autoloaded on boot.
rpmsg/imx_rpmsg_pingpong.ko is set to be autoloaded on boot.
All kernel module(s) have been built and prepared.

In the warnings you can see that my changed line of imx_rpmsg.c shows up, so it is compiled, just still somehow not linked.
I couldn’t find any torizoncore builder example with more then one .c file unfortunatelly.

No, I’m only checking the cheksum of the build that the right kernel is deployed. How can I check the source code?

Hi @swiss,

Ok, I got the problem here. TorizonCore Builder is able to create/modify kernel modules only, not the kernel code itself. imx_rpmsg_tty.c is the kernel module, so you’re able to modify it without problems. imx_rpmsg.c though, is the source code responsible to deal to manage the rpmsg, this is not a kernel module, it’s the built-in driver from Linux kernel. Therefore, it’s not possible to edit it (only imx_rpmsg_tty.c, as you saw).

So in this case, you will have to patch the kernel using Yocto Project, unfortunately. This is not straightforward if you haven’t used Yocto before. Here are the links for getting started with Yocto:

I you want to try Yocto, I can help you with that by creating a custom layer that will make those modifications to your TorizonCore kernel. However keep in mind that on every TorizonCore release, you will need to patch the kernel itself again and create your own image, which is not a problem itself, depending on how much time/effort you want to put into this project.

Unfortunately, as this requires some deep kernel changes, there is nothing we can do to avoid building TorizonCore again.

Let me know if you want any help with that.

Best Regards,
Hiago.

Hi @hfranco.tx ,
That’s really unfortunate to hear.
Since I have zero experience with this I would rather try to avoid this. I basically only need to increase the speed of the rpmsg to 200kBytes/s, which is around 2-3 times faster than what I am achieving right now. I think I will just try splitting my samples into 3-4 rpmsg channels since this can be achieved without changing the kernel.

In another topic you also mentioned that avoiding the tty could be even better:

Again no experience here but still sounds like a better solution than learning yocto etc.

Hi @swiss,

If you send me the patches I could create a custom Yocto TorizonCore for you to test, but again, you would have to apply those changes with Yocto in the future.

Although I mentioned more ways to deal with ttyRPMSG, I believe that splitting in more channels or increasing the buffer size would also need some kernel patches, correct? If that’s the case, then Yocto is needed, if not, then you should be fine without Yocto here.

Let me know if you need any help in setting up a custom layer for your build, I’ll be glad to help you with that.

Best Regards,
Hiago.

Hi @hfranco.tx,

Actually increasing the number of channels works without kernel changes, since they are only inside the tty file.

Thank you, I will contact you about this if there is no other viable option.

1 Like

Hi @swiss,

Yeah sorry, you’re correct. In this case, using Torizon Core Builder should be enough.

Sure! Feel free to send any requests here.

Best Regards,
Hiago.