RPMsg task stuck in rpmsg_rtos_init on M4

There seems to be a lot a things to be taken into consideration while implementing RPMsg with a customized configuration.

After reading some topics on catastrophic code performances in non-cached DDR for M4 code (and potential errors in atomic operations using cached DDR…) , we decided to use the following custom linker script based on TCM one :
[Edit according to comment from Mar 18]

/* Specify the memory areas */
/* m_text : almost 256kB of cacheable OCRAM  (EPDC + 128k - 4KB used by kernel */
/* m_data : 64kB of TCML + TCMU (avoid cache errors on atomic operations in DDR) */
MEMORY
{
  m_interrupts          (RX)  : ORIGIN = 0x00000000, LENGTH = 0x00000240
  m_text                (RX)  : ORIGIN = 0x20201000, LENGTH = 0x0003F000
  m_data                (RW)  : ORIGIN = 0x1FFF8000, LENGTH = 0x00010000
}

It suits our needs at the moment (~55KB data placed in TCM and ~140KB code placed in OCRAM)

Note : we first placed code at the begining of the 128kB OCRAM and then experienced linux not booting because of that, so we unexpectedly discovered linux was using the first 4KB of OCRAM for low power mode (even if we don’t use it and made no reference to it in our DTS).

Anyway, A7 - linux side seems to start RPMsg driver fine :

But on M4 - FreeRTOS side, RPMsg task enters rpmsg_rtos_init() and never comes out of it… until MU interruption triggers for the 3rd time and then the whole M4 crashes !

We’ve tried then reverted a lot of different munipulations in order to have RPMsg work, such as :

  • changing the VRING_BASE addresses in middleware/multicore/open-amp/porting/imx7d_m4
    /platform_info.c with a patch.

  • accordingly moving rpmsg reg memory in DTS

  • trying to prevent linux from allocating RAM in our RPMsg memory space (we’ve seen people allocating up to 1MB of reserved space for RPMsg, isn’t Uboot doing this on it’s own ?)

  • checking if everything is ok in SystemInit() method in platform/devices/MCIMX7D/startup/system_MCIMX7D_M4.c (this one is really odd though, why is memory region 4 overlapping with linux kernel and set as non-shared ??)alt text


SUM UP

So, actually this is what we have, and doesn’t work :
DTS, memory

alt text

We both limited linux RAM by whitelisting memory ranges to use (usable-memory) as seen in some patches, AND blacklisting not to use (reserved-memory, no-map) as seen in other patches.

DTS, rpmsg

alt text

platform_info.c

alt text

All MU configuration and inits methods are called in the right order, like in rpmsg_str_echo example !

At this point, we start running out of potential solutions…
Any hints would be greatly appreciated !

Woops, actual linker we use is this one (we mentionned the first 4KB of OCRAM but the screenshot wasn’t containing the patch)
alt text

Also noticed on this developer Toradex page that the message of succesful RPMsg driver on linux side isn’t the same :

Ours → imx_rpmsg_tty virtio0.rpmsg-openamp-demo-channel.-1.0: new channel: 0x400 → 0x0!

Other → imx_rpmsg_tty rpmsg0: new channel: 0x400 → 0x0!

Was this changed in a recent commit or could this indicate an error in initialization on linux side ?

hi @nicolas.b

We are looking into this issue and come soon back to you.

Meanwhile could you provide the following Information:

Which version of the Software (Bsp) of your module and which carrier board are you using?

Best regards,
Jaski

Hi @jaski.tx

I mentionned our config in this thread :

I’ll copy-paste it here.

I’m not the one in charge of the linux part but we use buildroot and use :

git clone -b 2016.11-toradex git://git.toradex.com/u-boot-toradex.git/
git clone -b toradex_4.9-1.0.x-imx git://git.toradex.com/linux-toradex.git
git clone -b colibri-imx7-m4-freertos-v8 git://git.toradex.com/freertos-toradex.git

Last update of these on our side was in september 2018.

We have a custom carrier board.

Best regards,

Nicolas

Hi @nicolas.b I don’t know if this info can help, but in my application which runs on VF61 (and not iMX7), on M4 - FreeRTOS side, RPMsg task enters rpmsg_rtos_init() and never comes out of it if the sources for M4 are build in DEBUG configuration.

If the sources for M4 are built in RELEASE, M4 core executes rpmsg_rtos_init().

I have no idea why this can happen.

I don’t know if you build the sources in RELEASE and/or DEBUG but maybe you can try.

Hi @vix, thank you for this information.

Actually, we made our own Makefile and it has a single configuration. It always includes debug symbols (-g3) and no optimization (-O0).

Just to be sure it wasn’t the same problem as yours we tested a compilation with all the flags of CMakeLists.txt from rpmsg-str_echo_freertos example, in release configuration (removed the -g3 and changed optimization to -Os).

We’re still stuck in the same method.

The difference for me is this one:

  • DEBUG (not working): -O0
  • RELEASE (working): -O2 and -DNDEBUG

Maybe NDEBUG macro can be the problem?
As far as I know it impacts how some low-level fuctions are handles (assert(), …)

Can you try definining NDEBUG macro too?

Sorry, forgot to mention we also added -D__NDEBUG macro

Could you try defining -DNDEBUG (without hte double underscore)?

Already did, tested whether it changed something or not, it didn’t.

It would be strange if it changed anything though, we got rid of the assert() methods…

I don’t know which toolchain you use for M4 firmware.
I use Keil/ARM toolchain and I’m in this situation.

We use the one suggested by toradex : gcc-arm-none-eabi-4_9-2015q3

Adding -D NDEBUG reduced the size of our output file (guess it successfully removed all the assert-related code) so our problem doesn’t come from an assert.

If it can help, I add the full DTS, kernel config and resulting DTB here : link text

Hi @nicolas.b

I add news from my case, but I don’t know if this can help you.

I discovered what what the real issue in my DEBUG configuration.

It was not an assert(), but a kind of heap and/or stack overflow. Maybe in DEBUG either more variables are allocated on the heap, or a deeper stack is used.

I discovered this after I checked the validity of the pointer returned by all my calloc()/malloc().

I saw that some of them returned NULL, and then I dereferenced it in my code (without any additional check - my fault).

I had to increase the space for the heap, and now I can successfully run both DEBUG and RELEASE.

Afterwards I discovered a strange thing on a portion of OCRAM starting from 0x3F060000 (as described here) and I need help from Toradex engineers.

Hi @vix

Thanks for the news !
On our side we still don’t have the means to debug our app so we really need their help too !

hi @nicolas.b

Unfortunately we don’t have further Information yet. We will come back to you during this week.

Best regards,
Jaski

Hi @nicolas.b

if you have a JTAG debugger (I have Keil ULINK Pro) you can try to connect to M4 core in “connect-only mode” and see what happens to the core.

This was extremely useful to me.

hi @vix

We’ll soon receive a JTAG debugger to investigate further on this matter.
Thank you for the advice, we’ll try this !

Hi @nicolas.b

sure JTAG is really helpful to debug this kind of issues.

Another suggestion that comes to my mind (since I’ve lost a lot of days in the past) is to double check all the printf() functions in M4 firmware.

If there are some of them (for debug purposes, as an example) and the “User I/O” has been redirected to UART, the data is sent to the UART, but the UART must be properly initialized (i.e. hardware configuration).

If the UART has not been properly configured, the M4 waits forever inside printf().