[solved] Colibri imx6 memory allocation error

Dear community, I have faced a strange behavior on colibri imx6 module.
Sometimes (I am unable to reproduce systematically the error) the board hangs completely and there is no way to interact with the system. When this issue occurs even after power off and then power on the board the problem could return after the boot sequence completed.

When this error happens it starts with a process that runs in memory allocation failure, then many other processes report the same error, but monitoring the situation with htop it seems that there is a huge amount of free memory.

Here is the complete log of the error.

Looking around with google it could be an issue related to the kernel memory allocated and someone suggest to fine tune the memory used by the kernel, but I am not sure on how to make this changes on the colibri-imx6.

As you can see in the log file I am using the latest stable kernel from the 2.6.1 BSP.

Any suggestion will be appreciated.

Best regards,
Stefano.

Please note that V2.6.1 is not considered a stable BSP but rather V2.6 is! And the latest stable BSP would actually be 2.7 at the time of this writing. Have a look at our release roadmap as well.

It seems that memory allocation in a interrupt context failed. Linux keeps some memory free so that the kernel can allocate it during interrupt context. In this case order 0 even run out, which is the smallest size of blocks available in the allocator (see also Physical Page Allocation). In other words, memory reserved for interrupt context seemed to really depleted completely.

In your case it seems that the rtl8192cu driver was involved. Increasing the reserved size might help alleviate the problem:

sysctl -w "vm.min_free_kbytes=65536"

However, if the system runs out of memory due to a bug, e.g. that the kernel does not handle a interrupt properly and allocates memory in a loop, this likely only will delay the issue.

If this does not help, I recommend trying the newer 2.7 stable BSP which comes with Linux 4.1.

Thank you Stefan for the explanation.

I saw that the rtl8192cu driver was in the stack, but I don’t think that it is the main root cause, probably some other process starts to earn memory and don’t release it.

Are there any method to set the vm.min_free_kbytes to 65536 on yocto?
In this way I don’t have to manually change on new installations.

To set such values you typically use a custom sysctl.conf. In OpenEmbedded this file seems to be part of procps, so you would need to create a bbappend for that recipe and provide a custom sysctl.conf (you can find the recipe in openembedded-core/meta/recipes-extended/procps).

Hi Stefan, unfortunately it seems that your suggestion doesn’t work for me…
I have tried to identify the root cause of the loop and I saw that I have a two processes that require a lot of memory, but the system starts to kill them even if there is about half of the system memory free. I read about this problem and it also known as memory fragmentation, and from my point of view is so heavy on my system because the whole memory is configured as DMA memory so the kernel must allocate it contiguously.

My question is: are there any way to tune the memory in order to reserve a portion to be non-DMA in order to let the kernel allocate it in non-contiguous way?

I attach the log of the oom crash for my process link text.
I have also try to tune the system with the vm.extfrag_threshold reducing form the default 500 to 100, but nothing changes.

Any help will be appreciated,
Stefano.

There are different classes of memory, and Linux tries to use them as smart as possible. By default the Linux kernel 3.14.52 as configured for Colibri iMX6 reserves 50% of memory for the CMA (continous memory allocator) or 256MiB, which ever is less. In your case this lead to 128MiB being reserved for the continous memory allocator. As far as I understand this memory can still be used for user space processes, however, it seems not to be usable for regular kernel memory.

As far as I know CMA is rather high by default on i.MX6 because the graphics subsystem allocates form this area too. Depending on your needs, and especially if you don’t use graphics, it makes sense to lower this value.

You can change the value using a kernel parameter cma. E.g. cma=32M should allocate only 32MiB of memory for the Continous Memory Allocator.

Note if you use a newer kernels you might need to adjust the CMA size in the device tree.

Hi Stefan, thank you very much for the hint.

Setting the CMA to 32MB solve our problem.

Best regards,
Stefano.

Perfect that it works. Thanks for the feedback.
Best regards
Jaski