Hi there,
I have an application (in a docker container) that crashes very rarely. Since I cannot reproduce the problem with debugging, I would like to get a core dump for the crash.
However, I have been trying different approaches but somehow I am still missing something.
I checked this link: Enable and Analyse Core Dumps in Linux | Toradex Developer Center. But this doesn’t seem to work for torizon and I guess the description is intended for applications that don’t run in a container.
For containers I guess one will need to adapt the docker compose file and also configure the Torizon Application Properties (I am using Visual Studio with the Toradex extension).
At the moment I cannot even get a normal program outside the container (e.g. nano) to dump when I kill it with a segmentation fault “kill -11 [PID]” - or I can’t find where it puts the core dump.
Can you help me with this?
Creating core dumps from an application within docker is certainly possible.
As you correctly pointed out, the page on the developer website is not intended for Docker.
However, the general procedure is similar, you just need to make sure that your container is ran with the following additions:
--init to make sure signals are handled properly
--ulimit core=-1 which is equivalent of ulimit -c unlimited, but for the container
-v /tmp:/tmp so that the destination directory for the core dump (/tmp) is shared between the container and host system
A possible resulting docker run would be the following:
docker run -d --rm --init --ulimit core=-1 -v /tmp:/tmp --name=test test
In a docker-compose format:
version: '3.8'
services:
test:
image: test
init: true
ulimits:
core: -1
volumes:
- /tmp:/tmp
container_name: test
When testing, please also make sure that the termination signal is sent to the actual process <executable-name>, not its /sbin/docker-init -- <executable-name> counterpart.
Thanks for your answer @bruno.tx ,
I tried as you said, but no core dump was created.
Addtionally I also set “ulimit -c unlimited” on the host machine and added “echo “/tmp/core.%e.%p” | sudo tee /proc/sys/kernel/core_pattern”. But no core dump. I think usually one would see a line “(core dumped)” in journalctl. But I cannot see that either. I start to have the feeling that torizon doesn’t support this. Or do you have another idea what I could be doing wrong?
Sorry for my late answer, usually I get an email telling me that someone has answered my question.
In the meantime I found another way how to localize the cause of my problem (using address sanitizer compiling options).
Anyway, I want to give you feedback on your solution. For me it still doesn’t work; I adjusted the docker compose and copied all your commands. I added a part in my program that forces a segmentation fault to trigger the core dump creation. The program crashes, but the core dump isn’t there.
For others having the same problem, here’s my solution using address sanitizer:
I am using Visual Studio with the Toradex extension. Go to the project properties and enable address sanitizer:
In the Torizon Application properties you will also need to add the libasan5 package to extrapackages. With these settings you will get a nice output in the journalctl log pointing you in the right direction.
