Program terminated with signal SIGUSR2, User defined signal 2

Software summary
------------------------------------------------------------
Bootloader:               U-Boot
Kernel version:           5.15.148-6.6.1+git.23a8e831749d #1-TorizonCore SMP PREEMPT Thu Feb 29 20:25:21 UTC 2024
Kernel command line:      root=LABEL=otaroot rootfstype=ext4 quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3 ostree=/ostree/boot.1/torizon/f5f0b9e40c1595ab904ce493a792a0b54e17f0dc3ce6832ddb889452bdd13704/0
Distro name:              NAME="TorizonCore"
Distro version:           VERSION_ID=6.6.1-build.14
Distro variant:           VARIANT="Docker"
Hostname:                 verdin-imx8mp-15229850
------------------------------------------------------------

Hardware info
------------------------------------------------------------
HW model:                 Toradex Verdin iMX8M Plus WB on Verdin Development Board
Toradex version:          0058 V1.1B
Serial number:            15229850
Processor arch:           aarch64
------------------------------------------------------------
[Hardware Configuration]
Verdin iMX8M Plus Evaluation Kit with Touchscreen
with:
SOM i.MX8M Plus Quad 4GB WB IT v1.1B
Dahlia Carrier Board v1.1D
Verdin DSI to LVDS rev 1.1A
Capacitive Touch Display 10.1" v1.0A

In short, I have developed a small skeleton of our program in C/C++ CMake using the Torizon IDE 2.4.2.

When I debug it locally using Local AMD64 profile under VS Code using WSL2 and my Ubuntu 22.04 Linux, everything works fine, and the small skeleton of our application works as expected.

Unfortunately, when I switch to Torizon ARMv8 profile for debugging on my Verdin i.MX 8M Plus under Torizon, the debugging starts which is petrfect, and hangs up with exiting because of SIGUSR2 or there is with an exception coming from SIGSEGV.

We use sisgnals in our application for communicating between threads, but we do not know why it works on a Linux distro like Ubuntu 22.04, and why we get errors or exception on the target device (Verdin SOM + Torizon).

Here is a small chuck of our program, but we can share with you the complete skeleton because this is a simple program using threading and signals, and timers.

There are a lot of comments, because this is a POC and we want to share if across our small R&D Team.

#include "main.h"


//What is the difference between sigaction and signal ? 
//sigaction and signal are both functions used in C for signal handling but they have some key differences:
// - sigaction: This is a function that allows detailed control over signal handling. It can be used to query or set the action to be taken 
//   on delivery of a specific signal. It takes three arguments: the signal number, a pointer to a struct sigaction (which specifies the new 
//   action to be taken), and a pointer to a struct sigaction (which is used to save the old action). The struct sigaction itself contains 
//   several members, including a signal handler and a sigset_t. The signal handler is a function that will be called when the signal is 
//   received, and the sigset_t specifies a set of signals to be blocked during execution of the signal handler.
// - signal: This is a simpler function that can be used to set a function to handle a signal. It takes two arguments: the signal number and 
//   a pointer to the signal handling function. The behavior of signal can vary across different systems.
//In general, sigaction is more powerful and reliable than signal, and is preferred in modern programming. 
//signal is simpler and more portable, but it has less functionality and its behavior can vary across different systems.

//What is the difference between sigemptyset and sigfillset ? 
//sigemptyset and sigfillset are both functions used in C for signal set manipulation, but they serve different purposes:
// - sigemptyset: This function initializes the signal set pointed to by its argument to empty. In other words, it removes all signals from 
//   the set. It is typically used to initialize a signal set before adding specific signals to it with sigaddset.
// - sigfillset: This function initializes the signal set pointed to by its argument to full, including all signals. In other words, it adds 
//   all signals to the set. It is typically used when you want to block all signals.
//Both functions take a pointer to a sigset_t as their argument. sigset_t is a data type used to represent a signal set.

//What is the difference between sigaddset and sigdelset ? 
//sigaddset and sigdelset are both functions used in C for signal set manipulation, but they serve different purposes:
// - sigaddset: This function adds a specified signal to a signal set. It takes two arguments: a pointer to a sigset_t (which represents the signal set), 
//   and the signal number. If the specified signal is not already in the set, sigaddset adds it.
// - sigdelset: This function removes a specified signal from a signal set. 
//   It takes the same arguments as sigaddset: a pointer to a sigset_t and the signal number. If the specified signal is in the set, sigdelset removes it.
//Both functions are used to manipulate signal sets, which are used in various signal handling functions to specify which signals to block, ignore, or handle.

//What is the difference between sigprocmask and pthread_sigmask ? 
//sigprocmask and pthread_sigmask are both functions used in C for manipulating the signal mask, but they have a key difference related to threads:
// - sigprocmask: This function is used to examine or change the signal mask of the calling process. It takes three arguments: an integer indicating how 
//   the signal mask should be changed, a pointer to a sigset_t (which specifies the set of signals to be used for the change), 
//   and a pointer to a sigset_t (which is used to save the old signal mask). 
//   However, in a multithreaded program, sigprocmask only affects the calling thread.
// - pthread_sigmask: This function is similar to sigprocmask, but it is designed to be used in multithreaded programs. It changes the signal mask of 
//   the calling thread, not the entire process. It takes the same arguments as sigprocmask.
// In summary, sigprocmask and pthread_sigmask do essentially the same thing, but sigprocmask is used in single-thread

bool keep_running = true;
PoolCop_t PoolCop;
pH_Ctrl_t pH_Ctrl;
int tickCallback = 0 ;

/* Function Definition
	ExitHandler is the function called when the signal SIGINT is raised  
	typically generated when Ctrl+C is pressed.
	This function set to false the global var keep_running which is used in
	- src/PoolCopilot/PoolCopilotMain.cpp (function PoolCopTaskLoop)
	- src/PoolCopilot/PoolCopilot.cpp (function PoolCopilot::taskthread())
	- src/PoolCopilot/PoolCopInterface.cpp (function PoolCopInterface::taskthread)
	which allows to exit from infinite loop as folllow:
	while (keep_running) {
		//Perform actions
	}
*/
void ExitHandler(int exit_signal){
	keep_running = false;
}

/* Function Definition
	CatchExitSignal sets up a signal handler for SIGINT signal which is typically 
	generated when Ctrl+C is pressed
*/
int CatchExitSignal() {
	//Signal Action Setup
	//declares a struct sigaction object named exit_action. This structure is 
	//used to describe an action to be taken in response to a specific signal.
	struct sigaction exit_action;

	//Signal Handler Assignment
	//the sa_handler field of exit_action to ExitHandler. This means that when 
	//the SIGINT signal is raised, the ExitHandler function will be called.
	exit_action.sa_handler = ExitHandler;

	//Signal Set Initialization
	//calls the sigemptyset function to initialize the signal set pointed to 
	//by &exit_action.sa_mask to empty. If the function fails, it prints an error 
	//message and returns 1.
	if (sigemptyset(&exit_action.sa_mask) != 0) {
		perror("ERROR: sigemptyset can't empty the exit signal");
		return 1;
	}

	//Signal Action Flags Setup
	//Sets the sa_flags field of exit_action to 0. This means that no special 
	//behavior is associated with the signal handler.
	//The sa_handler and sa_sigaction are both fields in the struct sigaction 
	//used for signal handling in C, but they serve different purposes:
	// - sa_handler: This is a pointer to a signal handling function. 
	//   When a signal is received, the system calls the function pointed to by 
	//   sa_handler. This function takes a single argument: the signal number.
	// - sa_sigaction: This is a pointer to a signal handling function that can 
	//   handle signal numbers, error values, and a context pointer. When a signal 
	//   is received, the system calls the function pointed to by sa_sigaction. 
	//   This function takes three arguments: the signal number, a pointer to a siginfo_t 
	//   (which can be used to get additional information about the signal), and a pointer 
	//   to a ucontext_t (which can be used to get the signal context).
	//The sa_flags field of the struct sigaction determines whether sa_handler or sa_sigaction is used. 
	//If the SA_SIGINFO flag is set, sa_sigaction is used; otherwise, sa_handler is used.
	// In summary, sa_handler is used for simple signal handling, while sa_sigaction is used 
	//when more information about the signal is required.
	exit_action.sa_flags = 0;

	//Signal Action Assignment
	//calls the sigaction function to change the action taken by the process 
	//on receipt of the SIGINT signal. If the function fails, it prints an 
	//error message and returns 1.
	if (sigaction(SIGINT, &exit_action, NULL) != 0) {
		perror("ERROR: sigaction can't catch SIGINT");
		return 1;
	}

	return 0;
}

/* Function Definition
	BlockSignals
	
*/
int BlockSignals(sigset_t *poolcoptask_sig) {
	/* Block SIGUSR1 for all threads. This signal will be used by a timer in
	 * PoolCopTask to pass a minimum time in the while loop. */
	if (sigemptyset(poolcoptask_sig) != 0) {
		perror("ERROR: sigemptyset can't empty the poolcoptask_sig signal");
		return 1;
	}
	if (sigaddset(poolcoptask_sig, SIGUSR1) != 0) {
		perror("ERROR: sigaddset can't add SIGUSR1 to the signal");
		return 1;
	}
	if (sigprocmask(SIG_BLOCK, poolcoptask_sig, NULL) != 0) {
		perror("ERROR: sigprocmask can't block the poolcoptask_sig signal");
		return 1;
	}
	return 0;
}

int main() {
	int ret = 0;

	sigset_t poolcoptask_sig;

	ret = CatchExitSignal();
	if (ret != 0) {
		return ret;
	}
	ret = BlockSignals(&poolcoptask_sig);
	if (ret != 0) {
		return ret;
	}

	std::thread PoolCopThread(PoolCopTask, poolcoptask_sig);

	PoolCopilotMain();
	if (PoolCopThread.joinable()){
		PoolCopThread.join();
	}

	sigprocmask(SIG_UNBLOCK, &poolcoptask_sig, NULL);
	return ret;
}



Many thanks for your help.

Sincerely,
François.

Hi, I’m the Toradex AI Assistant. I’m in early beta, and I would appreciate your feedback.
The Toradex Support team will follow up with you shortly!

Did you like my answer?

:+1: :-1:


Hello François,

The issue you’re encountering with SIGUSR2 and SIGSEGV signals during ARMv8 profile debugging on your Verdin i.MX 8M Plus under Torizon could be attributed to several factors. Let’s address them systematically:

  1. Signal Handling Differences: The behavior of signals and their handling can differ significantly between architectures and operating systems. While your application works as expected on Ubuntu 22.04 under WSL2, the ARMv8 environment on Torizon may have different default signal actions or signal delivery mechanisms, especially considering the real-time aspects and the specific Linux kernel configuration used by TorizonCore.

  2. SIGUSR2 Usage: SIGUSR2 is typically used for application-defined purposes. If your application uses SIGUSR2 for inter-thread communication, ensure that no other system or library component within the Torizon environment uses this signal for its purposes. Conflicts in signal usage can lead to unexpected behavior.

  3. SIGSEGV (Segmentation Fault): This signal indicates a memory access violation. If you’re encountering SIGSEGV, it suggests that your application might be trying to access memory that it’s not supposed to. This could be due to differences in memory layout, alignment requirements, or pointer handling between the AMD64 and ARMv8 architectures. Carefully review your code for any architecture-specific assumptions, especially in parts of the code that deal with memory management, pointer arithmetic, or low-level system calls.

  4. Debugging Approach:

    • Core Dumps: Enable core dumps on your Torizon device to capture the state of your application at the time of the crash. Analyzing the core dump with gdb can provide valuable insights into the cause of the SIGSEGV signal. To enable core dumps, you can use the command ulimit -c unlimited and ensure your application has the appropriate permissions to write the core file to disk.
    • Signal Handlers: Review the implementation of your signal handlers to ensure they are reentrant and do not invoke any non-async-signal-safe functions. The use of global variables (like keep_running) should be carefully managed to avoid race conditions or memory corruption.
    • Thread Safety: Ensure that your use of threads and signals is thread-safe. The interaction between threads and signals can be complex, especially regarding signal masking and handling. Since you’re using sigprocmask and pthread_sigmask, verify that signals are correctly blocked or unblocked in the appropriate threads.
  5. Hardware and Software Configuration:

Please, also check if these links can help you solve your problem:

How to Run a Hello World on the Cortex-M | Toradex Developer Center.

Hello @flepron,

It is possible that the debugger is stopping on SIGUSR2 because of a different default signal handling behavior from gdb when debugging locally and on the board.
To test this, you can add another setup command to the Torizon ARMv8 launch task on the .vscode/launch.json file:

...
{
    "name": "Torizon ARMv8",
    ...
    "setupCommands": [
        ...
        {
            "description": "Handle SIGUSR2",
            "text": "handle SIGUSR2 nostop noprint pass",
            "ignoreFailures": true
        }
    ],
    ...
},

The above change will make sure that the SIGUSR2 signal is effectively ignored by gdb and properly passed to the program.

Regarding the segmentation fault, it is hard to speculate on what may be causing it without understanding the code base.
It may even be related to gdb not passing the SIGUSR2 signal when it is expected in another thread, but this is just speculation.

If you want, we can have a look at why it runs locally without issues but crashes when debugging in a container.
For that, we will need the full project. It can be sent via the support email (support@toradex.com) or a private message to the moderators here on the community.
Please make sure to remove any credentials before sending it.

Best Regards,
Bruno

exit_action struct is on the stack. When exception happens, pointer to exception hanler stored in it, etc, is lost. At least try making it static.

Hello @bruno.tx

I’ve have tried the trick that you’ve advised to me, but unfortunately this does not work.

There is still this abnormal termination:

GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Warning: Debuggee TargetArchitecture not detected, assuming x86_64.
=cmd-param-changed,param="pagination",value="off"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".

Breakpoint 1, main () at /home/torizon/app/src/main.cpp:142
Loaded '/lib/ld-linux-aarch64.so.1'. Symbols loaded.
Loaded '/lib/aarch64-linux-gnu/libstdc++.so.6'. Symbols loaded.
Loaded '/lib/aarch64-linux-gnu/libgcc_s.so.1'. Symbols loaded.
Loaded '/lib/aarch64-linux-gnu/libc.so.6'. Symbols loaded.
Loaded '/lib/aarch64-linux-gnu/libm.so.6'. Symbols loaded.
[New Thread 0xffff92b5f180 (LWP 23)]
[New Thread 0xffff9234f180 (LWP 24)]
[New Thread 0xffff91b3f180 (LWP 25)]
[Switching to Thread 0xffff92b5f180 (LWP 23)]

Thread 2 "skeleton-poolco" hit Breakpoint 3, PoolCopTask (timer_sig=...) at /home/torizon/app/src/PoolCop/PoolCopMain.c:219
Execute debugger commands using "-exec <command>", for example "-exec info registers" will list registers in use (when GDB is the debugger)

Thread 2 "skeleton-poolco" hit Breakpoint 2, StartPoolCopTimer (timerid=0x1, its=...) at /home/torizon/app/src/PoolCop/PoolCopMain.c:119
119	in /home/torizon/app/src/PoolCop/PoolCopMain.c
PoolCopilot_thread is running
PoolCopInterface_thread is running
PoolCopTask is running
Checking pH
[Switching to Thread 0xffff93048a40 (LWP 21)]

Thread 1 "skeleton-poolco" hit Breakpoint 3, PoolCopTask (timer_sig=<error reading variable: Cannot access memory at address 0x109>) at /home/torizon/app/src/PoolCop/PoolCopMain.c:219
219	in /home/torizon/app/src/PoolCop/PoolCopMain.c
Random 1
PoolCopTask loop is ending
[Thread 0xffff93048a40 (LWP 21) exited]
[Thread 0xffff91b3f180 (LWP 25) exited]
[Thread 0xffff9234f180 (LWP 24) exited]
[Thread 0xffff92b5f180 (LWP 23) exited]
[New process 21]

Program terminated with signal SIGUSR2, User defined signal 2.
The program no longer exists.
The program '/home/torizon/app/skeleton-poolcop-app' has exited with code 0 (0x00000000).

So, I would really appreciate if you can have a look to my code base.

I will make a zip of my complete project and send it to support@toradex.com mentioning your name and the topic of this technical support thread “Program terminated with signal SIGUSR2, User defined signal 2

Many thanks for your help.

Sincerely,
François.

Hi @Edward

First thank you for your suggestion, this seems to be logical.

int CatchExitSignal() {
	//Signal Action Setup
	//declares a struct sigaction object named exit_action. This structure is 
	//used to describe an action to be taken in response to a specific signal.
	static struct sigaction exit_action;

	//Signal Handler Assignment
	//the sa_handler field of exit_action to ExitHandler. This means that when 
	//the SIGINT signal is raised, the ExitHandler function will be called.
	exit_action.sa_handler = ExitHandler;

	//Signal Set Initialization
	//calls the sigemptyset function to initialize the signal set pointed to 
	//by &exit_action.sa_mask to empty. If the function fails, it prints an error 
	//message and returns 1.
	if (sigemptyset(&exit_action.sa_mask) != 0) {
		perror("ERROR: sigemptyset can't empty the exit signal");
		return 1;
	}

	//Signal Action Flags Setup
	//Sets the sa_flags field of exit_action to 0. This means that no special 
	//behavior is associated with the signal handler.
	//The sa_handler and sa_sigaction are both fields in the struct sigaction 
	//used for signal handling in C, but they serve different purposes:
	// - sa_handler: This is a pointer to a signal handling function. 
	//   When a signal is received, the system calls the function pointed to by 
	//   sa_handler. This function takes a single argument: the signal number.
	// - sa_sigaction: This is a pointer to a signal handling function that can 
	//   handle signal numbers, error values, and a context pointer. When a signal 
	//   is received, the system calls the function pointed to by sa_sigaction. 
	//   This function takes three arguments: the signal number, a pointer to a siginfo_t 
	//   (which can be used to get additional information about the signal), and a pointer 
	//   to a ucontext_t (which can be used to get the signal context).
	//The sa_flags field of the struct sigaction determines whether sa_handler or sa_sigaction is used. 
	//If the SA_SIGINFO flag is set, sa_sigaction is used; otherwise, sa_handler is used.
	// In summary, sa_handler is used for simple signal handling, while sa_sigaction is used 
	//when more information about the signal is required.
	exit_action.sa_flags = 0;

	//Signal Action Assignment
	//calls the sigaction function to change the action taken by the process 
	//on receipt of the SIGINT signal. If the function fails, it prints an 
	//error message and returns 1.
	if (sigaction(SIGINT, &exit_action, NULL) != 0) {
		perror("ERROR: sigaction can't catch SIGINT");
		return 1;
	}

	return 0;
}

But I have still the same error:

GNU gdb (Debian 13.1-3) 13.1 Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "aarch64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word". Warning: Debuggee TargetArchitecture not detected, assuming x86_64. =cmd-param-changed,param="pagination",value="off"

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".

Breakpoint 1, main () at /home/torizon/app/src/main.cpp:143

Loaded '/lib/ld-linux-aarch64.so.1'. Symbols loaded.

Loaded '/lib/aarch64-linux-gnu/libstdc++.so.6'. Symbols loaded.

Loaded '/lib/aarch64-linux-gnu/libgcc_s.so.1'. Symbols loaded.

Loaded '/lib/aarch64-linux-gnu/libc.so.6'. Symbols loaded.

Loaded '/lib/aarch64-linux-gnu/libm.so.6'. Symbols loaded.

[New Thread 0xffff97e1f180 (LWP 23)]

PoolCopTask is running

[New Thread 0xffff9760f180 (LWP 24)]

[New Thread 0xffff96dff180 (LWP 25)]

[Thread 0xffff9760f180 (LWP 24) exited]

[Thread 0xffff97e1f180 (LWP 23) exited]

[Thread 0xffff98306a40 (LWP 21) exited]

[Thread 0xffff96dff180 (LWP 25) exited]

[New process 21]

Program terminated with signal SIGUSR2, User defined signal 2.

The program no longer exists.

The program '/home/torizon/app/skeleton-poolcop-app' has exited with code 0 (0x00000000).

I’ve even tried to set exit_action as a global var, but this did not correct the error as well.

Many thanks anyway.

Sincerely,
François.

Hi,

I didn’t see all of your codes but sharing variables among multithreads in a multiprocessor environment could be tricky.

For example keep_running variable that you have.
Lets say you set it to false in your ExitHandler executed on CPU0 (for example), How do you think your other threads runs on the CPU1 notice this change since each cpu cores have internal caches to access same variable? Somebody has to tell the other cpus that the variable has changed and internal cache has to be invalidated.

This is an interesting remark.
I was not aware of that cache.
One thing is sure, there is only one thread which sets keep_running to false.

Do you think that declaring keep_running as volatile could force the core to remove the caching for this variable ?

Many thanks.

Sincerely,
François.

no, volatile affects only c compiler’s code generation.
you need to add memory barriers for the variables that you want to share among threads.