Kernel crash triggered by rapid systemd service restarts on Verdin AM62 (6.6.138)

Technical Support
, ,
2

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.

Thank you for the detailed report. This is indeed a serious kernel stability issue affecting your Verdin AM62 system running BSP 7.6.1-devel. Based on your stack traces and the reproducible nature tied to systemd service restarts, this appears to be a memory management issue in the kernel’s page allocation subsystem.

Analysis of the Issue

The kernel panics you’re experiencing show consistent patterns:

  1. Memory allocation failures in rmqueue_bulk+0x384/0x610 and related memory management functions
  2. NULL pointer dereferences at virtual address 0000000000000208
  3. Alignment faults in cgroup management code (css_killed_work_fn)
  4. PID 1 (systemd) involvement in the latest crash, confirming the systemd correlation

The fact that running the same failing script in a shell loop doesn’t reproduce the issue strongly suggests this is related to systemd’s process management, cgroup handling, or service lifecycle management rather than the script itself.

Immediate Recommendations

1. Verify BSP Version and Consider Downgrade

Your system is running BSP 7.6.1-devel, which according to the Embedded Linux Release Matrix(1), is a development version. For production stability, consider using a stable release like BSP 6.8.0(1).

2. Enable Enhanced Logging

To gather more diagnostic information, you can enable kernel debugging as described in the community thread(2):

echo "module kernel +p" > /sys/kernel/debug/dynamic_debug/control

3. Memory Pressure Testing

Since the crashes involve memory allocation failures, test your system under different memory conditions:

  • Monitor memory usage during the service restart cycles
  • Check for memory leaks in systemd or related processes

4. Kernel Configuration Review

If you’re building a custom kernel(3), ensure your kernel configuration includes proper debugging options:

  • CONFIG_DEBUG_KERNEL=y
  • CONFIG_PRINTK=y
  • CONFIG_DYNAMIC_DEBUG=y

Next Steps

  1. Document the exact reproduction steps including timing and system load conditions
  2. Collect more detailed logs with the enhanced debugging enabled
  3. Test with a minimal Toradex reference image when possible to isolate custom layer contributions
  4. Monitor systemd journal for any service management anomalies during the restart cycles

This issue appears to be a kernel-level bug that may require upstream attention. The consistent memory allocation failures combined with systemd involvement suggest a race condition or resource leak in the process lifecycle management.

Would you be able to test this with a clean Toradex reference image to help isolate whether this is related to your custom layers or a core BSP issue?

Best regards, Toradex AI Assistant.

Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.