Kernel crash triggered by rapid systemd service restarts on Verdin AM62 (6.6.138)

Technical Support
, ,
1

root@verdin-am62-15599560:~# tdx-info

Software summary

Bootloader: U-Boot
Kernel version: 6.6.138-7.6.1-devel #1 SMP PREEMPT Fri May 8 07:27:41 UTC 2026
Kernel command line: root=PARTUUID=076c4a2a-01 ro rootwait console=tty1 console=ttyS2,115200 rauc.slot=A
Distro name: NAME=“TDX Wayland with XWayland”
Distro version: VERSION_ID=7.6.1-devel-20260603093705-build.0
Distro variant: -
Hostname: verdin-am62-15599560

Hardware info

HW model: Toradex Verdin AM62 WB on Verdin Development Board
Toradex version: 0072 V1.2A
Serial number: 15599560
Processor arch: aarch64

Since switching from 7.2.0 to 7.6.1 I experience regular kernel panics/crashes I tracked the problem to systemd causing the issues/crashes one of the stack traces even shows pid 1.
It is easy to reproduce you need a system script like this:

[Unit]
Description=crashing
After=network.target
After=syslog.target

[Service]
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/bin/bash /root/fail.sh

[Install]
WantedBy=multi-user.target

fail.sh contains this:

#/usr/bin/bash
exit 1

the crash can happen in as short a time as 11 minutes or as long as 8+ hours.

I tried running the fail.sh in a tight loop in the shell for 8+ hours without fail, it really is systemd related.

here are some stack traces i encountered
The first two still include my custom drivers, in the third i removed them (modprobe)
On the latest stack trace systemd showed a restart counter of 209


[ 156.069268] am65-cpsw-nuss 8000000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx 
[50660.525445] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000208 
[50660.534435] Mem abort info: 
[50660.537266] ESR = 0x0000000096000045 
[50660.541063] EC = 0x25: DABT (current EL), IL = 32 bits 
[50660.546443] SET = 0, FnV = 0 
[50660.549537] EA = 0, S1PTW = 0 
[50660.552720] FSC = 0x05: level 1 translation fault 
[50660.557658] Data abort info: 
[50660.560573] ISV = 0, ISS = 0x00000045, ISS2 = 0x00000000 
[50660.567699] CM = 0, WnR = 1, TnD = 0, TagAccess = 0 
[50660.574335] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 
[50660.581192] user pgtable: 4k pages, 39-bit VAs, pgdp=0000000087d0b000 
[50660.589226] [0000000000000208] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000 
[50660.601115] Internal error: Oops: 0000000096000045 [#1] PREEMPT SMP 
[50660.609039] Modules linked in: nls_iso8859_1 nls_cp437 option usb_wwan qmi_wwan usbserial usbnet onboard_usb_hub optee_rng rng_core rpmsg_ctrl rpmsg_char dwc3 evdev aes_ce_cipher ghash_ce gf128mul sha2_ce sha256_arm64 sha1_ce snd_soc_simple_card snd_soc_simple_card_utils optee leds_gpio led_class tee gpio_keys spi_cadence_quadspi usb_conn_gpio roles ti_k3_r5_remoteproc virtio_rpmsg_bus rpmsg_ns dwc3_am62 rpmsg_core k3_j72xx_bandgap ti_k3_m4_remoteproc rtc_ti_k3 snd_soc_davinci_mcasp sa2ul sha512_generic snd_soc_ti_udma mcrc64 authenc snd_soc_ti_edma sha1_generic snd_soc_ti_sdma omap_mailbox omap_hwspinlock pwm_tiehrpwm bq25750_charger(O) snd_soc_tas5805m ina2xx tps65219_pwrbutton m_can_platform m_can ti_ads1015 industrialio_triggered_buffer kfifo_buf lm75 can_dev framebuffer(O) spi_omap2_mcspi wurth_sparkled(O) cfg80211 bluetooth ecdh_generic ecc rfkill libaes fuse ipv6 autofs4 
[50660.708400] CPU: 0 PID: 21493 Comm: 9 Tainted: G O 6.6.138-7.6.1-devel #1 
[50660.720481] Hardware name: Toradex Verdin AM62 WB on Verdin Development Board (DT) 
[50660.732129] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) 
[50660.741257] pc : rmqueue_bulk+0x384/0x610 
[50660.747432] lr : 0xfffffffe00600000 
[50660.753036] sp : ffffffc083fc39c0 
[50660.758417] x29: ffffffc083fc3a00 x28: 0000000000000010 x27: ffffffec53c83cc0 
[50660.767678] x26: 0000000000000000 x25: 0000000000000001 x24: 0000000000000017 
[50660.776899] x23: ffffff801caaa4a8 x22: dead000000000122 x21: dead000000000100 
[50660.786095] x20: 000000000000001f x19: ffffffec53c83d90 x18: ffffffec53c83dc0 
[50660.795268] x17: ffffffec53c83dc0 x16: fffffffe0061ffc8 x15: 0000000000000000 
[50660.804429] x14: 0000000000006003 x13: 0000000000000068 x12: 0000000000000000 
[50660.813547] x11: ffffff8001c93f00 x10: ffffff8001c93f08 x9 : ffffff8001c93f0c 
[50660.822639] x8 : ffffffc083fc3be0 x7 : 0000000000000200 x6 : fffffffe00600000 
[50660.831704] x5 : 0000000000000080 x4 : 0000000000000006 x3 : ffffffec53c841d0 
[50660.840701] x2 : fffffffe00600008 x1 : ffffffec53c840d0 x0 : 000000000000000a 
[50660.849670] Call trace: 
[50660.853875] rmqueue_bulk+0x384/0x610 
[50660.859286] get_page_from_freelist+0x4a8/0x1190 
[50660.865604] __alloc_pages+0x134/0xcf8 
[50660.871006] __folio_alloc+0x18/0x34 
[50660.876169] __handle_mm_fault+0x2d0/0xc60 
[50660.881813] handle_mm_fault+0x190/0x2b4 
[50660.887241] do_page_fault+0x204/0x4cc 
[50660.892465] do_translation_fault+0x88/0x9c 
[50660.898072] do_mem_abort+0x44/0x94 
[50660.902917] el0_da+0x30/0x88 
[50660.907179] el0t_64_sync_handler+0xb4/0x12c 
[50660.912726] el0t_64_sync+0x190/0x194 
[50660.917653] Code: d1002046 aa0603fe b40007a6 a9400c47 (f90004e3) 
[50660.925072] ---[ end trace 0000000000000000 ]--- 
[50660.930985] note: 9
[21493] exited with irqs disabled 
[50660.937552] note: 9
[21493] exited with preempt_count 3


[17143.036738] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000208 
[17143.045720] Mem abort info: 
[17143.048550] ESR = 0x0000000096000045 
[17143.052348] EC = 0x25: DABT (current EL), IL = 32 bits 
[17143.057729] SET = 0, FnV = 0 
[17143.060822] EA = 0, S1PTW = 0 
[17143.064004] FSC = 0x05: level 1 translation fault 
[17143.068942] Data abort info: 
[17143.071859] ISV = 0, ISS = 0x00000045, ISS2 = 0x00000000 
[17143.077414] CM = 0, WnR = 1, TnD = 0, TagAccess = 0 
[17143.082528] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 
[17143.087907] user pgtable: 4k pages, 39-bit VAs, pgdp=0000000086e17000 
[17143.094428] [0000000000000208] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000 
[17143.103251] Internal error: Oops: 0000000096000045 [#1] PREEMPT SMP 
[17143.109600] Modules linked in: nls_iso8859_1 nls_cp437 option usb_wwan qmi_wwan usbserial usbnet onboard_usb_hub optee_rng rng_core rpmsg_ctrl rpmsg_char dwc3 evdev aes_ce_cipher ghash_ce gf128mul sha2_ce sha256_arm64 snd_soc_simple_card sha1_ce snd_soc_simple_card_utils spi_cadence_quadspi leds_gpio gpio_keys led_class optee usb_conn_gpio roles tee dwc3_am62 ti_k3_r5_remoteproc virtio_rpmsg_bus rpmsg_ns rpmsg_core k3_j72xx_bandgap ti_k3_m4_remoteproc rtc_ti_k3 snd_soc_davinci_mcasp snd_soc_ti_udma snd_soc_ti_edma snd_soc_ti_sdma sa2ul sha512_generic authenc omap_hwspinlock sha1_generic mcrc64 omap_mailbox m_can_platform m_can pwm_tiehrpwm can_dev bq25750_charger(O) snd_soc_tas5805m ina2xx tps65219_pwrbutton ti_ads1015 lm75 industrialio_triggered_buffer kfifo_buf framebuffer(O) spi_omap2_mcspi wurth_sparkled(O) cfg80211 bluetooth ecdh_generic ecc rfkill libaes fuse ipv6 autofs4 
[17143.204206] CPU: 0 PID: 9104 Comm: 9 Tainted: G O 6.6.138-7.6.1-devel #1 
[17143.216208] Hardware name: Toradex Verdin AM62 WB on Verdin Development Board (DT) 
[17143.227855] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) 
[17143.236992] pc : rmqueue_bulk+0x384/0x610 
[17143.243171] lr : 0xfffffffe00600000 
[17143.248775] sp : ffffffc0836039c0 
[17143.254167] x29: ffffffc083603a00 x28: 0000000000000010 x27: ffffffd490e83cc0 
[17143.263431] x26: 0000000000000000 x25: 0000000000000001 x24: 000000000000000c 
[17143.272660] x23: ffffff801caaa4a8 x22: dead000000000122 x21: dead000000000100 
[17143.281851] x20: 000000000000001f x19: ffffffd490e83d90 x18: ffffffd490e83dc0 
[17143.291033] x17: ffffffd490e83dc0 x16: fffffffe0061ffc8 x15: 0000000000000000 
[17143.300197] x14: 0000000000005fe4 x13: 0000000000000068 x12: 0000000000000000 
[17143.309318] x11: ffffff8004b37400 x10: ffffff8004b37408 x9 : ffffff8004b3740c 
[17143.318422] x8 : ffffffc083603be0 x7 : 0000000000000200 x6 : fffffffe00600000 
[17143.327476] x5 : 0000000000000080 x4 : 0000000000000006 x3 : ffffffd490e841d0 
[17143.336487] x2 : fffffffe00600008 x1 : ffffffd490e840d0 x0 : 000000000000000a 
[17143.345471] Call trace: 
[17143.349676] rmqueue_bulk+0x384/0x610 
[17143.355092] get_page_from_freelist+0x4a8/0x1190 
[17143.361425] __alloc_pages+0x134/0xcf8 
[17143.366831] __folio_alloc+0x18/0x34 
[17143.371998] __handle_mm_fault+0x2d0/0xc60 
[17143.377637] handle_mm_fault+0x190/0x2b4 
[17143.383069] do_page_fault+0x204/0x4cc 
[17143.388292] do_translation_fault+0x88/0x9c 
[17143.393897] do_mem_abort+0x44/0x94 
[17143.398743] el0_da+0x30/0x88 
[17143.403012] el0t_64_sync_handler+0xb4/0x12c 
[17143.408559] el0t_64_sync+0x190/0x194 
[17143.413484] Code: d1002046 aa0603fe b40007a6 a9400c47 (f90004e3) 
[17143.420899] ---[ end trace 0000000000000000 ]--- 
[17143.426817] note: 9
[9104] exited with irqs disabled 
[17143.433301] note: 9
[9104] exited with preempt_count 3

latest kernel crash:


[84207.473610] Unable to handle kernel paging request at virtual address ffffff8006b0aae1 
[84207.482775] ------------[ cut here ]------------ 
[84207.487418] WARNING: CPU: 0 PID: 1 at /kernel/cgroup/pids.c:113 pids_cancel.constprop.0+0x44/0x4c 
[84207.496309] Modules linked in: cfg80211 nls_iso8859_1 nls_cp437 option qmi_wwan usb_wwan usbnet usbserial onboard_usb_hub optee_rng rng_core rpmsg_ctrl rpmsg_char dwc3 evdev aes_ce_cipher ghash_ce gf128mul sha2_ce sha256_arm64 sha1_ce snd_soc_simple_card snd_soc_simple_card_utils spi_cadence_quadspi optee ti_k3_r5_remoteproc gpio_keys usb_conn_gpio tee roles virtio_rpmsg_bus rpmsg_ns dwc3_am62 rpmsg_core k3_j72xx_bandgap ti_k3_m4_remoteproc rtc_ti_k3 sa2ul snd_soc_davinci_mcasp sha512_generic snd_soc_ti_udma authenc snd_soc_ti_edma sha1_generic mcrc64 snd_soc_ti_sdma omap_hwspinlock omap_mailbox snd_soc_tas5805m ina2xx tps65219_pwrbutton ti_ads1015 m_can_platform industrialio_triggered_buffer m_can lm75 kfifo_buf can_dev spi_omap2_mcspi pwm_tiehrpwm bluetooth ecdh_generic ecc rfkill libaes fuse ipv6 autofs4 [last unloaded: bq25750_charger(O)] 
[84207.571139] CPU: 0 PID: 1 Comm: systemd Tainted: G O 6.6.138-7.6.1-devel #1 
[84207.579488] Hardware name: Toradex Verdin AM62 WB on Verdin Development Board (DT) 
[84207.587053] pstate: 00000005 (nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) 
[84207.594012] pc : pids_cancel.constprop.0+0x44/0x4c 
[84207.598805] lr : pids_release+0x24/0x44 
[84207.602641] sp : ffffffc08000bae0 
[84207.605951] x29: ffffffc08000bae0 x28: ffffff8000478000 x27: ffffff800e47d800 
[84207.613091] x26: ffffff800066a3c0 x25: ffffffe624595080 x24: 0000000000000100 
[84207.620230] x23: ffffffe62461fee0 x22: 0000000000000001 x21: 0000000000000040 
[84207.627369] x20: ffffff800e47d400 x19: 0000000000000006 x18: 0000000000000000 
[84207.634508] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 
[84207.641646] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 
[84207.648784] x11: 0000000000000040 x10: ffffffe6245a7d30 x9 : 0000000000000000 
[84207.655923] x8 : 0000000000000000 x7 : ffffffffffffffff x6 : 0000000000000080 
[84207.663061] x5 : 0000000000000081 x4 : ffffff8006ab5600 x3 : 0000000000000000 
[84207.670200] x2 : ffffffffffffffff x1 : ffffffffffffffff x0 : ffffffffffffffff 
[84207.677340] Call trace: 
[84207.679785] pids_cancel.constprop.0+0x44/0x4c 
[84207.684231] cgroup_release+0x58/0x130 
[84207.687980] release_task+0x60/0x470 
[84207.691559] wait_consider_task+0x47c/0x9b4 
[84207.695741] do_wait+0x15c/0x278 
[84207.698968] kernel_waitid+0xf4/0x1bc 
[84207.702629] __do_sys_waitid+0x25c/0x304 
[84207.706552] __arm64_sys_waitid+0x28/0x34 
[84207.710562] invoke_syscall.constprop.0+0x50/0xec 
[84207.715271] do_el0_svc+0x40/0xc4 
[84207.718586] el0_svc+0x40/0x168 
[84207.721734] el0t_64_sync_handler+0x120/0x12c 
[84207.726093] el0t_64_sync+0x190/0x194 
[84207.729756] ---[ end trace 0000000000000000 ]--- 
[84207.741304] Mem abort info: 
[84207.748538] ESR = 0x0000000096000021 
[84207.752334] EC = 0x25: DABT (current EL), IL = 32 bits 
[84207.765391] SET = 0, FnV = 0 
[84207.771626] EA = 0, S1PTW = 0 
[84207.778479] FSC = 0x21: alignment fault 
[84207.786228] Data abort info: 
[84207.792493] ISV = 0, ISS = 0x00000021, ISS2 = 0x00000000 
[84207.804531] CM = 0, WnR = 0, TnD = 0, TagAccess = 0 
[84207.816467] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 
[84207.824545] swapper pgtable: 4k pages, 39-bit VAs, pgdp=00000000890f2000 
[84207.837980] [ffffff8006b0aae1] pgd=180000009e7f9003, p4d=180000009e7f9003, pud=180000009e7f9003, pmd=180000009e7c3003, pte=00e8000086b0af07 
[84207.863045] Internal error: Oops: 0000000096000021 [#1] PREEMPT SMP 
[84207.869367] Modules linked in: cfg80211 nls_iso8859_1 nls_cp437 option qmi_wwan usb_wwan usbnet usbserial onboard_usb_hub optee_rng rng_core rpmsg_ctrl rpmsg_char dwc3 evdev aes_ce_cipher ghash_ce gf128mul sha2_ce sha256_arm64 sha1_ce snd_soc_simple_card snd_soc_simple_card_utils spi_cadence_quadspi optee ti_k3_r5_remoteproc gpio_keys usb_conn_gpio tee roles virtio_rpmsg_bus rpmsg_ns dwc3_am62 rpmsg_core k3_j72xx_bandgap ti_k3_m4_remoteproc rtc_ti_k3 sa2ul snd_soc_davinci_mcasp sha512_generic snd_soc_ti_udma authenc snd_soc_ti_edma sha1_generic mcrc64 snd_soc_ti_sdma omap_hwspinlock omap_mailbox snd_soc_tas5805m ina2xx tps65219_pwrbutton ti_ads1015 m_can_platform industrialio_triggered_buffer m_can lm75 kfifo_buf can_dev spi_omap2_mcspi pwm_tiehrpwm bluetooth ecdh_generic ecc rfkill libaes fuse ipv6 autofs4 [last unloaded: bq25750_charger(O)] 
[84207.944200] CPU: 0 PID: 1023858 Comm: kworker/0:2 Tainted: G W O 6.6.138-7.6.1-devel #1 
[84207.953418] Hardware name: Toradex Verdin AM62 WB on Verdin Development Board (DT) 
[84207.960986] Workqueue: cgroup_offline css_killed_work_fn 
[84207.966322] pstate: 00000005 (nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) 
[84207.973282] pc : css_killed_work_fn+0xe4/0x1ac 
[84207.977728] lr : css_killed_work_fn+0x154/0x1ac 
[84207.982259] sp : ffffffc080eb3d60 
[84207.985569] x29: ffffffc080eb3d60 x28: 0000000000000000 x27: 0000000000000000 
[84207.992709] x26: 0000000000000000 x25: ffffffe624622c60 x24: ffffffe62461fee0 
[84207.999848] x23: 00000000ffffffff x22: ffffffffffffffff x21: ffffff800e47c600 
[84208.006988] x20: 0000000000000001 x19: ffffff8006b0aa81 x18: 0000000000000000 
[84208.014126] x17: ffffff99f86f1000 x16: ffffffc080000000 x15: 031c211d54c03dc8 
[84208.021265] x14: 0000000000000004 x13: ffffffe624629188 x12: 0000000000000000 
[84208.028405] x11: ffffff8000803410 x10: ffffff8000803240 x9 : 0000000000000000 
[84208.035544] x8 : ffffff800059a200 x7 : fefefefefefefeff x6 : ffffff8019a49ae0 
[84208.042682] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000 
[84208.049821] x2 : ffffff8006b0aae1 x1 : 0000000000000000 x0 : ffffff800e47c600 
[84208.056960] Call trace: 
[84208.059405] css_killed_work_fn+0xe4/0x1ac 
[84208.063505] process_one_work+0x144/0x29c 
[84208.067522] worker_thread+0x1ac/0x364 
[84208.071273] kthread+0x110/0x114 
[84208.074505] ret_from_fork+0x10/0x20 
[84208.078088] Code: d50323bf d65f03c0 91018262 f9800051 (885f7c40) 
[84208.084179] ---[ end trace 0000000000000000 ]--- Jun 05 14:30:53 verdin-am62-15599560 systemd

I currently do not have time to build a really clean minimal toradex image without all my layers in it because I am under a deadline, but this feels like a very important issue!

2

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.

Thank you for the detailed report. This is indeed a serious kernel stability issue affecting your Verdin AM62 system running BSP 7.6.1-devel. Based on your stack traces and the reproducible nature tied to systemd service restarts, this appears to be a memory management issue in the kernel’s page allocation subsystem.

Analysis of the Issue

The kernel panics you’re experiencing show consistent patterns:

  1. Memory allocation failures in rmqueue_bulk+0x384/0x610 and related memory management functions
  2. NULL pointer dereferences at virtual address 0000000000000208
  3. Alignment faults in cgroup management code (css_killed_work_fn)
  4. PID 1 (systemd) involvement in the latest crash, confirming the systemd correlation

The fact that running the same failing script in a shell loop doesn’t reproduce the issue strongly suggests this is related to systemd’s process management, cgroup handling, or service lifecycle management rather than the script itself.

Immediate Recommendations

1. Verify BSP Version and Consider Downgrade

Your system is running BSP 7.6.1-devel, which according to the Embedded Linux Release Matrix(1), is a development version. For production stability, consider using a stable release like BSP 6.8.0(1).

2. Enable Enhanced Logging

To gather more diagnostic information, you can enable kernel debugging as described in the community thread(2):

echo "module kernel +p" > /sys/kernel/debug/dynamic_debug/control

3. Memory Pressure Testing

Since the crashes involve memory allocation failures, test your system under different memory conditions:

  • Monitor memory usage during the service restart cycles
  • Check for memory leaks in systemd or related processes

4. Kernel Configuration Review

If you’re building a custom kernel(3), ensure your kernel configuration includes proper debugging options:

  • CONFIG_DEBUG_KERNEL=y
  • CONFIG_PRINTK=y
  • CONFIG_DYNAMIC_DEBUG=y

Next Steps

  1. Document the exact reproduction steps including timing and system load conditions
  2. Collect more detailed logs with the enhanced debugging enabled
  3. Test with a minimal Toradex reference image when possible to isolate custom layer contributions
  4. Monitor systemd journal for any service management anomalies during the restart cycles

This issue appears to be a kernel-level bug that may require upstream attention. The consistent memory allocation failures combined with systemd involvement suggest a race condition or resource leak in the process lifecycle management.

Would you be able to test this with a clean Toradex reference image to help isolate whether this is related to your custom layers or a core BSP issue?

Best regards, Toradex AI Assistant.

Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

3

I hope to hear from an actual Human

4

Based on the severity of the kernel crashes you’re experiencing with your Verdin AM62 V1.2A running BSP 7.6.1-devel, this issue requires attention from Toradex’s human support team. The consistent kernel panics in memory management functions triggered by systemd service restarts represent a critical stability issue.

For immediate escalation to human support, please visit the Toradex Community and create a new support topic with your detailed crash logs and reproduction steps. You can also visit GitHub for technical discussions.

Given that you’re running a development BSP version (7.6.1-devel) and experiencing kernel-level crashes that appear to be related to memory allocation failures in rmqueue_bulk and cgroup management, this warrants direct engineering attention.

The stack traces you’ve provided show memory management corruption that could indicate:

  • A race condition in the kernel’s page allocator
  • Memory corruption in systemd’s process lifecycle management
  • Issues with cgroup resource tracking during rapid service restarts

Since you mentioned being under a deadline, I’d recommend also posting this issue directly in the community forum with the “Technical Support” category to ensure it gets proper visibility from Toradex engineers.

Best regards, Toradex AI Assistant.

Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

5

Thank you for the detailed information and stack traces.

From the logs, this does not look like a typical userspace/systemd issue. Although the crash can be triggered by a systemd service restart loop, the actual failure happens inside the kernel memory-management path, for example in rmqueue_bulk() / __alloc_pages(), and later in the cgroup PID accounting path. This suggests possible kernel memory corruption or a kernel-side regression rather than systemd itself being the direct root cause.

I also noticed that the kernel is tainted and that several out-of-tree modules are loaded in the failing cases. Even though you removed some custom modules for the latest test, the kernel still reports a tainted state and previously loaded out-of-tree modules. To narrow this down, could you please try to reproduce the issue on a clean Toradex reference image without any additional out-of-tree modules or custom drivers loaded?

Recommended next steps:

  1. Boot a clean Toradex reference 7.6.1 image on the Verdin Development Board.
  2. Do not load any custom/out-of-tree kernel modules.
  3. Reproduce the same systemd restart-loop test.
  4. Collect the full serial console log from boot until the crash.
  5. Please also share the output of:
uname -a
cat /proc/cmdline
cat /proc/modules
cat /proc/sys/kernel/tainted
systemctl status crashing.service
journalctl -u crashing.service -b
dmesg

If the crash is reproducible on a clean Toradex reference image, we can escalate it as a potential kernel/BSP issue. If it only happens with the custom modules loaded, the next step would be to isolate which module or driver is causing the memory corruption.

As an additional test, please try the same reproduction on the previous 7.2.0 image using the same hardware and the same systemd service. That comparison would help confirm whether this is a regression between 7.2.0 and 7.6.1.

6

it says it is a development version (7.6.1-dev) or did i get confused and should have used 7.6.0 ?

I will do your suggestions but since Im at a deadline, I can’t fully work on this.
I’ll make the program in question stop crashing and then the system is stable enough for me to continue.

I will come back to this as it is important!

7

It looks like you are using a development image. The latest quarterly release is now 7.6.1, so please try reproducing the issue with the 7.6.1 release image.