Persistently store kernel log output

Technical Support
, ,
1

Hello …

What is the recommended method in Torizon OS to persistently store kernel log output?

Keywords: OOPS, pstore, kernelpanic logs

Best
Gerald

2

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.

I’d be glad to help! Could you let me know which OS and version you are using?

To provide the most accurate guidance for persistently storing kernel log output including OOPS and kernel panic logs, I need to understand your specific platform setup. You can get this information by running the tdx-info command on your device - you can find more details about this command at this link.

This information will help me provide the most appropriate solution for your specific hardware and OS version combination.

Best regards,
Toradex AI Assistant.

Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

3

here it is:

Software summary

## Bootloader:               U-Boot
Kernel version:           5.15.183-6.8.3+git.defb81960f4f #1-TorizonCore SMP PREEMPT Tue Jun 3 13:26:20 UTC 2025
Kernel command line:      root=LABEL=otaroot rootfstype=ext4 quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3 ostree=/ostree/boot.1/torizon/86ce149e0226c872171c6a8c49bdc1fd283ac658c89ab10bd1d2c499ce4fb37c/0 board=bsi tcb=2.7 m4=1.2 ramoops.mem_address=0xa0000000 ramoops.mem_size=0x200000
Distro name:              NAME=“TorizonCore”
Distro version:           VERSION_ID=6.8.3-build.37
Distro variant:           VARIANT=“Docker”
Hostname:                 apalis-imx8-07202014

## Hardware info

## HW model:                 Toradex Apalis iMX8QM V1.1 on Apalis Ixora V1.2 Carrier Board
Toradex version:          0037 V1.1E
Serial number:            07202014
Processor arch:           aarch64
4

Persistent Kernel Log Storage in Torizon OS

Based on your Torizon OS 6.8.3 setup on Apalis iMX8QM, I can see that ramoops is already enabled in your kernel command line with the parameters ramoops.mem_address=0xa0000000 ramoops.mem_size=0x200000(1).

Current ramoops Configuration

Your system is configured with:

  • Memory address: 0xa0000000
  • Memory size: 0x200000 (2MB)

This means ramoops is already functional for capturing kernel panic and OOPS logs(1).

Accessing Persistent Kernel Logs

To access the stored kernel logs after a reboot following a kernel panic or OOPS:

  1. Mount the pstore filesystem:
mkdir -p /mnt/pstore
mount -t pstore pstore /mnt/pstore
  1. View the captured logs:
ls /mnt/pstore/

You should see files like:

  • console-ramoops-0 - Console output
  • dmesg-ramoops-0 - Kernel messages
  • dmesg-ramoops-1 - Additional kernel messages
  1. Read the panic logs:
cat /mnt/pstore/dmesg-ramoops-0
tail -n 10 /mnt/pstore/console-ramoops-0

Device Monitoring Integration

For comprehensive logging and monitoring, Torizon OS includes Fluent Bit for device monitoring(1). Fluent Bit can collect system metrics, logs, and send them to Torizon Cloud(1). This provides additional persistent logging capabilities beyond ramoops for regular system monitoring.

Important Notes

  • ramoops data persists only through soft reboots - if you have a complete power cycle (cold boot), the RAM contents are lost
  • The logs are stored in the reserved RAM area and will be available after kernel panics or unexpected reboots
  • Your current 2MB allocation should be sufficient for most kernel panic scenarios

Your system is already properly configured for persistent kernel log storage using ramoops!

Best regards, Toradex AI Assistant.

Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

5

If I trigger a kernel panic over:
`echo c > /proc/sysrq-trigger`

the module reboots

then I do:

mkdir -p /mnt/pstore

mount -t pstore pstore /mnt/pstore

ls /mnt/pstore/

and …
I see nothing.

What can I do?

6

Greetings @gerko,

For persistent logging do you just want something like this: Persistent Journald Logging | Toradex Developer Center

Or, are you looking for something else? If so what?

Best Regards,
Jeremias

7

Thank You @jeremias.tx ,

this gives me everything I need.

At the same time, I want to keep the constant write load on the SSD low and only write the log to the SSD on reboot.

So: I want to keep logging to RAM.

This should also work in the event of a kernel panic, where a reboot is announced in 5 seconds.

Are there any built-in solutions for this in journalctrl? Or do I need to use a systemctrl service?

8

Are there any built-in solutions for this in journalctrl? Or do I need to use a systemctrl service?

Hmm I don’t believe so. This is the documentation for journald: journald.conf

I don’t see a configuration or option for the conditional writing of logs to disk as you want. You’ll probably need some hook or something similar that triggers on reboots and kernel panics. Then the logs can be flushed to your SSD based on that hook trigger. Though I’m not sure at the moment how one would correctly create such a trigger, this would probably require some research.

Of course there may be some crashes or failures where the logs may still be difficult or hard to capture. Especially if the crash caused a very abrupt reboot, the journald system may not have had time to properly act. Which would cause the logs to be either missing or possibly corrupt based on the nature of the issue.

Best Regards,
Jeremias

9

Okay. It’s impossible to cover every eventuality.

For recurring problems, we can log using journalctrl or dmesg. We can then process the output on the SSH console even after the problem occured.

However, such a feature would be desirable for devices in the field at customer sites, especially during the initial setup phase.

Thank you.
Gerald

10

However, such a feature would be desirable for devices in the field at customer sites, especially during the initial setup phase.

Just to clarify, what specific feature are you requesting here?

The conditional persistent logging of kernel logs to disk. Is that the specific feature you’re asking, or are you asking something different? If so, then there may exist some other solutions or methods in the greater Linux community. I say this since this specifically isn’t anything unique to Torizon or Toradex as a whole.

Though the details of how the logging should behave could be specific to how an individual wants their product to behave. So maybe a “generic” solution wouldn’t quite work as well.

In any case, do you have any further questions here, or do you more or less have what you need now?

Best Regards,
Jeremias

11

Thank you @jeremias.tx ,

I think I have all the information I need.

Short: My goal was not to reinvent the wheel.

Normally, data is logged to RAM. Persistent logging isn’t good for the hardware.

We encountered a kernel panic on a kernel module under development under certain conditions. We discovered this bug during the development phase and were able to reproduce it.

If something like this happens at a customer’s site, it’s unfortunate, and it would be very useful to be able to view the kernel logs.

I thought that during the 5-second countdown in a kernel panic, the current log file could also be saved. That’s why I had the idea that something like this could be enabled… somehow.

Best regards
Gerald