Journald missing kernel messages

Hi,

On my custom build linux image there are no kernel moessages in journald. What may be the cause ?

This is the ouput of journalctl -k

-- Logs begin at Wed 2020-05-27 20:20:15 UTC, end at Wed 2020-05-27 20:35:05 UTC. --
-- No entries --

journald.conf:

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# See journald.conf(5) for details

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=login
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
SystemMaxUse=16M
#SystemKeepFree=
#SystemMaxFileSize=
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#TTYPath=/dev/console
MaxLevelStore=info
MaxLevelSyslog=info
#MaxLevelKMsg=notice
#MaxLevelConsole=info

kernel cmdline:

core_edp_mv=1300 usb_high_speed=1 user_debug=30 ip=off root=/dev/mmcblk0p2 ro rootfstype=ext3 rootwait asix_mac=00:14:2d:4c:cb:aa consoleblank=0 no_console_suspend=1 console=tty1 debug_uartport=hsport,0 vmalloc=128M mem=1012M@2048M fbmem=12M@3060M video=tegrafb0:640x480-16@60

Dear @mczerski,

Greetings and Thank you for writing to our community!

I have a couple of questions to ask you. Could you kindly answer them for me?

  • Have you already checked if the journalctl service is active? You could do that by checking systemd-journald.service is set to active?
  • Does just running journalctl show you any output?
  • Could you share the output of dmesg.txt and also verify if the /var/log is mounted as re, instead of just read-only (you can use mount command to check that)

Furthermore, here is linux - Why does journalctl say "-- no entries --"? - Unix & Linux Stack Exchange, which might be helpful to you.

Best Regards,

Janani

dmesg.txt is in attachments

systemctl status systemd-journald.service

● systemd-journald.service - Journal Service
   Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; vendor preset: enabled)
   Active: active (running) since Fri 2020-05-29 08:51:29 UTC; 4min 55s ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 119 (systemd-journal)
   Status: "Processing requests..."
   CGroup: /system.slice/systemd-journald.service
           └─119 /lib/systemd/systemd-journald

May 29 08:51:29 colibri-t30 systemd-journald[119]: Journal started
May 29 08:51:29 colibri-t30 systemd-journald[119]: Runtime journal (/run/log/journal/c3426ad563c94ae4b8699b4c476c7a82) is 6.1M, max 48.9M, 42.7M free.
May 29 08:51:29 colibri-t30 systemd-journald[119]: Runtime journal (/run/log/journal/c3426ad563c94ae4b8699b4c476c7a82) is 6.1M, max 48.9M, 42.7M free.
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

journalctl | head

-- Logs begin at Fri 2020-05-29 08:51:29 UTC, end at Fri 2020-05-29 08:53:15 UTC. --
May 29 08:51:29 colibri-t30 systemd-journald[119]: Journal started
May 29 08:51:29 colibri-t30 systemd-journald[119]: Runtime journal (/run/log/journal/c3426ad563c94ae4b8699b4c476c7a82) is 6.1M, max 48.9M, 42.7M free.
May 29 08:51:29 colibri-t30 systemd-sysctl[109]: Couldn't write '16' to 'kernel/sysrq', ignoring: No such file or directory
May 29 08:51:29 colibri-t30 systemd-sysctl[109]: Couldn't write 'fq_codel' to 'net/core/default_qdisc', ignoring: No such file or directory
May 29 08:51:29 colibri-t30 systemd-sysctl[109]: Couldn't write '1' to 'fs/protected_hardlinks', ignoring: No such file or directory
May 29 08:51:29 colibri-t30 systemd-fsck[110]: rootfs: clean, 31957/234784 files, 174335/921600 blocks
May 29 08:51:29 colibri-t30 systemd-sysctl[109]: Couldn't write '1' to 'fs/protected_symlinks', ignoring: No such file or directory
May 29 08:51:29 colibri-t30 systemd[1]: systemd-remount-fs.service: Failed to set invocation ID on control group /system.slice/systemd-remount-fs.service, ignoring: Operation not supported
May 29 08:51:29 colibri-t30 systemd[1]: Starting Remount Root and Kernel File Systems...[link text][1]

mount

/dev/mmcblk0p2 on / type ext3 (rw,noatime,errors=continue,user_xattr,acl,barrier=1,data=writeback)
devtmpfs on /dev type devtmpfs (rw,relatime,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
tmpfs on /etc/machine-id type tmpfs (ro,relatime,mode=755)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
tmpfs on /var/tmp type tmpfs (rw,relatime)
tmpfs on /var/volatile type tmpfs (rw,relatime)
tmpfs on /var/log type tmpfs (rw,relatime)
/dev/sda1 on /media/sda1 type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=writeback)
/dev/mmcblk1p1 on /media/mmcblk1p1 type vfat (rw,relatime,gid=6,fmask=0007,dmask=0007,allow_utime=0020,codepage=cp437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=100160k,mode=700)

the systemd journald documentation states that kernel messages are captured with kmsg, so I presume that they are read from /dev/kmsg.

root@colibri-t30:~# ll /dev/kmsg 
crw-------    1 root     root        1,  11 Jun  1 06:29 /dev/kmsg

If I try to cat this file I get:

root@colibri-t30:~# cat /dev/kmsg
cat: read error: Invalid argument

maybe this is a problem ? why I get this error ?
I can read /proc/kmsg though and it contains kernel logs.

So, journald in version 234 uses /dev/kmsg to get kernel logs into its journal files. And /dev/kmsg can be accessed from userspace in kernel 3.5+ (linux/dev-kmsg at master · torvalds/linux · GitHub). But angstrom linux for colibri-t30 uses kernel 3.1. So can anyone confirm that this just does not work in current version of angstrom linux for colibri-t30 ?

well, I think that I have my answer. In the NEWS file in systemd for version 189 there is:

CHANGES WITH 189:

    * Support for reading structured kernel messages from
      /dev/kmsg has now been added and is enabled by default.

    * Support for reading kernel messages from /proc/kmsg has now
      been removed. If you want kernel messages in the journal
      make sure to run a recent kernel (>= 3.5) that supports
      reading structured messages from /dev/kmsg (see
      above). /proc/kmsg is now exclusive property of classic
      syslog daemons again.

Why this is not documented somehwere on toradex website ?

Dear @mczerski ,

We are so glad that you arrived at your solution and thank you for updating us on that.
We will look into the documentation.

Best Regards,
Janani