Incorrect /etc/tmpfiles.d/00-create-volatile.conf breaks logfile storage

Technical Support
, , ,
1

In most scarthgap-7 builds, we’re just now noticing that we don’t have a valid /var/volatile/log directory (thus no usable /var/log symlink), and it turns out that something’s installing an incomplete copy of 00-create-volatile.conf in /etc/tmpfiles.d/. For most apps that just use syslog(), that doesn’t break the app–it just means the app’s log messages don’t get stored. It does, however, prevent mysqld from starting.

So now we need to track down what recipe is including the broken 00-create-volatile.conf? If we simply remove that, the fully-fleshed-out version in /usr/lib/tmpfiles.d appears to be sufficient.

2

Hi, I’m the Toradex AI Assistant.

I might make mistakes. Our support team will step in if needed. If I answered your question, please tick the “Solution” checkbox. For follow-up questions to me, please use @ToradexAI in your post.

Did you like my answer? :+1: :-1:

Hello! Thank you for reaching out with your question.

You’ve observed that in some scarthgap-7 builds, /var/volatile/log (and thus /var/log) isn’t properly set up, which can affect services like mysqld. This typically relates to how the 00-create-volatile.conf tmpfiles configuration is handled.

Key Points and Guidance:

  • Default Behavior: By default, Toradex distributions set up /var/log as a symlink to /var/volatile/log, and this volatile directory is managed in RAM for flash wear reduction. This setup is controlled via tmpfiles configuration, specifically by files like /etc/tmpfiles.d/00-create-volatile.conf or /usr/lib/tmpfiles.d/00-create-volatile.conf.
    Persistent journald logging documentation

  • Issue Description: If an incomplete 00-create-volatile.conf lands in /etc/tmpfiles.d/, it overrides the system-wide configuration in /usr/lib/tmpfiles.d/, possibly resulting in missing or misconfigured directories needed for proper logging operation.

  • Investigating the Cause:

    1. Recipe Search: Within your Yocto or OpenEmbedded build environment, you can search for which recipe is installing or modifying 00-create-volatile.conf using the following command in your build directory:

      grep -r "00-create-volatile.conf" meta-*/recipes*/

      Check for references in both meta-toradex-* layers and any custom layers you might be using.

    2. Removal as Workaround: You are correct—if you remove or rename the incomplete /etc/tmpfiles.d/00-create-volatile.conf, the system will use the (likely correct) configuration in /usr/lib/tmpfiles.d/. This usually restores expected behavior.

    3. Permanent Fix: The best long-term solution is to identify and correct the recipe responsible for installing the malformed config file. Adjust or patch the recipe so that either:

      • It installs the complete, correct 00-create-volatile.conf, or
      • Does not install an unnecessary or incomplete version at all.

  • Manual Hotfix:
    You can also create a custom bbappend for the offending recipe that either deletes or replaces the faulty file during the install step.

Summary

  1. Search your layer sources for the origin of the incomplete 00-create-volatile.conf.
  2. Either remove the incorrect file or ensure it is replaced with a correct version.
  3. Review layer modifications if you are using custom layers, as this is often where overrides occur.

For more details on the logging system’s default setup, see:
Persistent journald logging

If you need further assistance on how to patch a recipe or investigate with more advanced Yocto/OpenEmbedded tools, please let me know!

Best regards,
Toradex AI Assistant.

3

Yup, it’s apparently meta-toradex-torizon/recipes-core/systemd/systemd_%.bbappend creating that file:

do_install:append() {
    if [ ${@ oe.types.boolean('${VOLATILE_LOG_DIR}') } = True ]; then
        sed -i '/^d \/var\/log /d' ${D}${nonarch_libdir}/tmpfiles.d/var.conf
        echo 'L+ /var/log - - - - /var/volatile/log' >> ${D}${sysconfdir}/tmpfiles.d/00-create-volatile.conf
    else
        # Make sure /var/log is not a link to volatile (e.g. after system updates)
        sed -i '/\[Service\]/aExecStartPre=-/bin/rm -f /var/log' ${D}${systemd_system_unitdir}/systemd-journal-flush.service
    fi
4

Greetings @jarndraugr,

Thank you for bringing this to our attention. Something is definitely off here. On 7.2.0 I can confirm the following.

There is a symlink from /var/log to /var/volatile/log:

$ ls -l /var/log
lrwxrwxrwx 1 root root 17 May 10 00:11 /var/log -> /var/volatile/log

But, /var/volatile/log doesn’t actually exist in any form:

$ ls -l /var/volatile/
total 0

Furthermore, checking the logs for systemd-tmpfiles-setup.service I see the following:

Jan 08 18:22:59 verdin-imx8mp-06849059 systemd-tmpfiles[451]: Failed to open path '/var/log': No such file or directory
Jan 08 18:22:59 verdin-imx8mp-06849059 systemd-tmpfiles[451]: Failed to open path '/var/log': No such file or directory
Jan 08 18:22:59 verdin-imx8mp-06849059 systemd-tmpfiles[451]: Failed to open path '/var/log': No such file or directory
Jan 08 18:22:59 verdin-imx8mp-06849059 systemd-tmpfiles[451]: Failed to open path '/var/log': No such file or directory

Comparing with our version 6.8.2. OS based on Kirkstone I see the following. I see the following:

The symlink start and endpoints actually exist. There’s no errors in the service and the file has it’s usual contents plus the line we append in our bbappend:

cat /etc/tmpfiles.d/00-create-volatile.conf
#This goes hand-in-hand with the base-files of OE-Core. The file must
# be sorted before 'systemd.conf' becuase this attempts to create a file
# inside /var/log.


d               /run/lock               1777    -       -       -
d               /var/volatile/log               -       -       -       -
d               /var/volatile/tmp               1777    -       -
L+ /var/log - - - - /var/volatile/log

This file gets installed by the base systemd recipe in openembedded-core, and we just add that last line to it in our bbappend. Therefore something must have changed in the base systemd recipe in Scarthgap. Looking at the recipes between the two versions I see the issue now.

In Kirkstone the recipe had this line to install the file:

install -m 0644 ${WORKDIR}/00-create-volatile.conf ${D}${sysconfdir}/tmpfiles.d/

This led to it being installed in /etc/tmpfiles.d which we append to.

Now in Scarthgap it was changed to:

install -m 0644 ${WORKDIR}/00-create-volatile.conf ${D}${nonarch_libdir}/tmpfiles.d/

Which leads to it being installed in /usr/lib/tmpfiles.d instead. Which is okay, but we need to update our bbappend to append the additional line to the new install location. Right now it’s appending to the old location which creates a new file instead since nothing is there anymore by default.

Well that was an interesting issue to investigate, I’ll create a ticket for our team to review and update our bbappend. Thank you again for bringing this to our attention.

Best Regards,
Jeremias

6

Hi @jarndraugr,

Just to inform you, this issue should now be resolved with the following commit: systemd: Fix volatile logs · torizon/meta-toradex-torizon@51e8f15 · GitHub

Thank you again for bringing this to our attention.

Best Regards,
Jeremias