Here in Mectronic we are working on a complete Yocto system and we want to go deeper into a problem we encountered in the past.
On our old architecture, based on iMX6DL, some times it happened a kernel panic because the FS was corrupted.
This happened since the machine was turned on/off by a physical switch and it doesn’t perform a controlled shutdow process.
The fast solution we found, since the FS was very small, was to detect the Kernel panic and re-write the FS from the u-boot.
All sensible data were stored in an external SD card, so re write the entire FS was not a problem.
We would ask if does exist a better solution to implement this “recovery option” also on your system.
It should be better to find a more elegant solution since the FS we are developing is bigger than the old one.
Do you maybe have more information regarding this crashes? Which BSP version did you use on these modules? It shouldn’t happen that a power-off destroys the kernel. Maybe this was a bug in an older BSP version?
What you can do to reduce the risks is:
Mount the root partition read only
Use an initramfs as root filesystem
Try to write as little as possible to flash
Especially logging only as little as possible helps a lot. You can e.g. log to a ramdisk instead of eMMC and then do a backup once an hour. Of course this highly depends on your use case.
Also try to find the root cause if you see strange behaviour like the one described by you. Maybe it hides a serious problem.
Our old architecture was not a Toradex, we are testing Apalis iMX8QM 4GB V1.0B since we need high performance on our top class machine and our old system isn’t enough.
By the way it was an iMX6DL performing Yocto 1.7.3 and 3.10 Linux Kernel. Quite old :D.
I’m not very expirienced in this field so forgive me if now I say something wrong. I just try to describe what we noticed and what we tried to do.
The problem was not in the Kernel i think, simple it went to a kernel panic because a part of the file system wa corrupted and so the start up procedure failed.
Every times it happened I always check which “block” was corrupted, it was always a File system part, the Kernel was ok.
So I implemented a procedure in the kernel which detected this kind of kernel panic and reboot the machine. At this time u-boot started a procedure that had to re write the file system.
Obviously I’m not looking here for a solution regarding another product problem, I’m just investigating if this problem could be present also in our new system (Toradex iMX8qm based) and how to avoid it.
We writed just a little on the flash, something like a configuration file (less then 1KB) updated once a minute.
I imagined that maybe someone worked on a recovery solution better than mine (since my solution is not properly a recovery option), but maybe, as you suggested, initramfs should be the better solution.
I will try to go deeper inside, maybe does exist a guide in Toradex documentation? :D.
I’m a collegue of Merlin and we had a problem with initramfs. Our image is based on your console-tdx-image to which we have added recipes for cranksoftware and gstreamer. The result is a FS of about 200 MB and so performances at startup are extremely poor. We lose about 2/3 seconds on u-boot and another 5/6 second on kernel and in this way the boot process is extremely slow.
However, the time in U-Boot won’t change because it’s the loading time of the ramdisk… Alternatively you could just use a normal rootfs instead of the initramfs and make that read only… Then you don’t lose time for loading and unzipping. So the boot up should be faster.
Using a read only file system is the simpliest solution, but are you sure that we will not have any kernel panic due to FS failure in normal situations? I’m asking because in the past we had to bring some machines from abroad, with costs for us and inconveniences for customers, when they were used normally (switch on/off).
If you use eMMC in read only mode you should be on the safe side. With RAW NAND there are sometimes issues when not properly implemented. However, whether using an initramfs or using a read only partition should be exactly the same… If you had corruption in the past on a read only filesystem you would may also have corruption when loading the initramfs.
So switching from an initramfs to a read only filesystem is safe.
since it’s the first time i create a read-only rootfs i would like to ask you if everything i’ve done is correct:
I’ve added EXTRA_IMAGE_FEATURES += “read-only-rootfs” in apalis-imx8.conf file
I’ve edited fstab and changed mount options of /dev/root adding ro
It seems that everything is correct since if i try to touch test or mkdir test in the root directory i get an error that said that is a Read-only file system. Is there something else i have to do?
