Incomplete /etc/fstab mounting prevents system from booting - the root account is locked

I was working on mounting the filesystem for a SSD attached to our carrier board when the system rebooted leaving /etc/fstab in an incomplete state.

Now when the system reboots, I get a message (as expected) on the serial port that it isn’t able to mount the SSD but then it doesn’t give me an opportunity to fix anything. What options do I have to get past this? It doesn’t bring up the portainer interface and doesn’t connect to the OTA system nor can I SSH into it.

U-Boot 2020.04-5.5.0+git.81bc8894031d (Jan 01 1970 - 00:00:00 +0000)

CPU:   NXP i.MX8QM RevB A53 at 1200 MHz

DRAM:  4 GiB
MMC:   FSL_SDHC: 0, FSL_SDHC: 1, FSL_SDHC: 2
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
Model: Toradex Apalis iMX8 QuadMax 4GB IT V1.1C, Serial# 06945459

 BuildInfo:
  - SCFW 778670e2, SECO-FW d63fdb21, IMX-MKIMAGE 8947fea3, ATF 835a8f6
  - U-Boot 2020.04-5.5.0+git.81bc8894031d

switch to partitions #0, OK
mmc0(part 0) is current device
flash target is MMC:0
Net:   eth0: ethernet@5b040000
Fastboot: Normal
Normal Boot
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
1182 bytes read in 13 ms (87.9 KiB/s)
## Executing script at 83100000
5967 bytes read in 28 ms (208 KiB/s)
166807 bytes read in 34 ms (4.7 MiB/s)
43 bytes read in 26 ms (1000 Bytes/s)
Applying Overlay: apalis-imx8_hdmi_overlay.dtbo
2177 bytes read in 35 ms (60.5 KiB/s)
11612693 bytes read in 279 ms (39.7 MiB/s)
Uncompressed size: 28715520 = 0x1B62A00
9176228 bytes read in 228 ms (38.4 MiB/s)
## Flattened Device Tree blob at 83000000
   Booting using the fdt blob at 0x83000000
   Loading Ramdisk to fcd8d000, end fd64d4a4 ... OK
   Loading Device Tree to 00000000fcd41000, end 00000000fcd8cfff ... OK

Starting kernel ...

[    0.161466] 001: No BMan portals available!
[    0.162729] 001: No QMan portals available!
[    1.918594] 001: imx-audmix imx-audmix.0: failed to find SAI platform device
[    2.047617] 004: imx6q-pcie 5f000000.pcie: pcie_ext clock source missing or invalid
[    2.048909] 003: imx6q-pcie 5f010000.pcie: pcie_ext clock source missing or invalid
[    2.728310] 002: debugfs: Directory '59050000.sai' with parent 'apalis-imx8qm-sgtl5000' already present!
[    3.893617] 001: imx6q-pcie 5f010000.pcie: failed to initialize host
[    3.893626] 001: imx6q-pcie 5f010000.pcie: unable to add pcie port.
[    4.013334] 005: imx6q-pcie 5f000000.pcie: failed to initialize host
[    4.013341] 005: imx6q-pcie 5f000000.pcie: unable to add pcie port.
Starting version 244.5+
[    6.087952] 001: systemd[1]: Failed to mount /var/rootdirs/mnt/gojimedia.
[   10.617511] 000: debugfs: Directory '59090000.sai' with parent 'imx-audio-hdmi-tx' already present!
[   10.625340] 001: debugfs: File 'Capture' in directory 'dapm' already present!
You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl
Cannot open access to console, the root account is locked.
See sulogin(8) man page for more details.

Press Enter to continue.

Reloading system manager configuration
Starting default target

Ok I figured out how to get out of this situation and figured I would share what I did in case it is useful to someone else.

First you have to abort the boot process and drop into U-Boot by hitting a key on the serial port before it starts the boot process.

U-Boot 2020.04-5.5.0+git.81bc8894031d (Jan 01 1970 - 00:00:00 +0000)

CPU:   NXP i.MX8QM RevB A53 at 1200 MHz

DRAM:  4 GiB
MMC:   FSL_SDHC: 0, FSL_SDHC: 1, FSL_SDHC: 2
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
Model: Toradex Apalis iMX8 QuadMax 4GB IT V1.1C, Serial# 06945459

 BuildInfo:
  - SCFW 778670e2, SECO-FW d63fdb21, IMX-MKIMAGE 8947fea3, ATF 835a8f6
  - U-Boot 2020.04-5.5.0+git.81bc8894031d

switch to partitions #0, OK
mmc0(part 0) is current device
flash target is MMC:0
Net:   eth0: ethernet@5b040000
Fastboot: Normal
Normal Boot
Hit any key to stop autoboot:  0
Apalis iMX8 #

From there, you need to figure out where the corrupted file is located. Since I knew it was in /etc/fstab, it was a matter of figuring out where on the file system it actually exists.

Starting with the mmc command you can find all the file systems:

Apalis iMX8 # mmc list
FSL_SDHC: 0 (eMMC)
FSL_SDHC: 1
FSL_SDHC: 2

From there, you can then use the ext4ls command to traverse the directory.

Apalis iMX8 # ext4ls mmc 0:1
<DIR>       4096 .
<DIR>       4096 ..
<DIR>      16384 lost+found
            1182 boot.scr
<DIR>       4096 ostree
<DIR>       4096 boot

The actual file system is deep underneath one of the ostree directories.

Apalis iMX8 # ext4ls mmc 0:1 ostree/deploy/torizon/deploy
<DIR>       4096 .
<DIR>       4096 ..
<DIR>       4096 6d0da931c27b980efdd68c4868f16aa074ed35e2f880d7b3f5b8e6a36b2f76e3.0
<DIR>       4096 46794ca23935e6265fdb78b1d956869b2de7e23a4b2f79023c5a7409e9d9a78c.0
              82 46794ca23935e6265fdb78b1d956869b2de7e23a4b2f79023c5a7409e9d9a78c.0.origin
              82 6d0da931c27b980efdd68c4868f16aa074ed35e2f880d7b3f5b8e6a36b2f76e3.0.origin

This is where you will run into a minor problem. There is a command line limit of 80 characters and by the time you put in the long name you will overflow the command line processer and get weird error messages.

Apalis iMX8 # load mmc 0:1 ${scriptaddr} ostree/deploy/torizon/deploy/6d0da931c27b980efdd68c4                                                                              868f16aa074ed35e2f880d7b3f5b8e6a36b2f76e3.0/etc/fstab
Unknown command 'load' - try 'help'

Fortunately you can get away with using an environment variable and it makes it easier to do.

Apalis iMX8 # setenv /r ostree/deploy/torizon/deploy/6d0da931c27b980efdd68c4868f16aa074ed35e2f880d7b3f5b8e6a36b2f76e3.0

That makes it much easier to look at the actual root file system:

Apalis iMX8 # ls mmc 0:1 ${r}
<DIR>       4096 .
<DIR>       4096 ..
<SYM>          7 bin
<SYM>         17 home
<SYM>          7 lib
<SYM>         18 media
<SYM>         16 mnt
<SYM>         16 opt
<SYM>         14 ostree

From there if you need to fix a file that is corrupted, you need to load it into memory with ext4load and use the md command to confirm that. The startup script gives you a nice memory location ${scriptaddr} where you can stash it.

Apalis iMX8 # ext4load mmc 0 ${scriptaddr} ${r}/etc/fstab
650 bytes read in 37 ms (16.6 KiB/s)
Apalis iMX8 # md.b ${scriptaddr} ${filesize}
83100000: 23 20 73 74 6f 63 6b 20 66 73 74 61 62 20 2d 20    # stock fstab -
83100010: 79 6f 75 20 70 72 6f 62 61 62 6c 79 20 77 61 6e    you probably wan
83100020: 74 20 74 6f 20 6f 76 65 72 72 69 64 65 20 74 68    t to override th
...

Note that ${filesize} is set automatically by ext4load. Unfortunately in my case, this wasn’t the right version of /etc/fstab so I had to try again with the other location.

Apalis iMX8 # setenv r /ostree/deploy/torizon/deploy/46794ca23935e6265fdb78b1d956869b2de7e23a4b2f79023c5a7409e9d9a78c.0
Apalis iMX8 # ext4load mmc 0 ${scriptaddr} ${r}/etc/fstab
728 bytes read in 41 ms (16.6 KiB/s)
Apalis iMX8 # md.b ${scriptaddr} ${filesize}
83100000: 23 20 73 74 6f 63 6b 20 66 73 74 61 62 20 2d 20    # stock fstab -
83100010: 79 6f 75 20 70 72 6f 62 61 62 6c 79 20 77 61 6e    you probably wan
83100020: 74 20 74 6f 20 6f 76 65 72 72 69 64 65 20 74 68    t to override th
...

In this case I could see the errant line to fix

83100270: 66 61 75 6c 74 73 2c 73 79 6e 63 2c 6e 6f 61 75    faults,sync,noau
83100280: 74 6f 20 20 30 20 20 30 0a 0a 67 6f 6a 69 6d 65    to  0  0..gojime
83100290: 64 69 61 20 20 20 20 20 20 20 20 20 20 20 2f 6d    dia           /m
831002a0: 6e 74 2f 67 6f 6a 69 6d 65 64 69 61 20 20 20 20    nt/gojimedia
831002b0: 20 20 20 20 65 78 74 34 20 20 20 20 20 20 20 64        ext4       d

The base command allows pointing directly at it.

Apalis iMX8 # base 8310028a
Base Address: 0x8310028a

To change the first character to a # and comment it out, the mm.b command does the trick.

Apalis iMX8 # mm.b 0
8310028a: 67 ? 23
8310028b: 6f ? ^C

From there you can verify the entire contents to see that they are right with md.b ${scriptaddr} ${filesize}.

Now I need to write it back to the file system

Apalis iMX8 # ext4write mmc 0 ${scriptaddr} ${r}/etc/fstab ${filesize}
File System is consistent
file found, deleting
update journal finished
File System is consistent
update journal finished
728 bytes written in 4176 ms (0 Bytes/s)

With the file fixed, a quick powercycle to reboot and the system work like a champ again.

1 Like

Wow that is quite the solution to a fairly unique problem. For future reference, what might also work is the following. If you run ums 0 mmc 0 in U-Boot. This will let you mount the device’s eMMC as a USB device. I believe this is done via the USB OTG port. So if you run this command and connect to the device via USB OTG, then the filesystem will be made available like a USB drive on your PC.

That being said thought I’m glad to see you were able to come up with your own fairly impressive solution.

Best Regards,
Jeremias

Unfortunately we have not been able to determine why the USB port does not work with the iMX8 which limited the options available to me.

However, it does leave me with the question as to why the system gets into the state it does when it encounters a problem with failing to mount a drive.

I’m not sure about your USB issue, is this on your custom carrier board only?

As for the not booting if drive fails to mount issue. I’m not sure how you configured your /etc/fstab, but this is actually the “defaut” behavior for fstab entries that can’t be mounted. I believe you need to add a nofail parameter to your fstab entry. This allows the system to boot even if the entry can’t be mounted at boot time.

More details on this here: Mount an external drive at boot time only if it is plugged in - Ask Ubuntu

Best Regards,
Jeremias

Hello, I am using Colibri iMX7 Dual 1GB and I have encountered the same problem (fstab corrupted). It seems that the ext4write is not recognized. I have tried help and the command is not in the list. Why? Thank you.

@brnfde If you could please open another post describing your issue in detail. It’s been a while since the original issue so perhaps changes occurred that removed ext4write from U-Boot.

You could also try the alternative solution I recommend in my post here using ums 0 mmc 0.

Best Regards,
Jeremias