TEZI u-boot behavior

Hi Toradex! Good day,

Continue from this post: https://community.toradex.com/t/error-booting-tezi-from-mmcblk0p3/21227/12

When we do in-detailed testing, we figured TEZI will have issues to boot if the sdcard or eMMC partition has:
uninit_bg with unsigned_hash_directory

and it will only work if the filesystem features:

  1. metadata_csum activated
  2. uninit_bg with signed_hash_directory.

We are wondering how this can happen and whats the reasoning behind it.

TEZI: 5.7.4.1
Linux BSP 5.7.2
Toradex Apalis iMX6 V1.1C / V1.1D (MPN: 00281103/00281102)

Technical Support imx6 tezi

Hi @Charlie , how is the ext4 partition on mmcblk0p3 created and what options are passed to mkfs.ext4 ? For example, mkfs.ext4 -O^metadata_csum,^64bit /dev/mmcblk0p3.

Could you please detail how to uninit_bg with unsigned_hash_directory and uninit_bg with signed_hash_directory ?

partition 3 is created through OS installation, where we set in image.json.

image json per below.

{
    "config_format": 2,
    "autoinstall": true,
    "name": "project",
    "description": "Image without graphical interface",
    "version": "5.7.2",
    "release_date": "2024-01-23",
    "u_boot_env": "uEnv.txt",
    "prepare_script": "prepare.sh",
    "wrapup_script": "wrapup.sh",
    "marketing": "marketing.tar",
    "icon": "toradexlinux.png",
    "supported_product_ids": [
        "0027",
        "0028",
        "0029",
        "0035"
    ],
    "blockdevs": [
        {
            "name": "mmcblk0",
            "partitions": [
                {
                    "partition_size_nominal": 16,
                    "want_maximised": false,
                    "content": {
                        "label": "BOOT",
                        "filesystem_type": "FAT",
                        "mkfs_options": "",
                        "filename": "project-Image-apalis-imx6.bootfs.tar.xz",
                        "uncompressed_size": 6.19140625
                    }
                },
                {
                    "partition_size_nominal": 1536,
                    "want_maximised": false,
                    "content": {
                        "label": "RFS",
                        "filesystem_type": "ext4",
                        "mkfs_options": "",
                        "filename": "project-Image-apalis-imx6.tar.xz",
                        "uncompressed_size": 314.04296875
                    }
                },
                {
                    "partition_size_nominal": 256,
                    "want_maximised": false,
                    "content": {
                        "label": "TEZI",
                        "filesystem_type": "ext4",
                        "mkfs_options": "",
                        "filename": "project-TEZI-5741-apalis-imx6.tar.xz",
                        "uncompressed_size": 106.494
                    }
				},
                {
                    "partition_size_nominal": 1536,
                    "want_maximised": false,
                    "content": {
                        "label": "DATA",
                        "filesystem_type": "ext4",
                        "mkfs_options": "",
                        "filename": "project-files-apalis-imx6.tar.xz",
                        "uncompressed_size": 1
                    }
                }
            ]
        },
        {
            "name": "mmcblk0boot0",
            "erase": true,
            "content": {
                "filesystem_type": "raw",
                "rawfiles": [
                    {
                        "filename": "SPL",
                        "dd_options": "seek=2"
                    },
                    {
                        "filename": "u-boot.img",
                        "dd_options": "seek=138"
                    }
                ]
            }
        }
    ]
}

we never use mkfs.ext4 command to create this particular partition in our OS.

The issue started when we want to replace the content of TEZI 1.8 that we put in partition 3, with TEZI 5.7. With that, we use the approach of installing it using opkg.

bundle tezi in .ipk format → extract tezi into sdcard → copy tezi to partition 3.

The reason why we have to temporarily placed tezi in sdcard is because we would like to track the version of tezi that has been installed through opkg.

tune2fs output

passed unit #1 (sdcard - mmcblk1p1)

root@localhost:~# tune2fs -l /dev/mmcblk1p1
tune2fs 1.45.7 (28-Jan-2021)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          4ad90ae6-a3fc-db40-8777-fd3fa65953b3
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
**Filesystem flags:         signed_directory_hash**
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              123136
Block count:              492256
Reserved block count:     24612
Free blocks:              475382
Free inodes:              123125
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      240
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         7696
Inode blocks per group:   481
Flex block group size:    16
Filesystem created:       Thu Feb  8 05:57:59 2024
Last mount time:          Sun Sep 20 10:46:49 2020
Last write time:          Sun Sep 20 10:46:49 2020
Mount count:              1
Maximum mount count:      -1
Last checked:             Thu Jan  1 00:00:00 1970
Check interval:           0 (<none>)
Lifetime writes:          39 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      acecbb5f-f11d-a96a-fd6e-73afcd98f8b1
Journal backup:           inode blocks

Passed unit #2 (sdcard - mmcblk1p1)

root@localhost:~# tune2fs -l /dev/mmcblk1p1
tune2fs 1.45.7 (28-Jan-2021)
Filesystem volume name:   <none>
Last mounted on:          /media/sdcard
Filesystem UUID:          c36987d3-3dc9-45eb-a866-a5c2422b185e
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize **metadata_csum**
**Filesystem flags:         unsigned_directory_hash**
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              123136
Block count:              492256
Reserved block count:     24612
Free blocks:              453170
Free inodes:              123085
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      120
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         7696
Inode blocks per group:   481
Flex block group size:    16
Filesystem created:       Fri Nov 25 23:18:06 2022
Last mount time:          Sun Apr  3 02:16:52 2022
Last write time:          Sun Apr  3 02:16:52 2022
Mount count:              240
Maximum mount count:      -1
Last checked:             Fri Nov 25 23:18:06 2022
Check interval:           0 (<none>)
Lifetime writes:          35 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      c47a17d7-728b-4678-9202-d6a71f7b4ee2
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0x44bcf89c

Failed unit (sdcard-mmcblk1p1)

root@localhost:~# tune2fs -l /dev/mmcblk1p1
tune2fs 1.45.7 (28-Jan-2021)
Filesystem volume name:   <none>
Last mounted on:          /media/sdcard
Filesystem UUID:          85db12fe-f2e7-4e0f-937b-435a2ff38127
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file **uninit_bg** dir_nlink extra_isize
**Filesystem flags:         unsigned_directory_hash**
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              123136
Block count:              492256
Reserved block count:     24612
Free blocks:              417809
Free inodes:              122615
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      120
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         7696
Inode blocks per group:   481
Flex block group size:    16
Filesystem created:       Wed Jun 10 10:58:47 2020
Last mount time:          Fri Feb  2 10:42:56 2024
Last write time:          Fri Feb  2 10:42:56 2024
Mount count:              477
Maximum mount count:      -1
Last checked:             Wed Jun 10 10:58:47 2020
Check interval:           0 (<none>)
Lifetime writes:          69 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      f9ac4ac3-375d-4049-92d0-3693fe8229e6
Journal backup:           inode blocks

Hi @Charlie , thanks for sharing the information but I still have something unclear.

TEZI will have issues to boot if the sdcard or eMMC partition has:
uninit_bg with unsigned_hash_directory

Does it mean Tezi on eMMC partition 3 can not boot after the first installation? Your image.json above deployed Tezi to eMMC.

                {
                    "partition_size_nominal": 256,
                    "want_maximised": false,
                    "content": {
                        "label": "TEZI",
                        "filesystem_type": "ext4",
                        "mkfs_options": "",
                        "filename": "project-TEZI-5741-apalis-imx6.tar.xz",
                        "uncompressed_size": 106.494
                    }
				},

Or does it mean Tezi on eMMC partition 3 can not boot after it is updated by the following process? I guess this is your case now. The above case was addressed in another ticket.

bundle tezi in .ipk format → extract tezi into sdcard → copy tezi to partition 3.

The tune2fs outputs are all for sdcard. I don’t see the eMMC partition 3 filesystem features. In your image.json, "mkfs_options": "", for partition 3 is also empty. How aremetadata_csum, uninit_bg, and signed_hash_directory activated on eMMC partition 3 which hosts Tezi boot files?

It means that after we replaced TEZI that we copied from sdcard (with uninit_bg & unsigned_hash_directory on), the TEZI then will be unable to boot.

The solution that you’ve attached does not really solving the issue, or I would say does not explain why does it happened.

(The solution is to manually format the sdcard to have metadata_csum and 64bit on, but it would be hard to reformat sdcard on existing boards that have been released, and we would like to avoid user data in sdcard to be removed due to reformat)

We are wondering how such filesystem features can affect the behavior of TEZI and I dont have any place to have a good read on it.

P/S: we dont have such issue if we use TEZI 1.8

metadata_csum, and signed_hash_directory is activated on sdcard. we are not changing and will never change any filesystem features on eMMC partition 3 in this case.

You can find the output from tune2fs for eMMC partition 3

Filesystem volume name:   TEZI
Last mounted on:          /run/media/tmp
Filesystem UUID:          3a554c2f-99eb-4bd3-a5b3-786e4028dea8
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         unsigned_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              65536
Block count:              262144
Reserved block count:     13107
Free blocks:              137958
Free inodes:              65495
First block:              1
Block size:               1024
Fragment size:            1024
Reserved GDT blocks:      256
Blocks per group:         8192
Fragments per group:      8192
Inodes per group:         2048
Inode blocks per group:   256
Flex block group size:    16
Filesystem created:       Thu Jan  1 00:01:18 1970
Last mount time:          Sat Oct 16 04:29:26 2021
Last write time:          Sat Oct 16 04:29:26 2021
Mount count:              3
Maximum mount count:      -1
Last checked:             Thu Jan  1 00:01:18 1970
Check interval:           0 (<none>)
Lifetime writes:          117 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      21f6f141-b72f-4b14-a7b4-3bc673e91b7a
Journal backup:           inode blocks

Please bear in mind, we dont have the option to reformat eMMC partition 3 and we wish to maintain the filesystem features as above per as it is.

Hi @Charlie , what is the error on the serial console when Tezi on eMMC partition 3 fails to boot after finishing the process bundle tezi in .ipk format → extract tezi into sdcard → copy tezi to partition 3.?

After restart the instrument, it will goes into “cant get kernel image”

Hi @Charlie , please try dcache off before run boot_TEZI.

Hi Benjamin,

I did try that before. The issue still persist if I am using the bad sdcard (failed unit #1)

Hi @Charlie , to reproduce the issue, I would like to know how your sdcard is formated on a Linux host. The default feature options for a EXT4 partition comes from /etc/mke2fs.conf on the Linux PC. They can also be overwritten by the command mkfs.ext4 when creating the partition.

Is the uninit_bg with unsigned_hash_directory features mandatory for the EXT4 partition on your sd card?

Hi Benjamin,

Unfortunately the sdcard is formatted using windows or an app called DiskGenius…

Hi Benjamin,

Help me to understand, at which state is the best for u-boot (means: which filesystem features I have to turn on/off to make sure such kernel issue does not happen anymore.)

I did try another test by extracting the TEZI in partition 2 (mainly in folder /etc) and somehow I dont see any failures so far.

bundle the tezi in tar.gz --> extract it to "/etc" folder --> copy to emmcblk0p3

However, when I do a quick check on the tune2fs output, it is a bit contradict to my previous findings.

Partition 2 details per below:

Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize
Filesystem flags:         unsigned_directory_hash

I was wondering, does the issue comes from uboot unable to boot because of certain filesystem features, or is it because of opkg need certain requirement (e.g. maybe need to have 64bit to turn on?) to extract all the files to certain partition?

We would really like to deep dive into the issue without solving it blindly. Which is why I am reaching out to you maybe you have some answers :blush:

Hi @Charlie , Tezi v5’s U-boot has a different EXT4 filesystem driver than Tezi v1.8’s U-boot. struct ext_block_cache is missing in Tezi v1.8 U-boot. This could be the reason that the issue doesn’t happen on Tezi v1.8.

long int read_allocated_block(struct ext2_inode *inode, int fileblock,
			      struct ext_block_cache *cache)

The following are the default EXT4 features from the partition created by Tezi v5 during installation which should work with the u-boot.

Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize
Filesystem flags:         unsigned_directory_hash 

I suggest formatting your SD card on a Linux PC for better EXT4 features enabling. Here are features from my SD card which is formatted by DiskGenius v5.5.1.1508 x64 free edition. They are different from yous.

Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 

Thanks a lot for this! Do you mind to share me the full code or point me where to read so I can get a good read?

long int read_allocated_block(struct ext2_inode *inode, int fileblock,
			      struct ext_block_cache *cache)

I was wondering, may I know what dcache off does and how it influence the uboot?

I am sorry for disappear for weeks…

There are two URLs in the last replay that point to the source code. dcache off will not use the data caches of iMX6 SoC and it takes longer to load files from eMMC.

Hi Benjamin!

Thanks for the explanation. It helps alot for us to see it in brighter view.

Just an update… We noticed start address for one of tezi.itb component is different when this “misaligned address” issued appeared. It convinced us better than filesystem features might be not the sole root cause, but the operation of copying the TEZI into desired folder (by opkg, or copy command, or even tar extraction) caused some issues on the TEZI packages itself.

Is there any consequences of activating “dcache off” every time we would like to install operating system besides longer time to load files from eMMC?

Thanks!

Hi Charlie, we are unaware of other drawbacks. In my last test, it took much longer if dcache was off.

1 Like