How to Deploy initramfs External to Kernel on i.MX8

jonk4m · July 22, 2022, 8:24pm

I’m using a tdx reference image and this guide to deploy initramfs external to the Kernel. I successfully generated a .cpio.gz file, wrapped it for U-Boot using:

mkimage -A arm -O linux -T ramdisk -n "Initial Ram Disk" -d Apalis-iMX8_Reference-Minimal-Image-rt.rootfs.cpio.gz initramfs.img

Then copied it to a USB drive. With the i.MX8 in the Ixora carrier board, I powered on, stopped it from auto-booting, and ran from U-Boot:

load usb 0:1 ${ramdisk_addr_r} initramfs.img " which spit out: "49482086 bytes read in 3085 ms (15.3 MiB/s)

Next I ran:

fatwrite mmc 0:1 ${ramdisk_addr_r} initramfs.img ${filesize}

only to see:

Error: no space left: 49482086
Error: writing contents

** Unable to write "initramfs.img" from mmc 0:1 **

I’m not sure why I am seeing this error.

Here’s additional information:

mmc info:

Device: FSL_SDHC
Manufacturer ID: 13
OEM: 14e
Name: S0J56
Bus Speed: 52000000
Mode: MMC High Speed (52MHz)
Rd Block Len: 512
MMC version 5.1
High Capacity: Yes
Capacity: 14.8 GiB
Bus Width: 8-bit
Erase Group Size: 512 KiB
HC WP Group Size: 8 MiB
User Capacity: 14.8 GiB WRREL
Boot Capacity: 31.5 MiB ENH
RPMB Capacity: 4 MiB ENH

mmc part:

Partition Map for MMC device 0  --   Partition Type: DOS
Part  Start Sector      Num Sectors    UUID        Type
  1   8192             98304          ea83ffae-01 0c
  2   106496           30973952       ea83ffae-02 83

fatls mmc 0:
   169408   imx8qm-apalis-v1.1-ixora-v1.2.dtb
   102464   dpfw.bin
       43   overlays.txt
            overlays/
   169098   imx8qp-apalis-v1.1-ixora-v1.2.dtb
   167782   imx8qm-apalis-ixora-v1.1.dtb
   106496   hdmitxfw.bin
   166541   imx8qp-apalis-v1.1-eval.dtb
     5754   boot.scr
 10625018   Image.gz
   167405   imx8qp-apalis-v1.1-ixora-v1.1.dtb
   167667   imx8qm-apalis-v1.1-ixora-v1.1.dtb
   166842   imx8qm-apalis-eval.dtb
   166807   imx8qm-apalis-v1.1-eval.dtb

13 file(s), 1 dir(s)

print filesize:
filesize=2f30966

SATA is obviously available to the Kernel (since I can Read/Write data to the SSD during normal operation) but not U-Boot by default on i.MX8, so my ultimate goal is to use an initramfs bundled or external to the kernel to then tell the kernel (init file) to bring up the root filesystem on-board the SSD. In this scenario, the kernel and U-boot will remain on the eMMC with the initramfs, but the actual filesystem that will regularly be used will reside on the SSD.

rafael.tx · July 25, 2022, 10:27am

Hello @jonk4m,
What is the size of your boot partition? For our default images the boot partition is not big, and it seems that your initrd has 48Mb of size. At least in my apalis it would not fit with the standard partitioning:

root@apalis-imx8-06852132:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                14.5G    209.3M     13.5G   1% /
devtmpfs                  1.3G      4.0K      1.3G   0% /dev
tmpfs                     1.8G         0      1.8G   0% /dev/shm
tmpfs                     1.8G      6.5M      1.8G   0% /run
tmpfs                     1.8G         0      1.8G   0% /sys/fs/cgroup
tmpfs                     1.8G         0      1.8G   0% /tmp
tmpfs                     1.8G     20.0K      1.8G   0% /var/volatile
/dev/mmcblk0p1           47.2M     11.6M     35.6M  25% /boot
/dev/sda1                57.2G      4.9G     49.4G   9% /media/sda1
tmpfs                   371.3M         0    371.3M   0% /run/user/0
root@apalis-imx8-06852132:~# 
root@apalis-imx8-06852132:~# cat /etc/issue
TDX Wayland with XWayland 5.7.0-devel-20220630184016+build.0 (dunfell) \n \l
Apalis-iMX8_Reference-Minimal-Image

jonk4m · July 25, 2022, 12:37pm

Rafael,

You are absolutely right, thank you!

root@apalis-imx8-06944687:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                14.5G    194.6M     13.5G   1% /
devtmpfs                  1.3G      4.0K      1.3G   0% /dev
tmpfs                     1.8G         0      1.8G   0% /dev/shm
tmpfs                     1.8G      6.4M      1.8G   0% /run
tmpfs                     1.8G         0      1.8G   0% /sys/fs/cgroup
tmpfs                     1.8G         0      1.8G   0% /tmp
tmpfs                     1.8G     16.0K      1.8G   0% /var/volatile
/dev/mmcblk0p1           47.2M     11.6M     35.6M  25% /boot
tmpfs                   371.3M         0    371.3M   0% /run/user/0
root@apalis-imx8-06944687:~# cat /etc/issue
TDX Wayland with XWayland 5.6.0+build.18 (dunfell) \n \l
Apalis-iMX8_Reference-Minimal-Image

I am trying to fit 49MB into 35.6MB just like you said.

So would it be easier to reduce my initrd size or change the default boot partition size?

jonk4m · July 25, 2022, 3:44pm

As a follow-up, I realized I could just use

bitbake -k initramfs-debug-image

to get a much smaller footprint for the ramfs. I then ran

mkimage -A arm -O linux -T ramdisk -n "Initial Ram Disk" -d initramfs-debug-image-rt-apalis-imx8-20220725132934.rootfs.cpio.gz initramfs.img

and copied the .img file to a usb drive.

After stopping U-Boot, I run:

env default -f -a
saveenv
usb start
load usb 0:1 ${ramdisk_addr_r} initramfs.img  #Load the drive's image into RAM
fatwrite mmc 0:1 ${ramdisk_addr_r} initramfs.img ${filesize} #Write the drive's image from RAM into the eMMC
setenv fdt_file ${soc}-apalis${variant}-${fdt_board}.dtb #define the fdt file
setenv boot_file zImage #"image" is the original 
saveenv
load mmc 0:1 ${fdt_addr_r} ${fdt_file} #load the fdt file into RAM
load mmc 0:1 ${ramdisk_addr_r} initramfs.img #load the drive's initramfs file back into RAM
load mmc 0:1 ${kernel_addr_r} ${boot_file} #load the kernel into RAM
bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} #boot the kernel and then initramfs

However, I get:

Wrong Image Format for bootm command
ERROR: can't get kernel image!

from the bootm command. I’m confused why the bootm command would not work in this case. I’ve also tried it with “setenv boot_file image” instead.

When I run

load mmc 0:1 ${kernel_addr_r} ${boot_file}

I don’t get an error message, but I also don’t get a “bytes read: xxx” message either.

rafael.tx · July 25, 2022, 4:18pm

Good that you found out a way to get a smaller image. Another option would be to edit your image.json file, on the part that is creating the boot partition and using a bigger size:

"partitions": [                                                                                                                                                                                                                                                                                                    
                {                                                                                                                                                                                                                                                                                                              
                    "partition_size_nominal": 48,                                                                                                                                                                                                                                                                              
                    "want_maximised": false,                                                                                                                                                                                                                                                                                   
                    "content": {                                                                                                                                                                                                                                                                                               
                        "label": "BOOT",                                                                                                                                                                                                                                                                                       
                        "filesystem_type": "FAT",                                                                                                                                                                                                                                                                              
                        "mkfs_options": "",                                                                                                                                                                                                                                                                                    
                        "filename": "Reference-Multimedia-Image-verdin-imx8mp.bootfs.tar.xz",                                                                                                                                                                                                                                  
                        "uncompressed_size": 10.52734375                                                                                                                                                                                                                                                                       
                    }                                                                                                                                                                                                                                                                                                          
                },

At the moment, our tezi automatic image creation process calculates the size of the boot partition based on the contents that need to be written there meta-toradex-bsp-common/classes/image_type_tezi.bbclass:

# Make an educated guess of the needed boot partition size                                                                                                                                                                                                                                                                     
# max(16MB, 3x the size of the payload rounded up to the next 2^x number)                                                                                                                                                                                                                                                      
def get_bootfs_part_size(d):                                                                                                                                                                                                                                                                                                   
    from math import log                                                                                                                                                                                                                                                                                                       
    part_size = 3 * 2 ** (1 + int(log(get_uncompressed_size(d, 'bootfs'), 2)))                                                                                                                                                                                                                                                 
    return max(16, part_size)

As for your second question, it is really odd that you don’t see the “bytes read:xxx”, for me that would mean that the kernel image was not loaded to RAM and this could cause bootm to fail.

I just did a quick test here, and you should see something similar to this if the kernel image was properly loaded:

Apalis iMX6 # fatload mmc 0:1 $kernel_addr_r zImage
7264776 bytes read in 276 ms (25.1 MiB/s)

jonk4m · July 25, 2022, 5:20pm

Rafeal,

Thanks again for the information!

When I update the boot_file variable to be:

setenv boot_file Image.gz

I manage to get

fatload mmc 0:1 ${kernel_addr_r} Image.gz
10625018 bytes read in 344 ms (29.5 MiB/s)

However, I still get the same kernel error message

Apalis iMX8 # bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}
Wrong Image Format for bootm command
ERROR: can't get kernel image!

No other image type (Image, zImage, uImage, etc.) seems to generate the “byte read” output from the command.

jonk4m · July 25, 2022, 5:36pm

Additional Info :

Apalis iMX8 # fatinfo mmc 0:1
Interface:  MMC
  Device 0: Vendor: Man 000013 Snr 1f8ff265 Rev: 1.0 Prod: S0J56X
            Type: Removable Hard Disk
            Capacity: 15176.0 MB = 14.8 GB (31080448 x 512)
Filesystem: FAT32 "BOOT"



Apalis iMX8 # fatinfo mmc 0:1
Interface:  MMC
  Device 0: Vendor: Man 000013 Snr 1f8ff265 Rev: 1.0 Prod: S0J56X
            Type: Removable Hard Disk
            Capacity: 15176.0 MB = 14.8 GB (31080448 x 512)
Filesystem: FAT32 "BOOT       "
Apalis iMX8 # fatls mmc 0:1 boot  
Apalis iMX8 # fatls mmc 0:1     
   169408   imx8qm-apalis-v1.1-ixora-v1.2.dtb
   102464   dpfw.bin
       43   overlays.txt
            overlays/
   169098   imx8qp-apalis-v1.1-ixora-v1.2.dtb
   167782   imx8qm-apalis-ixora-v1.1.dtb
   106496   hdmitxfw.bin
   166541   imx8qp-apalis-v1.1-eval.dtb
     5754   boot.scr
 10625018   Image.gz
   167405   imx8qp-apalis-v1.1-ixora-v1.1.dtb
   167667   imx8qm-apalis-v1.1-ixora-v1.1.dtb
   166842   imx8qm-apalis-eval.dtb
   166807   imx8qm-apalis-v1.1-eval.dtb
  2004407   initramfs.img

14 file(s), 1 dir(s)
 


Apalis iMX8 # printenv
arch=arm
baudrate=115200
board=apalis-imx8
board_name=Apalis iMX8QM
board_rev=v1.0
boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
boot_efi_binary=if fdt addr ${fdt_addr_r}; then bootefi bootmgr ${fdt_addr_r};else bootefi bootmgr ${fdtcontroladdr};fi;load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/boot/bootaa64.efi; if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r};else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi
boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}${boot_syslinux_conf}
boot_fdt=try
boot_file=Image.gz
boot_net_usb_start=usb start
boot_pci_enum=pci enum
boot_prefixes=/ /boot/
boot_script_dhcp=boot.scr
boot_scripts=boot.scr
boot_syslinux_conf=extlinux/extlinux.conf
boot_targets=mmc1 mmc2 mmc0 usb0 dhcp 
bootcmd=run distro_bootcmd
bootcmd_dhcp=setenv devtype dhcp; run boot_net_usb_start; run boot_pci_enum; if dhcp ${scriptaddr} ${boot_script_dhcp}; then source ${scriptaddr}; fi;setenv efi_fdtfile ${fdtfile}; setenv efi_old_vci ${bootp_vci};setenv efi_old_arch ${bootp_arch};setenv bootp_vci PXEClient:Arch:00011:UNDI:003000;setenv bootp_arch 0xb;if dhcp ${kernel_addr_r}; then tftpboot ${fdt_addr_r} dtb/${efi_fdtfile};if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r}; else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi;fi;setenv bootp_vci ${efi_old_vci};setenv bootp_arch ${efi_old_arch};setenv efi_fdtfile;setenv efi_old_arch;setenv efi_old_vci;
bootcmd_mfg=select_dt_from_module_version && fastboot 0
bootcmd_mmc0=devnum=0; run mmc_boot
bootcmd_mmc1=devnum=1; run mmc_boot
bootcmd_mmc2=devnum=2; run mmc_boot
bootcmd_usb0=devnum=0; run usb_boot
bootcount=1
bootdelay=1
commit_atf=835a8f6
commit_mkimage=8947fea3
commit_scfw=99e479ce
commit_secofw=2f7f6f59
console=ttyLP1 earlycon
cpu=armv8
defargs=pci=nomsi
distro_bootcmd=scsi_need_init=; for target in ${boot_targets}; do run bootcmd_${target}; done
efi_dtb_prefixes=/ /dtb/ /dtb/current/
ethaddr=00:14:2d:69:f7:af
fastboot_dev=mmc0
fdt_addr_r=0x83000000
fdt_board=eval
fdt_file=imx8qm-apalis-v1.1-eval.dtb
fdtcontroladdr=fd650ba0
fdtfile=imx8qm-apalis-v1.1-eval.dtb
fileaddr=96000000
filesize=167a
finduuid=part uuid mmc ${mmcdev}:2 uuid
hdp_addr=0x9c000000
hdp_file=hdmitxfw.bin
ipaddr=192.168.10.2
kernel_addr_r=0x96000000
load_efi_dtb=load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${prefix}${efi_fdtfile}
loadaddr=0x87000000
loadhdp=${load_cmd} ${hdp_addr} ${hdp_file}
loadm4image_0=${load_cmd} ${loadaddr} ${m4_0_image}
loadm4image_1=${load_cmd} ${loadaddr} ${m4_1_image}
m4_0_image=m4_0.bin
m4_1_image=m4_1.bin
m4boot_0=run loadm4image_0; dcache flush; bootaux ${loadaddr} 0
m4boot_1=run loadm4image_1; dcache flush; bootaux ${loadaddr} 1
mmc_boot=if mmc dev ${devnum}; then devtype=mmc; run scan_dev_for_boot_part; fi
mmcargs=setenv bootargs console=${console},${baudrate} root=PARTUUID=${uuid} rootwait mmcdev=0
mmcautodetect=yes
mmcdev=0
mmcpart=1
mmcroot=/dev/mmcblk0p2 rootwait rw
netmask=255.255.255.0
panel=NULL
preboot=setenv fdtfile ${soc}-apalis${variant}-${fdt_board}.dtb
ramdisk_addr_r=0x8a000000
rootpath=/srv/nfs
scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;run scan_dev_for_efi;
scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done; setenv devplist
scan_dev_for_efi=setenv efi_fdtfile ${fdtfile}; for prefix in ${efi_dtb_prefixes}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile}; then run load_efi_dtb; fi;done;if test -e ${devtype} ${devnum}:${distro_bootpart} efi/boot/bootaa64.efi; then echo Found EFI removable media binary efi/boot/bootaa64.efi; run boot_efi_binary; echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfile
scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${boot_syslinux_conf}; then echo Found ${prefix}${boot_syslinux_conf}; run boot_extlinux; echo SCRIPT FAILED: continuing...; fi
scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; done
scriptaddr=0x83100000
scsi_boot=run scsi_init; if scsi dev ${devnum}; then devtype=scsi; run scan_dev_for_boot_part; fi
scsi_init=if ${scsi_need_init}; then scsi_need_init=false; scsi scan; fi
sec_boot=no
serial#=06944687
serverip=192.168.10.1
setup=run loadhdp; hdp load ${hdp_addr}; run mmcargs
soc=imx8qm
soc_type=imx8qm
update_uboot=askenv confirm Did you load u-boot-dtb.imx (y/N)?; if test "$confirm" = "y"; then setexpr blkcnt ${filesize} + 0x1ff && setexpr blkcnt ${blkcnt} / 0x200; mmc dev 0 1; mmc write ${loadaddr} 0x0 ${blkcnt}; fi
usb_boot=usb start; if usb dev ${devnum}; then devtype=usb; run scan_dev_for_boot_part; fi
variant=-v1.1
vendor=toradex
ver=U-Boot 2020.04-5.6.0+git.7d1febd4af77 (Feb 16 2022 - 09:00:30 +0000)
video=imxdpufb5:off video=imxdpufb6:off video=imxdpufb7:off

Environment size: 5317/8188 bytes

The commands I ran:

env default -f -a
saveenv
usb start
load usb 0:1 ${ramdisk_addr_r} initramfs.img 
fatwrite mmc 0:1 ${ramdisk_addr_r} initramfs.img ${filesize}

setenv fdt_board ixora-v1.2
setenv fdt_file ${soc}-apalis${variant}-${fdt_board}.dtb
setenv boot_file Image.gz
saveenv
load mmc 0:1 ${fdt_addr_r} ${fdt_file}
load mmc 0:1 ${ramdisk_addr_r} initramfs.img
fatload mmc 0:1 ${kernel_addr_r} ${boot_file}

Apalis iMX8 # bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}
Wrong Image Format for bootm command
ERROR: can't get kernel image!

rafael.tx · July 26, 2022, 11:05am

Now thinking a little bit more, I think that the Image.gz file is not the right one. Bootm requires a specially formatted image that is created with the mkimage tool and this is also true for the kernel.
Of the top of my head I don’t know if the kernel build already generates this image automatically, they are usually called uImage. Could you check if this file is generated and if not try to generate it with mkimage?

jonk4m · July 26, 2022, 12:38pm

So I believe the options are:

Bootm - uImage format
Booti - Image format (newest version of U-Boot has Booti support Image.gz)
Bootz - zImage format

So I tried just unzipping my Image.gz after reading this. So now my commands are:

env default -f -a; saveenv
usb start
run setup
setenv fdt_file imx8qm-apalis-v1.1-eval.dtb 
setenv boot_file Image.gz
saveenv
load usb 0:1 ${ramdisk_addr_r} initramfs.img #load the initrd into RAM
load mmc 0:1 ${fdt_addr_r} ${fdt_file} #load the fdt file into RAM
fatload mmc 0:1 ${loadaddr} ${boot_file} #load the Image.gz file 
unzip ${loadaddr} ${kernel_addr_r} #unzip the Image.gz file into RAM since booti can't do Image.gz
setenv boot_file Image #change back just in case
saveenv
booti ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}

However, I get:

Apalis iMX8 # booti ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}
## Loading init Ramdisk from Legacy Image at 8a000000 ...
   Image Name:   Initial Ram Disk
   Image Type:   ARM Linux RAMDisk Image (gzip compressed)
   Data Size:    2004343 Bytes = 1.9 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 83000000
   Booting using the fdt blob at 0x83000000
   Loading Ramdisk to fd466000, end fd64f577 ... OK
ERROR: Failed to allocate 0x2bb97 bytes below 0xffe00000.
device tree - allocation error
FDT creation failed! hanging...### ERROR ### Please RESET the board ###

Did I run out of RAM?

Loading Ramdisk to fd466000, end fd64f577 ... OK

which in decimal is (0xFD466000) 4,249,247,744 to (0xFD64F577) 4,251,252,087.
And 4,251,252,087 - 4,249,247,744 = 2,004,343 Bytes. This makes sense since U-Boot outputted

2004407 bytes read in 127 ms (15.1 MiB/s)

After the "load usb 0:1 ${ramdisk_addr_r} initramfs.img " command.

Now 0x2BB97 is 179,095 Bytes, and it says “ERROR: Failed to allocate 0x2bb97 bytes below 0xffe00000” where 0xFFE00000 is 4,292,870,144 Bytes.

I wonder if the bootm_size and bootm_low parameters could be modified to solve this.

rafael.tx · July 28, 2022, 8:46pm

Hello @jonk4m,
At this point I would suggest that maybe you try to enable the debugging output for the booti.c and also bootm.c by adding

#define DEBUG

at the beginning of both files and then recompiling u-boot. I think that this could give you a better understanding of what’s going on with the boot process. By looking at the booti.c file, it seems to me that this command is prepared to decompress the kernel so that you could provide it with the zipped image.
I have never tested this so I cannot be 100% sure.
With the debug output enabled you should have some extra printfs that will show you where each piece is being allocated before the final stages are reached. It could also be helpful to try to follow the code to see where exactly is this error getting out.