Troubleshooting Mender OTA Update Failure Due to U-Boot Environment Read Error on Verdin iMX8MP with Custom Yocto Image

Technical Support
1

Hello,

I am facing an issue with OTA updates via Mender on a custom Yocto image for the Verdin iMX8MP (branch scratchgap ).

Error observed in Mender:

2025-11-08 14:41:13.639 +0000 UTC info: Running Mender client 5.0.2
2025-11-08 14:41:13.64 +0000 UTC info: Deployment with ID 2d57a34a-568c-4337-9f62-b4d775b6e2e5 started.
2025-11-08 14:41:13.85 +0000 UTC info: Installing artifact...
2025-11-08 14:41:14.026 +0000 UTC info: Update Module output (stderr): Cannot read environment, using default
2025-11-08 14:41:14.026 +0000 UTC info: Update Module output (stderr): Cannot read default environment from file
2025-11-08 14:41:14.027 +0000 UTC error: Process returned non-zero exit status: Download: Update Module returned non-zero status: Process exited with status 243
2025-11-08 14:41:14.028 +0000 UTC error: Operation canceled: GET https://s3.docker.mender.io/mender/ac767bfa-9985-4c91-8874-6f00ccfdacaf?... HTTP request cancelled

The update fails with an error related to reading the U-Boot environment:
Cannot read environment, using default.

Observations on the system:

  • Content of /etc/fw_env.config:
/dev/mmcblk1 0x800000 0x20000
/dev/mmcblk1 0x1000000 0x20000
  • The command fw_printenv -c /etc/fw_env.config returns:
Cannot read environment, using default
Cannot read default environment from file
  • Direct reading of the supposed environment storage zones on /dev/mmcblk1 shows blocks filled with zeros, meaning no valid environment at these addresses:
$ dd if=/dev/mmcblk1 bs=1 skip=$((0x800000)) count=$((0x20000)) | hexdump -C | head -n 40
# all bytes 0x00

In the U-Boot boot logs:

Loading Environment from MMC... Reading from MMC2... OK
mmc1 is current device
Scanning mmc 11...
Found U-Boot script boot.scr ...

This indicates that the U-Boot environment is actually loaded from MMC2 (/dev/mmcblk2), while the current fw_env.config points to MMC1 (/dev/mmcblk1).

Problematic question:

  • How to precisely identify the correct location of the U-Boot environment on the Verdin iMX8MP hardware?
  • How to adjust the configuration so that fw_printenv and Mender point to this valid environment area?
  • What are the recommendations to verify or initialize this environment so that Mender OTA updates function without errors?

Context:

I am building a custom Yocto image based on the branche scarthgap for the Verdin iMX8MP. I am trying to implement and test OTA updates through Mender. The Mender client works but updates always fail due to an error reading the U-Boot environment.

Thanks in advance for your advice and help!

2

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.

Identifying the Correct U-Boot Environment Location

Based on your U-Boot boot logs showing “Loading Environment from MMC… Reading from MMC2… OK”, the environment is indeed stored on MMC2 (/dev/mmcblk2) rather than MMC1 (/dev/mmcblk1) as configured in your /etc/fw_env.config file (1).

Correcting the fw_env.config File

For the Verdin iMX8MP, the U-Boot environment is typically stored in the eMMC boot partition (2). You need to update your /etc/fw_env.config file to point to the correct device and offset.

Based on the community experience with similar issues, the correct configuration should be (3):

/dev/mmcblk2boot0 -0x2200 0x2000

The key points are (3):

  • Use the raw device (/dev/mmcblk2boot0) rather than a partition
  • The U-Boot environment is stored at a specific offset within the eMMC boot area, separate from regular partitions
  • The negative offset (-0x2200) indicates the environment is stored relative to the end of the boot partition

Verifying the Configuration

After updating your /etc/fw_env.config file, test the configuration by running (4):

# fw_printenv

This should now successfully read the U-Boot environment variables without the “Cannot read environment” error (4).

Understanding Environment Storage Location

The U-Boot environment on Toradex modules is stored in a predefined location in the board’s non-volatile memory (1). As part of the boot process, U-Boot checks the Cyclic Redundancy Check (CRC) of the configured environment area, and if valid, loads it from flash (1).

For eMMC-based modules like the Verdin iMX8MP, the environment is located at the end of the primary eMMC hardware boot area partition, just before the config block (2).

Additional Troubleshooting

If you continue to experience issues:

  1. Verify the block device exists: Check that /dev/mmcblk2boot0 is available on your system
  2. Check U-Boot configuration: The environment location is defined at U-Boot compilation time and must match the fw_env.config settings (1)
  3. Ensure proper permissions: The fw_printenv and fw_setenv commands must be executed as root (4)

Important Notes for Custom Yocto Images

When building custom Yocto images, ensure that:

  • The U-Boot configuration matches your target hardware
  • The fw_env.config file is properly configured for your specific setup
  • Both U-Boot and the Linux tools are compiled from the same source with the same configuration (1)

This should resolve your Mender OTA update issues by allowing proper access to the U-Boot environment variables.

Best regards, Toradex AI Assistant.

Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

3

Hello @ToradexAI,

Indeed, by configuring the /etc/fw_env.config file with the following line:

text

/dev/mmcblk2boot0 -0x2200 0x2000

I can correctly read the U-Boot environment variables using the command fw_printenv:

text

rroot@verdin-imx8mp-08626446:/# fw_printenv
arch=arm
baudrate=115200
board=verdin-imx8mp
board_name=verdin-imx8mp
boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
boot_efi_binary=load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/boot/bootaa64.efi; if fdt addr -q ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r};else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi
boot_efi_bootmgr=if fdt addr -q ${fdt_addr_r}; then bootefi bootmgr ${fdt_addr_r};else bootefi bootmgr;fi
boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}${boot_syslinux_conf}
boot_net_usb_start=usb start
boot_pci_enum=pci enum
boot_prefixes=/ /boot/
boot_script_dhcp=boot.scr
boot_scripts=boot.scr.uimg boot.scr
boot_syslinux_conf=extlinux/extlinux.conf
boot_targets=mmc1 mmc2 dhcp 
bootcmd=run distro_bootcmd
bootcmd_dhcp=devtype=dhcp; run boot_net_usb_start; run boot_pci_enum; if dhcp ${scriptaddr} ${boot_script_dhcp}; then source ${scriptaddr}; fi;setenv efi_fdtfile ${fdtfile}; setenv efi_old_vci ${bootp_vci};setenv efi_old_arch ${bootp_arch};setenv bootp_vci PXEClient:Arch:00011:UNDI:003000;setenv bootp_arch 0xb;if dhcp ${kernel_addr_r}; then tftpboot ${fdt_addr_r} dtb/${efi_fdtfile};if fdt addr -q ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r}; else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi;fi;setenv bootp_vci ${efi_old_vci};setenv bootp_arch ${efi_old_arch};setenv efi_fdtfile;setenv efi_old_arch;setenv efi_old_vci;
bootcmd_mmc1=devnum=1; run mmc_boot
bootcmd_mmc2=devnum=2; run mmc_boot
bootcount=1
bootdelay=1
carrier_serial#=11490742
console=ttymxc2
cpu=armv8
distro_bootcmd=setenv nvme_need_init; for target in ${boot_targets}; do run bootcmd_${target}; done
efi_dtb_prefixes=/ /dtb/ /dtb/current/
eth1addr=00:14:2d:93:a1:0e
ethact=ethernet@30bf0000
ethaddr=00:14:2d:83:a1:0e
ethprime=eth0
fdt_addr_r=0x50200000
fdt_board=dev
fdtcontroladdr=ffacf940
fdtfile=imx8mp-verdin-wifi-dev.dtb
foo=bar
initrd_addr=0x43800000
initrd_high=0xffffffffffffffff
kernel_addr_r=0x48200000
kernel_comp_addr_r=0x40200000
kernel_comp_size=0x08000000
load_efi_dtb=load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${prefix}${efi_fdtfile}
loadaddr=0x48200000
mmc_boot=if mmc dev ${devnum}; then devtype=mmc; run scan_dev_for_boot_part; fi
nvme_boot=run boot_pci_enum; run nvme_init; if nvme dev ${devnum}; then devtype=nvme; run scan_dev_for_boot_part; fi
nvme_init=if ${nvme_need_init}; then setenv nvme_need_init false; nvme scan; fi
preboot=test -n ${fdtfile} || setenv fdtfile imx8mp-verdin-${variant}-${fdt_board}.dtb
ramdisk_addr_r=0x50300000
scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;run scan_dev_for_efi;
scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then part uuid ${devtype} ${devnum}:${distro_bootpart} distro_bootpart_uuid ; run scan_dev_for_boot; fi; done; setenv devplist
scan_dev_for_efi=setenv efi_fdtfile ${fdtfile}; for prefix in ${efi_dtb_prefixes}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile}; then run load_efi_dtb; fi;done;run boot_efi_bootmgr;if test -e ${devtype} ${devnum}:${distro_bootpart} efi/boot/bootaa64.efi; then echo Found EFI removable media binary efi/boot/bootaa64.efi; run boot_efi_binary; echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfile
scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${boot_syslinux_conf}; then echo Found ${prefix}${boot_syslinux_conf}; run boot_extlinux; echo EXTLINUX FAILED: continuing...; fi
scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; done
scriptaddr=0x50280000
serial#=08626446
soc=imx8m
stderr=serial@30880000
stdin=serial@30880000
stdout=serial@30880000
test_var=123
update_uboot=askenv confirm Did you load flash.bin (y/N)?; if test "$confirm" = "y"; then setexpr blkcnt ${filesize} + 0x1ff && setexpr blkcnt ${blkcnt} / 0x200; mmc dev 2 1; mmc write ${loadaddr} 0x0 ${blkcnt}; fi
usb_boot=usb start; if usb dev ${devnum}; then devtype=usb; run scan_dev_for_boot_part; fi
usb_ignorelist=0x1050:*,
variant=wifi
vendor=toradex
ver=U-Boot 2024.07-7.5.0-devel+git.3f772959501c (Jul 01 2024 - 18:07:18 +0000)

However, when performing an OTA update with Mender, I get this error:

2025-11-10 09:08:48.493 +0000 UTC info: Running Mender client 5.0.2
2025-11-10 09:08:48.493 +0000 UTC info: Deployment with ID ... started.
2025-11-10 09:08:48.494 +0000 UTC info: Sending status update to server
2025-11-10 09:08:48.861 +0000 UTC info: Installing artifact...
2025-11-10 09:08:49.318 +0000 UTC info: Update Module output (stderr): sh: bad number
2025-11-10 09:08:49.318 +0000 UTC info: Update Module output (stderr): sh: invalid number ''
2025-11-10 09:08:49.319 +0000 UTC error: Process returned non-zero exit status: Download: Update Module returned non-zero status: Process exited with status 1
2025-11-10 09:08:49.32 +0000 UTC error: Operation canceled: GET https://s3.docker.mender.io/mender/93753450-c8f2-4d72-b790-489ec98b1be8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=mender%2F20251110%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251110T090848Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3D%22release-test.mender%22&response-content-type=application%2Fvnd.mender-artifact&x-id=GetObject&X-Amz-Signature=77b03731cecd7110d2b6ba0c7babcca3fbd8d75b8c8e1b2829547c62c3b84cb2: HTTP request cancelled
2025-11-10 09:08:49.361 +0000 UTC info: Sending status update to server

I suppose this error is due to the absence of Mender environment variables such as mender_boot_part, since none appear in the output of fw_printenv.

I assume that this should be implemented with the meta-mender-toradex-nxp layer, but this layer is only available on the Kirkstone branch. So how can I do this while working on the Scarthgap branch?

Context:
I am building a custom Yocto image based on the Scarthgap branch for the Verdin iMX8MP. I am trying to implement and test OTA updates through Mender. The Mender client works but updates always fail due to an error reading the U-Boot environment.

Thanks in advance for your advice and help!

4

Mender Integration on Scarthgap Branch

Great that you’ve resolved the U-Boot environment access! However, you’re correct that the missing Mender-specific environment variables like mender_boot_part are causing the OTA update failures.

Understanding the Issue

The error sh: bad number and sh: invalid number '' indicates that Mender’s update module is trying to read environment variables that don’t exist or have empty values (1). This is typical when Mender-specific U-Boot environment variables haven’t been properly initialized.

Mender Integration Challenges on Scarthgap

You’re facing a known challenge: the meta-mender-toradex-nxp layer that provides seamless Mender integration for Toradex modules was primarily developed for the Kirkstone branch. For Scarthgap, you’ll need to handle the Mender integration manually.

Required Mender Environment Variables

For proper Mender operation, you typically need these U-Boot environment variables:

  • mender_boot_part - Current active boot partition
  • mender_boot_part_hex - Boot partition in hexadecimal format
  • upgrade_available - Flag indicating if an upgrade is available
  • bootcount - Boot attempt counter (you already have this)

Manual Setup Approach

Since you’re working with a custom Yocto image on Scarthgap, you’ll need to:

  1. Initialize Mender Variables in U-Boot

    You can set these variables manually using fw_setenv (2):

    fw_setenv mender_boot_part 1
fw_setenv mender_boot_part_hex 1
fw_setenv upgrade_available 0

  2. Ensure Proper Partitioning

    Mender requires a dual-partition setup for A/B updates. Your system needs:

    • Two rootfs partitions for A/B switching
    • Proper partition layout compatible with Mender’s expectations

  3. Custom Integration

    Since meta-mender-toradex-nxp isn’t available for Scarthgap, you may need to:

    • Manually integrate Mender’s core functionality
    • Adapt the Kirkstone layer components to work with Scarthgap
    • Use Mender’s generic integration approach rather than the Toradex-specific layer

Alternative Approach: Torizon OS

For OTA updates on Toradex hardware, consider using Torizon OS instead of a custom Yocto image (3). Torizon OS is specifically designed for OTA updates and provides:

  • Built-in OTA update capabilities via OSTree/Aktualizr
  • Container-based application deployment
  • Secure boot and update mechanisms
  • Full support for Toradex hardware including Verdin iMX8MP

Torizon OS is built using Yocto but provides a ready-to-use distribution with integrated OTA functionality (3).

Debugging Steps

To further diagnose the Mender issue:

  1. Check Mender Configuration

    • Verify /etc/mender/mender.conf has correct server settings
    • Ensure device authentication is properly configured

  2. Examine Update Module

    • Check which update module Mender is trying to use
    • Verify the update module has access to required environment variables

  3. Test Environment Variable Access

    • Run the same commands Mender uses to read environment variables
    • Verify all required variables are accessible

The integration of Mender with custom Yocto images requires careful attention to bootloader configuration, partition layout, and environment variable management (1). Given the complexity, you might find Torizon OS provides a more straightforward path to achieving reliable OTA updates on your Verdin iMX8MP.

Best regards, Toradex AI Assistant.

Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

5

Hi @stan!

Thanks for reaching out!

Since we already offer Torizon Cloud—which provides online/offline updates, device monitoring, remote access, fleet management, and more—we’ve focused our efforts there rather than supporting other update mechanisms.

By choosing Torizon OS and Torizon Cloud, you get all these features out of the box, without the added complexity of maintaining a custom BSP for your device.

That said, at the moment we don’t have any plans to support third-party OTA tools in the near future, either in our image or in the community.

Let us know if you have any other questions.

1 Like