What are people using to recover bricked devices in the field (assume no laptop available)?? This is similar to updating the image on the device without plugging in to a laptop?
There are conflicting reports on the use of SD card / USB for such purposes (conflicting being: can you do it or not).
I’ve got as far as “booting/loading kernel”, but I’m still not quite there for booting from USB.
I think a lot of that is going to come down to how exactly the device is bricked. You might wait for someone from toradex to respond, but from my understanding there is the boot loader (uboot), which launches the kernel.
Uboot has a lot of ways it can be configured that determines different boot steps. I had one of my updates failed, but was able to revert with the torizon rollback feature by setting the right variables. You can change settings to setup to boot from an sd card or other devices, but you will need serial console (aka laptop, and the connectors on your board) access to change the settings if uboot has the wrong configuration (ex. the configuration is to boot directly from the emmc without checking other boot sources).
If uboot is broken, then you will have to enter recovery mode and reflash uboot using uuu, or a similar method. It sounds like that’s not your problem.
For a previous board not from toradex, we had it setup to boot from the sd card, and used the sd card to copy files to the flash, and run nromally when no sd card was inserted. I have also several times used an SD card to reinstall tezi during testing with toradex boards, but my uboot variables were set to boot from emmc first instead of an sd card.
thanks for you reply.
We’re working through u-boot and making (trying) it boot from different places.
There are older resources on the Toradex website talking about USB and SD boot mechanisms, but then comments on the forum suggesting it is impossible.
If u-boot is trashed, then yes, plugging in seems the only way (and a fair compromise).
How feasible is:
if USB detected, boot from there.
if SD detected, write that image to eMMC and boot.
The other scenario here is easily allowing engineers to upgrade the O/S without having to connect via cable. There are several other cases where this would be useful; especially in the field where cable access is not feasible.
We’re looking at A/B partitioning for both the Kernel and Rootfs and I have a booting image that get’s us towards this case.
This should be possible, I have definitely done similar things before.
First, uboot is configured to check boot options in a certain order (assuming your uboot isn’t radically different then the versions I have used). If you printenv and find the exact boot commands that are being run, then setenv / savee to set it up right, you should be able to control the boot commands, but the exact way this works will depend on how uboot was compiled, and you might need to change your uboot. I am not exactly sure how much these vary between boards usually.
For the imx8mp I am using right now, and current uboot install, my commands look a bit like…
bootcmd=run distro_bootcmd
distro_bootcmd=setenv nvme_need_init; for target in ${boot_targets}; do run bootcmd_${target}; done
boot_targets=mmc1 mmc2 dhcp
bootcmd_mmc1=devnum=1; run mmc_boot
bootcmd_mmc2=devnum=2; run mmc_boot
…
mmc1 is the sd card for me, and mmc2 is the internal flash. This means it will try to boot from the sd card, and then from the internal flash. It will init the sd card, and then search for a boot script, typically boot.scr, which will give it instructions on how to boot from the files on that storage.
If you have a working board, its easy to just clone to an sd card using dd, which will at least let you run from the sd card. One old solution my company used was having our applications detect if we were running on the sd card, and displaying a page that allowed cloning to the flash, until we decide to just always use an sd card and never use the onboard flash.
It seems like the toradex preferred way (making assumptions here) is to load your image using their easy installer (tezi) and just setup the auto install. It will handle using their images to copy to the internal flash, but I haven’t had the chance to try loading a custom image onto a tezi image yet, we are still developing our application we will use with a toradex board.
I’m just working through u-boot boot.scr. I keep creating erroneous scripts, but I have worked out how to copy a good one from USB to emmc within uboot so I can keep re-trying.
My base script is the one from Toradex which tries mmc SD and dhcp, but does look similar to your suggestion.
Once I’ve got TOTAL control over boot.scr, I’ll move over to SD like you describe.
Thanks for all the details and discussion on this topic. What you have figured out is correct. If the device has a working u-boot then you can modify the uboot to boot from SD card or USB drive, and this could save you.
However, if you want to write an image into eMMC once the device is bricked, you could first boot Toradex Easy Installer (TEZI) from an external medium (SD/USB) and then auto-install your custom TEZI compatible BSP/Torizon OS image. This solution is shared on this post: Verdin-imx8mm - easy installer and custom image in a single place - #7 by bruno.tx?
MMC: no card present
switch to partitions #0, OK
mmc2(part 0) is current device
Scanning mmc 2:1...
Found U-Boot script /boot.scr
in the boot up text.
U-boot has loaded and checked for an SD card FIRST. No SD card is loaded, so moves on to eMMC where it find MY boot.scr file; we then load dtd, kernel etc etc.
Where are details on the intial boot process, SD, then eMMC. I’m not trying to change, just fully understand.
(if I put the SAME image on an SD card that I write to eMMC, behaviour is as expected).
Yes, most of these chips have switches or internal fuses to determine a boot order. In most cases this boot order is to a flash chip that loads uboot.
“Recovery mode” is often switching one of these switches to make the boot order setup so you can load uboot using UUU (universal update utility) with a usb connection to your computer. Of course there are other ways to boot, and for security purposes you can burn internal fuses to make it so no one can edit the boot (loader) process.
As far as why uboot is looking at the sd card itself, that is either hard coded into uboot, or the uboot environment variables tell it to. The environment variables are stored in flash, and loaded by uboot to determine user configured boot settings.
If you want to change the uboot boot order, you need to change the environment variable commands that tell it to look at mmc1 first, and save the variables, or you need to compile a new uboot that is hard coded to do what you want, and flash that.
I don’t think you are wanting to recompile uboot, or change the boot setting to not load uboot from the flash chip, so you just need to use a boot (or upgrade) script to change the environment variables if that’s what you want, or just set them within uboot itself using a serial console.
Carrier: Toradex UNKNOWN CARRIER BOARD V1.1B, Serial# 11420461
SEC0: RNG instantiated
Net: eth1: ethernet@30be0000, eth0: ethernet@30bf0000 [PRIME]
Hit any key to stop autoboot: 0
MMC: no card present
switch to partitions #0, OK
mmc2(part 0) is current device
Scanning mmc 2:1...
Found U-Boot script /boot.scr
after “hit any key” I can enter the uboot prompt.
printevn shows me this:
boot_targets=mmc1 mmc2 dhcp
which must be it!!!
I did notice that if I trash boot.scr within mmc2, the things start talking about networking, AKA, “dhcp”, so that all makes a little more sense (it did not make sense at first).
one day i will ask someone about the pre-uboot stage, but it’s probably not needed right now.
thanks for all your help, once I’ve booted from SD, I’ll try USB.