Bricked Colibri Imx8 QuadPlus - clicking noise

Technical Support
, , , ,
1

I tried to upgrade my IMX8 QuadPlus (1.0B) modules from the BSP 3.0 builds that we have been using up to now with a newer firmware based on the BSP 5.0. I started with some minimal modifications to the Linux kernel (i2c sensors, 24 bit LCD) plus some of our custom apps (which does not matter in our case).

I packed the resulting tezi installer (with autoinstall set) on an SD-Card along with Tezi 2.0b6, put the SOM in a Viola board, put it into recovery mode and let it install. A few minutes at 30W power consumption later it was finished but never came to boot.

Interestingly, there is not output on the boot UART, but when putting it into recovery mode again, I can see

There is a test-pin on the SOM to the left of the Wifi module, seemingly the 1v8 supply - it seems to be dragged to a low state at constant intervals which correspond to audible clicks:

[upload|XgYK/Dvycukypvlqy3w17s7vM9w=]

The signal in detail seems quite consistent, so I still hope its a solvable software issue.
[upload|oe/ZLoK0h0kuavOyplRshZLbZQg=]

My guess would be an early init issue with the first-level u-boot; how can I debug/recover from this?
BR
Simon

P.S. The meta-layer revisions were:

meta-toradex-bsp-common
commit 588e17d17a72c1cc9457da2848fba2d805c4b2ea
Author: Oleksandr Suvorov <oleksandr.suvorov@toradex.com>
Date:   Thu Aug 6 16:41:31 2020 +0300

meta-toradex-demos
commit 93c179932a1a282bb56823ee202e387ced41f388
Author: Max Krummenacher <max.krummenacher@toradex.com>
Date:   Mon Sep 7 15:57:36 2020 +0200

meta-toradex-distro
commit 1e0d10b2851ac455618ab7e4c7db86c9ad1227a0
Author: Max Krummenacher <max.krummenacher@toradex.com>
Date:   Wed Sep 16 16:17:17 2020 +0200

meta-toradex-nxp
commit bbdecb1ea9ecc59b8777de9566fec45e0c99097b
Author: Marcel Ziswiler <marcel.ziswiler@toradex.com>
Date:   Thu Sep 17 12:22:23 2020 +0200
2

I would be willing to use a JTAG/SWD interface to upload p.ex the bootloader - an Olimex ARM-USB-OCD is on its way.

The mx8 variants are not mentioned yet on the JTAG knowledge base page - JTAG | Toradex Developer Center

Are there any specific instructions available for this SOM?

3

Later BSPs are only for V1.0C and later module versions. For older hardware versions one would need to explicitly use the following legacy machine:

http://git.toradex.com/cgit/meta-toradex-nxp.git/tree/conf/machine/colibri-imx8x-v10b.conf?h=dunfell-5.x.y

Luckily, one should be able to use the regular USB recovery mode e.g. with the Toradex Easy Installer to get that module back to life. But remember to explicitly install a suitable image otherwise it will be bricked again…

4

Thanks for the information, will build the image for this machine then; I’m sceptical about installability though: The SOM connects via OTG but disconnects again after 3-4 seconds (the Linux host reports an URB error), haven’t been able to move the tezi image over or even start the transfer.

Could I transfer an u-boot image alone (should be done in <1s) that sets the right startup parameters?

5

And you sure you did the proper procedure to get into recovery mode? Either press the recovery mode button on the Colibri Evaluation board or short the recovery mode pads on the module WHILE you power-cycle. Plus on those older early access samples you might have to hold it for 10 or MORE seconds. Only then it will be in recovery mode properly.

6

Ok, seems I am back to the old revision at least: I consistently saw errors when downloading newer Tezis but when downgrading it to 2.0b5 I could recover and flash a 3.x image.

Will try a new distro next.
Thanks a lot
Simon

7

The second 1.0B board that we have ist still refusing to accept a Tezi installer (tried b5,b6,b7, I didn’t find older ones…):

U-Boot 2018.03-toradex_imx_v2018.03_4.14.98_2.3.0_bringup+g9393e0c9bd (May 19 2020 - 21:36:48 +0000)

CPU:   Freescale i.MX8QXP revB A35 at 1200 MHz at 38C
DRAM:  2 GiB
PMIC:  device id: 0x40, revision id: 0x31, emrev 0, prog id fff
MMC:   FSL_SDHC: 0, FSL_SDHC: 1
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
Model: Toradex Colibri iMX8 QuadXPlus 2GB Wi-Fi / BT IT V1.0B, Serial# 06614352

 BuildInfo: 
  - SCFW b929edfe, SECO-FW f449a0d2, IMX-MKIMAGE d7f9440d, ATF bb209a0
  - U-Boot 2018.03-toradex_imx_v2018.03_4.14.98_2.3.0_bringup+g9393e0c9bd 

Detect USB boot. Will enter fastboot mode!
Net:   eth0: ethernet@5b040000 [PRIME]
Fastboot: Normal
Boot from USB for uuu
Hit any key to stop autoboot:  0 
Detect USB boot. Will enter fastboot mode!
Starting download of 2246 bytes

downloading of 2246 bytes finished
Detect USB boot. Will enter fastboot mode!
Starting download of 37580496 bytes
..........................................................................
..........................................................................
..........................................................................
................................................................
downloading of 37580496 bytes finished
## Executing script at 82e00000
This script is only meant for Apalis iMX8 / Apalis iMX8X / Colibri iMX8X
and Verdin iMX8MM

In comparison with the other 1.0B-Board the difference seems to be the board-variable which is set to ‘colibri-imx8x’ compared to ‘colibri-imx8qxp’ on the working module. I changed the variable and saved it, rebooted into recovery mode again to get

U-Boot 2018.03-toradex_imx_v2018.03_4.14.98_2.3.0_bringup+g9393e0c9bd (May 19 2020 - 21:36:48 +0000)

CPU:   Freescale i.MX8QXP revB A35 at 1200 MHz at 41C
DRAM:  2 GiB
PMIC:  device id: 0x40, revision id: 0x31, emrev 0, prog id fff
MMC:   FSL_SDHC: 0, FSL_SDHC: 1
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
Model: Toradex Colibri iMX8 QuadXPlus 2GB Wi-Fi / BT IT V1.0B, Serial# 06614352

 BuildInfo: 
  - SCFW b929edfe, SECO-FW f449a0d2, IMX-MKIMAGE d7f9440d, ATF bb209a0
  - U-Boot 2018.03-toradex_imx_v2018.03_4.14.98_2.3.0_bringup+g9393e0c9bd 

Detect USB boot. Will enter fastboot mode!
Net:   eth0: ethernet@5b040000 [PRIME]
Fastboot: Normal
Boot from USB for uuu
Hit any key to stop autoboot:  0 
Detect USB boot. Will enter fastboot mode!
Starting download of 2246 bytes

downloading of 2246 bytes finished
Detect USB boot. Will enter fastboot mode!
Starting download of 37576576 bytes
..........................................................................
..........................................................................
..........................................................................
................................................................
downloading of 37576576 bytes finished
## Executing script at 82e00000
## Loading kernel from FIT Image at 83000000 ...
Could not find configuration node
ERROR: can't get kernel image!

Where can I find an appropriate Tezi?

8

Another thing you might be doing wrong is starting the recovery-linux.sh script too early. The very latest Toradex Easy Installer 2.0b6 does first try doing USB recovery of older/obsolete SoC silicon die revisions after which it only accepts newer ones. So one would have to bring the module into recovery mode properly and only then start the script on the host.

9

Something really fishy seems to be going on in your case. The environment should actually really not matter in recovery mode. Anyway, you could try stopping in U-Boot and explicitly erasing the whole primary eMMC hardware area boot partition.

mmc dev 0 1
mmc erase 0x0 0xffffff

That most likely fails but should show you what end address/size might be suitable.

10

@marcel.tx thanks for the tip to clear the mmc - I did that and could recover the module by booting Tezi version 2.0b5 and set the module parameters manually (board type; version to 1.0B with capital B if that matters) .

Starting from there, Tezi 2.0b5 shows

  • both Tezi 2.0b5 as well as 2.0b6 as installable,
  • my original 3.0-Image is installable
  • the Toradex 5.1.0 minimal image is grayed out.

From there, I tried the following:

  • Reflashing 2.0b5 works fine,
  • flashing the original 3.0-based image works fine
  • Flashing and booting Tezi 2.0b6 results in the ‘bricked clicking’ state, had to re-install 2.0b5 to go on

Then I started playing with a freshly built demo image (Colibri-iMX8X-V10B_Reference-Minimal-Image-Tezi_5.1.0):

  • The 5.1.0 image is grayed out, upon inspection of the image.json file I saw that only the config_format is set to “4” (sic! - a string value instead of an int?!).
  • I changed this to two and installed - big step forward: the bootloader comes up at least but the boot config is broken, it does not find a kernel:

U-boot output as follows:

U-Boot 2020.04-5.1.0-devel+git.3d6a3c3b0841 (Oct 30 2020 - 20:23:52 +0000)

CPU:   NXP i.MX8QXP RevB A35 at 1200 MHz at 59C

DRAM:  2 GiB
MMC:   FSL_SDHC: 0, FSL_SDHC: 1
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
Model: Toradex Colibri iMX8 QuadXPlus 2GB Wi-Fi / BT IT V1.0B, Serial# 06614352

 BuildInfo: 
  - SCFW 732e719a, SECO-FW 376e3c15, IMX-MKIMAGE 6745ccdc, ATF 
  - U-Boot 2020.04-5.1.0-devel+git.3d6a3c3b0841 

flash target is MMC:0
Net:   eth0: ethernet@5b040000 [PRIME]
Fastboot: Normal
Normal Boot
Hit any key to stop autoboot:  0 
MMC: no card present
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
4126 bytes read in 17 ms (236.3 KiB/s)
## Executing script at 83200000
122607 bytes read in 15 ms (7.8 MiB/s)
14 bytes read in 11 ms (1000 Bytes/s)
Booting from mmc failed!
SCRIPT FAILED: continuing...
Found U-Boot script /boot.scr
4126 bytes read in 17 ms (236.3 KiB/s)
## Executing script at 83200000
122607 bytes read in 14 ms (8.4 MiB/s)
14 bytes read in 11 ms (1000 Bytes/s)
Booting from mmc failed!
SCRIPT FAILED: continuing...
starting USB...

… and so on failing further. So u-boot did load the boot script, loaded imx8qxp-colibri-eval-v3.dtb (122607 bytes), overlays.txt (14 bytes) and then failed. [EDIT:] ((How can I debug the boot script to get more debug info (i.e. edit & regenerate checksums)?))

[EDIT:] The problem here is Tezi 2.0b5 which tries to load Image as the kernel file, whereas the generated image packages Image.gz in bootfs. I repackaged the bootfs tar file with an uncompressed version to match the u-boot variables et voila - booting.

Simon

P.S. As for entering the OTG rescue mode - this works consistently well, I have an additional shell running with the journal output. When the line

Nov 17 09:35:07 travelmate kernel: hid-generic 0003:1FC9:012F.003E: hiddev0,hidraw1: USB HID v1.10 Device [NXP       SemiConductor Inc  SE Blank 8QXP ] on usb-0000:00:14.0-2/input0

appears the transfer works 100% (including a delay of 2s until I reach the keyboard, pliers on the solder bridge until reboot).

11

the Toradex 5.1.0 minimal image is grayed out.

Yes, this requires a more recent Toradex Easy Installer.

Flashing and booting Tezi 2.0b6 results in the ‘bricked clicking’ state, had to re-install 2.0b5 to go on

Yes, later Toradex Easy Installer versions may only be flashed to V1.0C and later modules. Initial early access prototypes and samples like the V1.0B you have are no longer supported. However, any of those should still be recoverable via USB recovery mode.

The 5.1.0 image is grayed out, upon inspection of the image.json file I saw that only the config_format is set to “4” (sic! - a string value instead of an int?!).

Yes, it simply requires a later Toradex Easy Installer as there is/was a critical bug which leads to all the later trouble you are seeing if you do not adhere to this.

I changed this to two and installed - big step forward: the bootloader comes up at least but the boot config is broken, it does not find a kernel:

No, big step backward: you are now experiencing the critical bug I mentioned above!

You really should just be using the very latest Toradex Easy Installer in USB recovery mode and all should be fine. Alternatively, you may want to try later V1.0C or later hardware which will be the only one supported long term going forward.