IMX8MP Reboot Infinitly

unablesalt · December 18, 2023, 5:34pm

Hello Toradex team,

I hope you are all doing great! And you are ready for Christmas !

Hardware:

SoM Verdin IMX8M Plus Quad 4GB WB IT v1.1A
Heatsink Verdin Industrial Heatsink Type 1 V1.0B
Dual-Band Wi-Fi/Bluetooth PCB Antenna 100mm V1.0A
Custom Carrier Board based on Verdin

uname:

Linux verdin-imx8mp-14777482 5.15.129-6.4.0+git.67c3153d20ff #1-TorizonCore SMP PREEMPT Wed Sep 27 12:30:36 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Images tested:

torizon-core-docker-verdin-imx8mp-Tezi_6.4.0+build.5.tar (STABLE Release)

Guest OS:

macOS (M1 Pro ARM64)
Linux ubuntu (VM x86_64)

Issue:
So we are flashing our SOM using Unattended Flashing over Ethernet and we have a weird issue for certain SOM (3 already on 70 flashed). The SOM launch and infinite reboot (see picture)

Did you ever faced a similar issue with this hardware ? And do you have any idea how we could prevent this from happening ? Is it related to the image flashed ?

Best regards,
M

rudhi.tx · December 20, 2023, 12:34pm

Hello @unablesalt,

Thanks for reaching out!

I would like to gather more information to troubleshoot the issue:

Can you reproduce this issue on a Toradex carrier board for Verdin family?
Could you please share the specifications of the power supply you are using?
Can you reproduce this issue using your custom carrier board but with a different SoM?
What other devices/peripherals are connected to your carrier board?
Can you reproduce this issue if you disconnect all the other external devices/peripherals and keep just the SoM on the carrier board?

Also, wishing you a merry christmas

unablesalt · January 3, 2024, 3:16pm

Hi @rudhi.tx,

I hope you had a good time during Christmas time !

Sorry for the late reply, it has been a rush over the next few weeks and some vacation in between.

So for your questions:

Yes I can reproduce the same error with our Verdin, it have the same message as shown in the picture above
For the alimentation, we are using a the Mornsun LM150-23B24R2, we only need 50-60 Watts so it’s clearly enough
The Carrier board works with another SOM without any issue, so I don’t think that’s the hardware causing the issue.
The list is a bit long, goes from USB device, i2c devices, and others…
Yes, we can reproduce the bug without any device pluged, unfortunately. (actual the first thing we tried)

I hope you will be able to help us.

Best regards,
M

rudhi.tx · January 3, 2024, 5:50pm

Hello @unablesalt,

Happy New Year

Thanks for the detailed answer! This sounds more like an issue with that particular piece of module. Just to rule out any other possibilities, could you please check the following:

Load Toradex Easy Installer into that module. Check if that works - you should be able to see the GUI of TEZI. Then install Toradex Easy Installer into that module from the feeds. See if you can boot into the Toradex Easy Installer.

Please send me any interesting boot logs that you might get.

rudhi.tx · January 11, 2024, 11:55am

Hello @unablesalt,

Hope you are doing well! Do you have any updates?

unablesalt · January 12, 2024, 10:46am

Hi @rudhi.tx,

Thank you for reaching back so quickly.

Sorry I haven’t had the time to check, I’ll do it next week and keep you updated.

Best,
M

unablesalt · January 15, 2024, 1:36pm

Hi @rudhi.tx,

I hope you had a good weekend.

I tried multiple time this morning the boot in recovery and flash the Toradex Easy Installer without success. The device appear but when I launch recovery-linux.sh nothing happens.

I think the SOM might be brick…

I have tried another one that as a similar issue, and I was able to restore it using ToradexEasyInstaller_5.7.3 you can find the log:
2024.01.15.minicom.log (9.1 KB)

Then when I tried to flash again Toradex Easy Installer 5.7.3, it’s working same as before. I can boot and reboot without any issue (tried ~5 times).

But for Toradex Easy Installer 6.5.0+build.4 (2024-01-08) I have the similar issue as above after reboot:

reboot: Restarting system

U-Boot SPL 2022.04-6.5.0+git.28dc906f6107 (Dec 22 2023 - 09:31:10 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
Training FAILED
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
DDR configured as single rank
SEC0:  RNG instantiated
Normal Boot
WDT:   Started watchdog@30280000 with servicing (60s timeout)
Trying to boot from BOOTROM
Boot Stage: Primary boot
Find img info 0x&4802a600, size 888
�eed continue download 1024
...

I haven’t tried to flash my custom image of Torizon. And all the steps describe above has been done with the Verdin Development Board V1.1D.

Is it possible that the SOMM is not working with 6+ version ? What should I do now ?

Best regards,
M

rudhi.tx · January 17, 2024, 3:14pm

Hello @unablesalt,

This sounds like a hardware issue to me. You can initiate the RMA process here: Return Material Authorization (RMA)

SvenAlmgren · March 5, 2024, 1:56pm

Hi! We also have this issue on two iMX8MP Q 4GB WB IT 1.1A modules.

We run our custom Linux with custom u-boot on a custom carrier, but we’ve also verified this on a Dahlia board running the latest 6.5.0+build.9 (2024-01-08) Toradex Embedde Linux Reference Minimal Image. Also tried with 6.5.0 upstream, and had the same issue during 6.4.0.

5.7.5+build.27 seems to work (I have reset the board over 100 times without the issue manifesting), but looking at the output CAAM seems not to be enabled on that build.

If I press reset a couple of times the board will eventually boot.

If I disable CAAM during SPL init the issue goes away entirely.

diff --git a/board/toradex/verdin-imx8mp/spl.c b/board/toradex/verdin-imx8mp/spl.c
index b901038e90..2b66bab5be 100644
--- a/board/toradex/verdin-imx8mp/spl.c
+++ b/board/toradex/verdin-imx8mp/spl.c
@@ -50,6 +50,7 @@ void spl_dram_init(void)

 void spl_board_init(void)
 {
+#if 0
        if (IS_ENABLED(CONFIG_FSL_CAAM)) {
                struct udevice *dev;
                int ret;
@@ -58,6 +59,7 @@ void spl_board_init(void)
                if (ret)
                        printf("Failed to initialize caam_jr: %d\n", ret);
        }
+#endif

        /*
         * Set GIC clock to 500Mhz for OD VDD_SOC. Kernel driver does

The first board we saw this on booted most of the times, probably around 49 of 50 boots worked, or even more. The second board instead fail about 5 times out of 6.

CAAM is enabled during second u-boot initialization with our fix, so the issue seems to be related to CAAM during SPL and it’s always near Find img info in the console output, but it varies a bit with the exact config used during u-boot build. I don’t have the configs saved from my testings but I have some outputs saved.

The most common output is with the reset occurring right after BL31, and BL31 is only printed on the first (cold) boot, or after an external reset (the reset button). The loop seems to reach BL31 (or if it’s skipped, I’m not an expert here…)

This particular example got to print half of the secondary u-boot header before the reset.

U-Boot SPL 2022.04-6.4.0-devel+git.15fa90038d82 (Jan 31 2024 - 13:09:35 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
Training FAILED
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
DDR configured as single rank
SEC0:  RNG instantiated
Normal Boot
WDT:   Started watchdog@30280000 with servicing (60s timeout)
Trying to boot from BOOTROM
Boot Stage: Primary boot
Find img info 0x&4802e000, size 888
Download 1079296, Total size 1079856
NOTICE:  BL31: v2.6(release):lf_v2.6-g3c1583ba0a
NOTICE:  BL31: Built : 11:00:38, Nov 21 2022


U-Boot 2022.04-6.4.0-devel+git.15fa90038d82 (Jan 31 2024 - 13:
U-Boot SPL 2022.04-6.4.0-devel+git.15fa90038d82 (Jan 31 2024 - 13:09:35 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
Training FAILED
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
DDR configured as single rank
SEC0:  RNG instantiated
Normal Boot
WDT:   Started watchdog@30280000 with servicing (60s timeout)
Trying to boot from BOOTROM
Boot Stage: Primary boot
Find img info 0x&4802e000, size 888

U-Boot SPL 2022.04-6.4.0-devel+git.15fa90038d82 (Jan 31 2024 - 13:09:35 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
Training FAILED
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
DDR configured as single rank
SEC0:  RNG instantiated
Normal Boot
WDT:   Started watchdog@30280000 with servicing (60s timeout)
Trying to boot from BOOTROM
Boot Stage: Primary boot
Find img info 0x&4802e000, size 888

Disabling CAAM during SPL have so far fixed the issue on both the problematic modules, and works with all other modules we have too.

The same thing happens with the upstream image from Toradex Easy Installer

U-Boot SPL 2022.04-6.5.0+git.28dc906f6107 (Dec 22 2023 - 09:31:10 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
Training FAILED
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
DDR configured as single rank
SEC0:  RNG instantiated
Normal Boot
WDT:   Started watchdog@30280000 with servicing (60s timeout)
Trying to boot from BOOTROM
Boot Stage: Primary boot
Find img info 0x&4802a600, size 888
Need continue download 1024

U-Boot SPL 2022.04-6.5.0+git.28dc906f6107 (Dec 22 2023 - 09:31:10 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
Training FAILED
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
DDR configured as single rank
SEC0:  RNG instantiated
Normal Boot
WDT:   Started watchdog@30280000 with servicing (60s timeout)
Trying to boot from BOOTROM
Boot Stage: Primary boot
Find img info 0x&4802a600, size 888
Need continue download 1024

U-Boot SPL 2022.04-6.5.0+git.28dc906f6107 (Dec 22 2023 - 09:31:10 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
Training FAILED
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
DDR configured as single rank
SEC0:  RNG instantiated
Normal Boot
WDT:   Started watchdog@30280000 with servicing (60s timeout)
Trying to boot from BOOTROM
Boot Stage: Primary boot
Find img info 0x&4802a600, size 888
Need continue download 1024