Unable customize 6.1.0 or do any deploy on it

I encountered a strange issue while experimenting with the 6.1.0 December monthly release as a step up from our current development with 5.x

I am able to flash the basic image with TEZI and it boots (though without any display).

However, I was completely unable to make any sort of customizations to it - including trying to deploy a 5.x build via a deploy from my computer and torizoncore-builder. I tried dialling it back to the bare minimum set of changes (essentially just the LVDS DTS files) and it still failed.

In all cases, the system just bootloops and the serial output seems to suggest this is just the watchdog timer resetting it. In the last attempt to downgrade from 6.1 back to a known-good 5.6.x tcbuild configuration, it wasn’t even able to recover and just bootlooped endlessly (normally, it would just roll back after 3 cycles on the failed boot attempts) and I had to use TEZI to recover.

I’ve since gone back to 5.6.x for development, but I wanted to ask - is 6.1.x really that unstable at this stage? It has a few improvements from systemd v252 we would like to use eventually.

Greetings @bw908,

Just to clarify are you using TorizonCore Builder to jump between 5.X versions and 6.X versions of TorizonCore?

5.X and 6.X TorizonCore are different major versions of the OS and by definition have breaking changes between them. In fact it’s expected that trying to do a simple update/transition like this will more often than not result in failure.

We do plan to have a method to update between 5.X and 6.X versions, but it’s still a work in progress.

For the time being only transition to other versions of the OS with the same major version. For example from 5.6 to 5.7. If you want to play around with 6.X then completely re-flash the device with TEZI and don’t attempt to use TorizonCore Builder to jump between major versions for the time being.

Best Regards,
Jeremias

Not entirely, this was all within the context of having 6.x flashed via TEZI and then attempting to do a customized deploy of it from TCB. I only mentioned the cross-version jump as a final thing in case it was relevant.

So then what and how exactly did customizing with TorizonCore Builder fail? If you could give me an exact step by step of what you did so I can try and reproduce.

I understand you flashed 6.1.0 on your device, then what did you do from there? And what kind of failures did you see? (including any relevant logs/error messages).

Best Regards,
Jeremias

TorizonCore builder was perfectly happy to build (build and images unpack output) and deploy the new 6.1.0 image - the problem was on the device itself.
After torizoncore-builder deploy ... (to the device running the build I was trying to customize) It would fail to boot 3 times and then the update would be rolled back.

I did not think to capture the output or investigate much more at the time as I was trying to get a task completed, but I connected to the serial terminal at one point and saw the resets were happening because it was enabling the 60 second watchdog and resetting the system - the output simply stopped shortly after the watchdog was enabled, which seemed like the system had just locked up.

If it is helpful I can try to reproduce and capture the serial terminal output tomorrow.

Well it would help to know your exact steps, including what kind of customization you were trying to deploy. In case I want to try and replicate your results on my own setup for further investigation.

Best Regards,
Jeremias

I reproduced it today and the steps are as follows:

  • I extracted the contents of December’s torizon-core-docker-verdin-imx8mp-TEZI-6.1.0-[blahblahh]-build.4.tar to a USB drive and installed it via TEZI.

  • sshed in and deal with the mandatory password change

  • Created a tcbuild.yaml specifying the same .tar I extracted above, with the only customization(s) being the image name/description, and the following DTS overlays.

     - device-trees/overlays/verdin-imx8mp_mezzanine-lvds-single-channel_overlay.dts
     - device-trees/overlays/verdin-imx8mp_mezzanine-touch-atmel-mxt_overlay.dts
     - my_dts/lvds-panel.dts
    

where my_dts/lvds-panel.dts is a carbon copy of the DTS for the toradex 10" display. (just copied in preparation for a different display. This DTS works just fine on the 5.6.x build)

  • Build image with torizoncore-builder build - success.

  • Unpack with images unpack output - success

  • deploy with deploy command to the just-flashed development board. (Completes successfully, reboots the device)

  • SoM reboots. Display never comes up, and SSH connection times out Device reboots three times and then rolls back the attempted update. The serial log from the console USB port looks like this - starting from the login prompt on the running 6.1.x delpoy.

verdin-imx8mp-07091509 login: 
[   41.925578] watchdog: watchdog0: watchdog did not stop!
[   42.172245] imx2-wdt 30280000.watchdog: Device shutdown: Expect reboot!
[   42.179469] reboot: Restarting system
␀␀
U-Boot SPL 2022.04-6.1.0-devel+git.944a4ccb70e4 (Jan 01 1970 - 00:00:00 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
Training FAILED
Quad die, dual rank failed, attempting dual die, single rank configuration.
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
SEC0:  RNG instantiated
Normal Boot
WDT:   Started watchdog@30280000 with servicing (60s timeout)
Trying to boot from BOOTROM
Boot Stage: Primary boot
Find img info 0x&4802d000, size 888
Download 916480, Total size 917264
NOTICE:  BL31: v2.6(release):lf_v2.6-g9202efc9aa
NOTICE:  BL31: Built : 00:00:00, Jan  1 1970


U-Boot 2022.04-6.1.0-devel+git.944a4ccb70e4 (Jan 01 1970 - 00:00:00 +0000)

CPU:   i.MX8MP[8] rev1.1 1600 MHz (running at 1200 MHz)
CPU:   Industrial temperature grade (-40C to 105C) at 40C
Reset cause: POR
DRAM:  4 GiB
Core:  89 devices, 23 uclasses, devicetree: separate
WDT:   Started watchdog@30280000 with servicing (60s timeout)
MMC:   FSL_SDHC: 1, FSL_SDHC: 2
Loading Environment from MMC... OK
In:    serial@30880000
Out:   serial@30880000
Err:   serial@30880000
Model: Toradex 0058 Verdin iMX8M Plus Quad 4GB WB IT V1.0E
Serial#: 07091509
Carrier: Toradex Verdin Development Board V1.1C, Serial# 10952597
Net:   eth1: ethernet@30be0000, eth0: ethernet@30bf0000 [PRIME]
Saving Environment to MMC... Writing to MMC(2)... OK
Normal Boot
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc2(part 0) is current device
Scanning mmc 2:1...
Found U-Boot script /boot.scr
973 bytes read in 0 ms
## Executing script at 50280000
5413 bytes read in 2 ms (2.6 MiB/s)
88435 bytes read in 4 ms (21.1 MiB/s)
139 bytes read in 2 ms (67.4 KiB/s)
Applying Overlay: verdin-imx8mp_mezzanine-lvds-single-channel_overlay.dtbo
1424 bytes read in 3 ms (462.9 KiB/s)
Applying Overlay: verdin-imx8mp_mezzanine-touch-atmel-mxt_overlay.dtbo
288 bytes read in 2 ms (140.6 KiB/s)
Applying Overlay: lvds-panel.dtbo
666 bytes read in 2 ms (325.2 KiB/s)
13376273 bytes read in 42 ms (303.7 MiB/s)
10536287 bytes read in 34 ms (295.5 MiB/s)
   Uncompressing Kernel Image
## Flattened Device Tree blob at 50200000
   Booting using the fdt blob at 0x50200000
   Loading Device Tree to 00000000ffac8000, end 00000000ffb00fff ... OK

Starting kernel ...

[    0.925180] caam_jr 30901000.jr: failed to flush job ring 0
[    1.008797] imx6q-pcie 33800000.pcie: Failed to get CLKREQ reset control
Starting version 250.5+
␀␀
U-Boot SPL 2022.04-6.1.0-devel+git.944a4ccb70e4 (Jan 01 1970 - 00:00:00 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
Training FAILED
Quad die, dual rank failed, attempting dual die, single rank configuration.
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
SEC0:  RNG instantiated
Normal Boot
WDT:   Started watchdog@30280000 with servicing (60s timeout)
Trying to boot from BOOTROM
Boot Stage: Primary boot
Find img info 0x&4802d000, size 888
Download 916480, Total size 917264
NOTICE:  BL31: v2.6(release):lf_v2.6-g9202efc9aa
NOTICE:  BL31: Built : 00:00:00, Jan  1 1970


U-Boot 2022.04-6.1.0-devel+git.944a4ccb70e4 (Jan 01 1970 - 00:00:00 +0000)

CPU:   i.MX8MP[8] rev1.1 1600 MHz (running at 1200 MHz)
CPU:   Industrial temperature grade (-40C to 105C) at 40C
Reset cause: POR
DRAM:  4 GiB
Core:  89 devices, 23 uclasses, devicetree: separate
WDT:   Started watchdog@30280000 with servicing (60s timeout)
MMC:   FSL_SDHC: 1, FSL_SDHC: 2
Loading Environment from MMC... OK
In:    serial@30880000
Out:   serial@30880000
Err:   serial@30880000
Model: Toradex 0058 Verdin iMX8M Plus Quad 4GB WB IT V1.0E
Serial#: 07091509
Carrier: Toradex Verdin Development Board V1.1C, Serial# 10952597
Net:   eth1: ethernet@30be0000, eth0: ethernet@30bf0000 [PRIME]
Saving Environment to MMC... Writing to MMC(2)... OK
Normal Boot
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc2(part 0) is current device
Scanning mmc 2:1...
Found U-Boot script /boot.scr
973 bytes read in 1 ms (950.2 KiB/s)
## Executing script at 50280000
5413 bytes read in 2 ms (2.6 MiB/s)
88435 bytes read in 3 ms (28.1 MiB/s)
139 bytes read in 2 ms (67.4 KiB/s)
Applying Overlay: verdin-imx8mp_mezzanine-lvds-single-channel_overlay.dtbo
1424 bytes read in 2 ms (695.3 KiB/s)
Applying Overlay: verdin-imx8mp_mezzanine-touch-atmel-mxt_overlay.dtbo
288 bytes read in 2 ms (140.6 KiB/s)
Applying Overlay: lvds-panel.dtbo
666 bytes read in 3 ms (216.8 KiB/s)
13376273 bytes read in 43 ms (296.7 MiB/s)
10536287 bytes read in 33 ms (304.5 MiB/s)
   Uncompressing Kernel Image
## Flattened Device Tree blob at 50200000
   Booting using the fdt blob at 0x50200000
   Loading Device Tree to 00000000ffac8000, end 00000000ffb00fff ... OK

Starting kernel ...

[    0.923127] caam_jr 30901000.jr: failed to flush job ring 0
[    1.006276] imx6q-pcie 33800000.pcie: Failed to get CLKREQ reset control
Starting version 250.5+
␀␀
U-Boot SPL 2022.04-6.1.0-devel+git.944a4ccb70e4 (Jan 01 1970 - 00:00:00 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
Training FAILED
Quad die, dual rank failed, attempting dual die, single rank configuration.
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
SEC0:  RNG instantiated
Normal Boot
WDT:   Started watchdog@30280000 with servicing (60s timeout)
Trying to boot from BOOTROM
Boot Stage: Primary boot
Find img info 0x&4802d000, size 888
Download 916480, Total size 917264
NOTICE:  BL31: v2.6(release):lf_v2.6-g9202efc9aa
NOTICE:  BL31: Built : 00:00:00, Jan  1 1970


U-Boot 2022.04-6.1.0-devel+git.944a4ccb70e4 (Jan 01 1970 - 00:00:00 +0000)

CPU:   i.MX8MP[8] rev1.1 1600 MHz (running at 1200 MHz)
CPU:   Industrial temperature grade (-40C to 105C) at 40C
Reset cause: POR
DRAM:  4 GiB
Core:  89 devices, 23 uclasses, devicetree: separate
WDT:   Started watchdog@30280000 with servicing (60s timeout)
MMC:   FSL_SDHC: 1, FSL_SDHC: 2
Loading Environment from MMC... OK
In:    serial@30880000
Out:   serial@30880000
Err:   serial@30880000
Model: Toradex 0058 Verdin iMX8M Plus Quad 4GB WB IT V1.0E
Serial#: 07091509
Carrier: Toradex Verdin Development Board V1.1C, Serial# 10952597
Net:   eth1: ethernet@30be0000, eth0: ethernet@30bf0000 [PRIME]
Saving Environment to MMC... Writing to MMC(2)... OK
Normal Boot
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc2(part 0) is current device
Scanning mmc 2:1...
Found U-Boot script /boot.scr
973 bytes read in 0 ms
## Executing script at 50280000
5413 bytes read in 2 ms (2.6 MiB/s)
88435 bytes read in 4 ms (21.1 MiB/s)
139 bytes read in 2 ms (67.4 KiB/s)
Applying Overlay: verdin-imx8mp_mezzanine-lvds-single-channel_overlay.dtbo
1424 bytes read in 3 ms (462.9 KiB/s)
Applying Overlay: verdin-imx8mp_mezzanine-touch-atmel-mxt_overlay.dtbo
288 bytes read in 2 ms (140.6 KiB/s)
Applying Overlay: lvds-panel.dtbo
666 bytes read in 2 ms (325.2 KiB/s)
13376273 bytes read in 42 ms (303.7 MiB/s)
10536287 bytes read in 34 ms (295.5 MiB/s)
   Uncompressing Kernel Image
## Flattened Device Tree blob at 50200000
   Booting using the fdt blob at 0x50200000
   Loading Device Tree to 00000000ffac8000, end 00000000ffb00fff ... OK

Starting kernel ...

[    0.924067] caam_jr 30901000.jr: failed to flush job ring 0
[    1.010241] imx6q-pcie 33800000.pcie: Failed to get CLKREQ reset control
Starting version 250.5+
␀␀
U-Boot SPL 2022.04-6.1.0-devel+git.944a4ccb70e4 (Jan 01 1970 - 00:00:00 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
Training FAILED
Quad die, dual rank failed, attempting dual die, single rank configuration.
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
SEC0:  RNG instantiated
Normal Boot
WDT:   Started watchdog@30280000 with servicing (60s timeout)
Trying to boot from BOOTROM
Boot Stage: Primary boot
Find img info 0x&4802d000, size 888
Download 916480, Total size 917264
NOTICE:  BL31: v2.6(release):lf_v2.6-g9202efc9aa
NOTICE:  BL31: Built : 00:00:00, Jan  1 1970


U-Boot 2022.04-6.1.0-devel+git.944a4ccb70e4 (Jan 01 1970 - 00:00:00 +0000)

CPU:   i.MX8MP[8] rev1.1 1600 MHz (running at 1200 MHz)
CPU:   Industrial temperature grade (-40C to 105C) at 40C
Reset cause: POR
DRAM:  4 GiB
Core:  89 devices, 23 uclasses, devicetree: separate
WDT:   Started watchdog@30280000 with servicing (60s timeout)
MMC:   FSL_SDHC: 1, FSL_SDHC: 2
Loading Environment from MMC... OK
In:    serial@30880000
Out:   serial@30880000
Err:   serial@30880000
Model: Toradex 0058 Verdin iMX8M Plus Quad 4GB WB IT V1.0E
Serial#: 07091509
Carrier: Toradex Verdin Development Board V1.1C, Serial# 10952597
Net:   eth1: ethernet@30be0000, eth0: ethernet@30bf0000 [PRIME]
Saving Environment to MMC... Writing to MMC(2)... OK
Normal Boot
Warning: Bootlimit (3) exceeded. Using altbootcmd.
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc2(part 0) is current device
Scanning mmc 2:1...
Found U-Boot script /boot.scr
973 bytes read in 0 ms
## Executing script at 50280000
Saving Environment to MMC... Writing to MMC(2)... OK
5413 bytes read in 2 ms (2.6 MiB/s)
Rollback enabled. Booting previously deployed version.
90443 bytes read in 3 ms (28.8 MiB/s)
118 bytes read in 2 ms (57.6 KiB/s)
Applying Overlay: verdin-imx8mp_hdmi_overlay.dtbo
2219 bytes read in 2 ms (1.1 MiB/s)
Applying Overlay: verdin-imx8mp_dsi-to-hdmi_overlay.dtbo
3165 bytes read in 4 ms (772.5 KiB/s)
Applying Overlay: verdin-imx8mp_spidev_overlay.dtbo
561 bytes read in 3 ms (182.6 KiB/s)
13376273 bytes read in 44 ms (289.9 MiB/s)
10536287 bytes read in 37 ms (271.6 MiB/s)
   Uncompressing Kernel Image
## Flattened Device Tree blob at 50200000
   Booting using the fdt blob at 0x50200000
   Loading Device Tree to 00000000ffac7000, end 00000000ffb00fff ... OK

Starting kernel ...

Starting version 250.5+
[    2.869855] systemd[1]: Current system time is further ahead 15y after build time, but cannot correct: Invalid argument
[    5.306089] fsl-aud2htx 30cb0000.aud2htx: failed to pcm register
[   17.037744] The last update failed and you are in a rollback state.
[   17.044372] Unfortunately the logs are unavailable, probably because the update failed before starting userspace applications.

TorizonCore 6.1.0-devel-202212+build.4 verdin-imx8mp-07091509 ttymxc2

I think I see what happened now. The overlays you applied, these overlays are made for 5.X.Y TorizonCore. These are not necessarily compatible with 6.X.Y TorizonCore. Between 5.X.Y and 6.X.Y the Linux kernel version is different and the underlying device trees themselves are different so overlays from one version are incompatible with the other. In fact we are still migrating all the overlays to have 6.X.Y versions of these.

That said what happened here is you applied some in-compatible overlays that cause the system to not be able to boot properly. This triggers our automated rollback as a safety measure, since the system can’t boot and it reverts to it’s previous “good” state.

Best Regards,
Jeremias

Ah, ok. That makes sense.

Is there a rough ETA on when new compatible overlays will be available?

Unfortunately I don’t have a rough ETA I can provide you at this time. The upcoming quarterly release is being finalized but it’s not guaranteed the new overlays will be ready to use at this time as well.

I’ll see if I can get anything more concrete for you.

Best Regards,
Jeremias