We are using Colibri iMX6S version 1.1A as hardware und Qt for device creation 5.14.2 with Linux kernel 4.14.159. Currently about 100 installations are in use worldwide.
Recently two customers return their boards because the boards were not bootable. The dmesg output shows that both boards run into the same error situation:
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.14.159-00041-g1f43bce17a57-dirty (colibri@schall-linux-18) (gcc version 8.2.1 20180802 (GNU Toolchain for the A-profile Architecture 8.2-2019.01 (arm-rel-8.28))) #33 SMP Mon Mar 8 17:07:58 CET 2021
[ 0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] OF: fdt: Machine model: Toradex Colibri iMX6DL/S on Colibri Evaluation Board V3
[ 0.000000] Memory policy: Data cache writeback
[ 0.000000] OF: reserved mem: failed to allocate memory for node 'linux,cma'
[ 0.000000] cma: Reserved 128 MiB at 0x18000000
[ 0.000000] CPU: All CPU(s) started in SVC mode.
[ 0.000000] percpu: Embedded 15 pages/cpu s30796 r8192 d22452 u61440
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 65024
[ 0.000000] Kernel command line: enable_wait_mode=off galcore.contiguousSize=50331648 ip=off root=/dev/mmcblk0p2 rw,noatime rootfstype=ext3 rootwait fec_mac=00:14:2d:a3:36:a1 no_console_suspend=1 console=ttymxc0,115200n8 video=mxcfb0:dev=lcd,1280x800M@60,if=RGB24 video=mxcfb1:off fbmem=32M
[ 0.000000] PID hash table entries: 1024 (order: 0, 4096 bytes)
[ 0.000000] Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
[ 0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
[ 0.000000] Memory: 116028K/262144K available (8192K kernel code, 494K rwdata, 2040K rodata, 1024K init, 411K bss, 15044K reserved, 131072K cma-reserved, 0K highmem)
[ 0.000000] Virtual kernel memory layout:
[ 0.000000] vector : 0xffff0000 - 0xffff1000 ( 4 kB)
[ 0.000000] fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
[ 0.000000] vmalloc : 0x90800000 - 0xff800000 (1776 MB)
[ 0.000000] lowmem : 0x80000000 - 0x90000000 ( 256 MB)
[ 0.000000] pkmap : 0x7fe00000 - 0x80000000 ( 2 MB)
[ 0.000000] modules : 0x7f000000 - 0x7fe00000 ( 14 MB)
[ 0.000000] .text : 0x80008000 - 0x80900000 (9184 kB)
[ 0.000000] .init : 0x80b00000 - 0x80c00000 (1024 kB)
[ 0.000000] .data : 0x80c00000 - 0x80c7bb80 ( 495 kB)
[ 0.000000] .bss : 0x80c7bb80 - 0x80ce2838 ( 412 kB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] RCU event tracing is enabled.
[ 0.000000] RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
[ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[ 0.000000] L2C: DT/platform modifies aux control register: 0x32050000 -> 0x32450000
[ 0.000000] L2C-310 erratum 769419 enabled
[ 0.000000] L2C-310 enabling early BRESP for Cortex-A9
[ 0.000000] L2C-310 full line of zeros enabled for Cortex-A9
[ 0.000000] L2C-310 ID prefetch enabled, offset 16 lines
[ 0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
[ 0.000000] L2C-310 cache controller enabled, 16 ways, 512 kB
[ 0.000000] L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x76450001
[ 0.000000] Switching to timer-based delay loop, resolution 333ns
[ 0.000007] sched_clock: 32 bits at 3000kHz, resolution 333ns, wraps every 715827882841ns
[ 0.000024] clocksource: mxc_timer1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 637086815595 ns
[ 0.001606] Console: colour dummy device 80x30
[ 0.001631] Calibrating delay loop (skipped), value calculated using timer frequency.. 6.00 BogoMIPS (lpj=30000)
[ 0.001647] pid_max: default: 32768 minimum: 301
[ 0.001779] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[ 0.001797] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[ 0.002380] CPU: Testing write buffer coherency: ok
[ 0.002422] CPU0: Spectre v2: using BPIALL workaround
[ 0.002768] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.003202] Setting up static identity map for 0x10100000 - 0x10100060
[ 0.003345] Hierarchical SRCU implementation.
[ 0.003935] smp: Bringing up secondary CPUs ...
[ 0.004461] smp: Brought up 1 node, 1 CPU
[ 0.004473] SMP: Total of 1 processors activated (6.00 BogoMIPS).
[ 0.004484] CPU: All CPU(s) started in SVC mode.
[ 0.005291] devtmpfs: initialized
[ 0.014602] random: get_random_u32 called from bucket_table_alloc+0x108/0x238 with crng_init=0
[ 0.015121] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[ 0.015377] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.015398] futex hash table entries: 512 (order: 3, 32768 bytes)
[ 0.020137] pinctrl core: initialized pinctrl subsystem
[ 0.021035] NET: Registered protocol family 16
[ 0.028107] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.028873] cpuidle: using governor menu
[ 0.029191] CPU identified as i.MX6DL, silicon rev 1.4
[ 0.036140] vdd1p1: supplied by regulator-dummy
[ 0.036646] vdd3p0: supplied by regulator-dummy
[ 0.037081] vdd2p5: supplied by regulator-dummy
[ 0.037512] vddarm: supplied by regulator-dummy
[ 0.038006] vddpu: supplied by regulator-dummy
[ 0.038449] vddsoc: supplied by regulator-dummy
[ 0.049197] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[ 0.049210] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.050092] imx6dl-pinctrl 20e0000.iomuxc: initialized IMX pinctrl driver
[ 0.075800] SCSI subsystem initialized
[ 0.076007] usbcore: registered new interface driver usbfs
[ 0.076058] usbcore: registered new interface driver hub
[ 0.076175] usbcore: registered new device driver usb
[ 0.076363] usb_phy_generic 2000000.aips-bus:usbphy_nop1: 2000000.aips-bus:usbphy_nop1 supply vcc not found, using dummy regulator
[ 0.076524] usb_phy_generic 2000000.aips-bus:usbphy_nop2: 2000000.aips-bus:usbphy_nop2 supply vcc not found, using dummy regulator
[ 0.077353] i2c-gpio i2c@0: using pins 109 (SDA) and 108 (SCL)
[ 0.078272] i2c i2c-0: IMX I2C adapter registered
[ 0.078290] i2c i2c-0: can't use DMA, using PIO instead.
[ 0.079010] i2c i2c-1: IMX I2C adapter registered
[ 0.079027] i2c i2c-1: can't use DMA, using PIO instead.
[ 0.079229] Linux video capture interface: v2.00
[ 0.079287] pps_core: LinuxPPS API ver. 1 registered
[ 0.079295] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 0.079317] PTP clock support registered
[ 0.109644] imx-ipuv3 2400000.ipu: IPU DMFC NORMAL mode: 1(0~1), 5B(4,5), 5F(6,7)
[ 0.110959] mxc_vdoa 21e4000.vdoa: i.MX Video Data Order Adapter(VDOA) driver probed
[ 0.111249] MIPI CSI2 driver module loaded
[ 0.111399] Advanced Linux Sound Architecture Driver Initialized.
[ 0.113266] clocksource: Switched to clocksource mxc_timer1
[ 0.113371] VFS: Disk quotas dquot_6.6.0
[ 0.113451] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[ 0.121594] NET: Registered protocol family 2
[ 0.122179] TCP established hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.122214] TCP bind hash table entries: 2048 (order: 2, 16384 bytes)
[ 0.122253] TCP: Hash tables configured (established 2048 bind 2048)
[ 0.122360] UDP hash table entries: 256 (order: 1, 8192 bytes)
[ 0.122386] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[ 0.122535] NET: Registered protocol family 1
[ 0.122911] RPC: Registered named UNIX socket transport module.
[ 0.122922] RPC: Registered udp transport module.
[ 0.122930] RPC: Registered tcp transport module.
[ 0.122938] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 0.123732] hw perfevents: no interrupt-affinity property for /soc/pmu, guessing.
[ 0.123905] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[ 0.125446] Bus freq driver module loaded
[ 0.126640] workingset: timestamp_bits=30 max_order=16 bucket_order=0
[ 0.131033] NFS: Registering the id_resolver key type
[ 0.131061] Key type id_resolver registered
[ 0.131069] Key type id_legacy registered
[ 0.131086] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 0.131590] ntfs: driver 2.1.32 [Flags: R/W].
[ 0.131873] fuse init (API version 7.26)
[ 0.139971] io scheduler noop registered
[ 0.139985] io scheduler deadline registered
[ 0.140144] io scheduler cfq registered (default)
[ 0.140155] io scheduler mq-deadline registered
[ 0.140164] io scheduler kyber registered
[ 0.142145] imx-weim 21b8000.weim: Driver registered.
[ 0.143921] pwm-backlight backlight: backlight supply power not found, using dummy regulator
[ 0.146607] fbcvt: 1280x800@60: CVT Name - 1.024MA
[ 0.146634] mxc_sdc_fb fb@0: registered mxc display driver lcd
[ 0.155467] mxc_sdc_fb fb@0: 1280x800 h_sync,r,l: 128,72,200 v_sync,l,u: 6,3,22 pixclock=83500000 Hz
[ 0.183358] imx-ipuv3 2400000.ipu: IPU DMFC DP HIGH RESOLUTION: 1(0,1), 5B(2~5), 5F(6,7)
[ 0.217364] mxc_sdc_fb fb@0: 1280x800 h_sync,r,l: 128,72,200 v_sync,l,u: 6,3,22 pixclock=83500000 Hz
[ 0.254386] Console: switching to colour frame buffer device 160x50
[ 0.288947] mxc_sdc_fb fb@1: mxcfb1 is turned off!
[ 0.291080] imx-pgc-pd imx-pgc-power-domain.0: Linked as a consumer to 20dc000.gpc
[ 0.291620] PU : no governor for states
[ 0.291641] imx-pgc-pd imx-pgc-power-domain.1: Linked as a consumer to 20dc000.gpc
[ 0.293314] pfuze100-regulator 0-0008: Full layer: 2, Metal layer: 1
[ 0.294058] pfuze100-regulator 0-0008: FAB: 0, FIN: 0
[ 0.294071] pfuze100-regulator 0-0008: pfuze100 found.
[ 0.315759] 2020000.serial: ttymxc0 at MMIO 0x2020000 (irq = 24, base_baud = 5000000) is a IMX
[ 1.187278] console [ttymxc0] enabled
[ 1.191997] 21e8000.serial: ttymxc1 at MMIO 0x21e8000 (irq = 68, base_baud = 5000000) is a IMX
[ 1.201299] 21ec000.serial: ttymxc2 at MMIO 0x21ec000 (irq = 69, base_baud = 5000000) is a IMX
[ 1.210468] 21f0000.serial: ttymxc3 at MMIO 0x21f0000 (irq = 70, base_baud = 5000000) is a IMX
[ 1.219632] 21f4000.serial: ttymxc4 at MMIO 0x21f4000 (irq = 71, base_baud = 5000000) is a IMX
[ 1.229769] [drm] Initialized vivante 1.0.0 20120216 for platform:Vivante GCCore on minor 0
[ 1.238196] [drm] Initialized vivante 1.0.0 20120216 on minor 0
[ 1.252277] brd: module loaded
[ 1.264648] loop: module loaded
[ 1.269480] libphy: Fixed MDIO Bus: probed
[ 1.274028] vcan: Virtual CAN interface driver
[ 1.278482] CAN device driver interface
[ 1.283833] fec 2188000.ethernet: 2188000.ethernet supply phy not found, using dummy regulator
[ 1.292944] pps pps0: new PPS source ptp0
[ 1.298116] libphy: fec_enet_mii_bus: probed
[ 1.303528] fec 2188000.ethernet eth0: registered PHC device 0
[ 1.309804] usbcore: registered new interface driver asix
[ 1.315333] usbcore: registered new interface driver ax88179_178a
[ 1.321466] usbcore: registered new interface driver cdc_ether
[ 1.327410] usbcore: registered new interface driver net1080
[ 1.333108] usbcore: registered new interface driver cdc_subset
[ 1.339111] usbcore: registered new interface driver zaurus
[ 1.344783] usbcore: registered new interface driver cdc_ncm
[ 1.350448] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 1.356998] ehci-mxc: Freescale On-Chip EHCI Host driver
[ 1.362489] usbcore: registered new interface driver usb-storage
[ 1.368623] usbcore: registered new interface driver usbserial
[ 1.374517] usbcore: registered new interface driver usbserial_generic
[ 1.381074] usbserial: USB Serial support registered for generic
[ 1.387133] usbcore: registered new interface driver ftdi_sio
[ 1.392906] usbserial: USB Serial support registered for FTDI USB Serial Device
[ 1.400267] usbcore: registered new interface driver pl2303
[ 1.405894] usbserial: USB Serial support registered for pl2303
[ 1.412977] usbmisc_imx 2184800.usbmisc: 2184800.usbmisc supply vbus-wakeup not found, using dummy regulator
[ 1.423904] imx_usb 2184000.usb: 2184000.usb supply vbus not found, using dummy regulator
[ 1.435894] ci_hdrc ci_hdrc.0: EHCI Host Controller
[ 1.440806] ci_hdrc ci_hdrc.0: new USB bus registered, assigned bus number 1
[ 1.473289] ci_hdrc ci_hdrc.0: USB 2.0 started, EHCI 1.00
[ 1.479477] hub 1-0:1.0: USB hub found
[ 1.483328] hub 1-0:1.0: 1 port detected
[ 1.605543] Goodix-TS 1-005d: ID 928, version: 1060
[ 1.626038] random: fast init done
[ 1.636458] input: Goodix Capacitive TouchScreen as /devices/soc0/soc/2100000.aips-bus/21a8000.i2c/i2c-1/1-005d/input/input0
[ 1.651984] rtc-ds1307 1-0068: registered as rtc0
[ 1.657457] snvs_rtc 20cc000.snvs:snvs-rtc-lp: registered as rtc1
[ 1.663795] i2c /dev entries driver
[ 1.667983] IR NEC protocol handler initialized
[ 1.672522] IR RC5(x/sz) protocol handler initialized
[ 1.677644] IR RC6 protocol handler initialized
[ 1.682180] IR JVC protocol handler initialized
[ 1.686733] IR Sony protocol handler initialized
[ 1.691354] IR SANYO protocol handler initialized
[ 1.696076] IR Sharp protocol handler initialized
[ 1.700783] IR MCE Keyboard/mouse protocol handler initialized
[ 1.706631] IR XMP protocol handler initialized
[ 1.711731] mxc_v4l2_output v4l2_out: V4L2 device registered as video16
[ 1.718517] mxc_v4l2_output v4l2_out: V4L2 device registered as video17
[ 1.726649] imx2-wdt 20bc000.wdog: timeout 60 sec (nowayout=0)
[ 1.733111] sdhci: Secure Digital Host Controller Interface driver
[ 1.739337] sdhci: Copyright(c) Pierre Ossman
[ 1.743714] sdhci-pltfm: SDHCI platform and OF driver helper
[ 1.750397] sdhci-esdhc-imx 2190000.usdhc: Got CD GPIO
[ 1.814507] mmc1: SDHCI controller on 2190000.usdhc [2190000.usdhc] using ADMA
[ 1.883335] mmc0: SDHCI controller on 2198000.usdhc [2198000.usdhc] using ADMA
[ 1.897340] mxc_vpu 2040000.vpu_fsl: VPU initialized
[ 1.903792] galcore: clk_get vg clock failed, disable vg!
[ 1.911157] Galcore version 6.2.4.190076
[ 2.127546] caam 2100000.caam: ERA source: CAAMVID.
[ 2.132488] caam 2100000.caam: AXI pipeline throttling enabled.
[ 2.139618] caam 2100000.caam: device ID = 0x0a16010000000100 (Era 4)
[ 2.146111] caam 2100000.caam: job rings = 2, qi = 0, dpaa2 = no
[ 2.156261] caam_jr 2101000.jr0: Entropy delay = 3200
[ 2.178885] mmc0: new DDR MMC card at address 0001
[ 2.184824] mmcblk0: mmc0:0001 Q2J54A 3.64 GiB
[ 2.190456] mmcblk0boot0: mmc0:0001 Q2J54A partition 1 2.00 MiB
[ 2.197645] mmcblk0boot1: mmc0:0001 Q2J54A partition 2 2.00 MiB
[ 2.205086] mmcblk0: p1 p2
[ 2.222152] caam_jr 2101000.jr0: Instantiated RNG4 SH0.
[ 2.282907] caam_jr 2101000.jr0: Instantiated RNG4 SH1.
[ 2.295595] caam algorithms registered in /proc/crypto
[ 2.304818] caam_jr 2101000.jr0: registering rng-caam
[ 2.310965] caam-snvs 20cc000.caam-snvs: can't get snvs clock
[ 2.317161] caam-snvs 20cc000.caam-snvs: violation handlers armed - non-secure state
[ 2.325253] hidraw: raw HID events driver (C) Jiri Kosina
[ 2.330787] usbcore: registered new interface driver usbhid
[ 2.336425] usbhid: USB HID core driver
[ 2.343825] sgtl5000 0-000a: sgtl5000 revision 0x11
[ 2.378177] fsl-ssi-dai 2028000.ssi: No cache defaults, reading back from HW
[ 2.399910] imx-sgtl5000 sound: sgtl5000 <-> 2028000.ssi mapping ok
[ 2.407481] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 2.413412] NET: Registered protocol family 10
[ 2.418831] Segment Routing with IPv6
[ 2.422586] ip6_tables: (C) 2000-2006 Netfilter Core Team
[ 2.428106] NET: Registered protocol family 17
[ 2.432567] can: controller area network core (rev 20170425 abi 9)
[ 2.438850] NET: Registered protocol family 29
[ 2.443327] can: raw protocol (rev 20170425)
[ 2.447603] can: broadcast manager protocol (rev 20170425 t)
[ 2.453287] can: netlink gateway (rev 20170425) max_hops=1
[ 2.459106] Key type dns_resolver registered
[ 2.465142] Registering SWP/SWPB emulation handler
[ 2.479439] imx_usb 2184000.usb: 2184000.usb supply vbus not found, using dummy regulator
[ 2.491880] ci_hdrc ci_hdrc.1: EHCI Host Controller
[ 2.496872] ci_hdrc ci_hdrc.1: new USB bus registered, assigned bus number 2
[ 2.533302] ci_hdrc ci_hdrc.1: USB 2.0 started, EHCI 1.00
[ 2.539322] hub 2-0:1.0: USB hub found
[ 2.543109] hub 2-0:1.0: 1 port detected
[ 2.548217] (NULL device *): hwmon_device_register() is deprecated. Please convert the driver to use hwmon_device_register_with_info().
[ 2.560658] imx_thermal 2000000.aips-bus:tempmon: Commercial CPU temperature grade - max:95C critical:95C passive:85C
[ 2.573787] rtc-ds1307 1-0068: setting system clock to 2022-03-04 09:40:50 UTC (1646386850)
[ 2.582573] ALSA device list:
[ 2.585583] #0: imx6dl-colibri-sgtl5000
[ 2.597900] EXT4-fs (mmcblk0p2): mounting ext3 file system using the ext4 subsystem
[ 2.606279] EXT4-fs (mmcblk0p2): INFO: recovery required on readonly filesystem
[ 2.613639] EXT4-fs (mmcblk0p2): write access will be enabled during recovery
[ 2.706381] EXT4-fs (mmcblk0p2): recovery complete
[ 8.134971] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x400e00
[ 8.427015] random: crng init done
After booting linux from a SD card the eMMC root partition of one the boards mounts without any problems making the board bootable too without the need to run fsck file check. So it seems not to be a problem of the ext4 file system but of the eMMC itsself.
Diving into the Linux kernel I see that this error is emitted by the function mmc_blk_cmd_recovery() in file drivers/mmc/core/block.c. This function queries eMMC to get the state of eMMC, recognises an error but classifies the state as continuable to the calling function. The calling function makes some further checks and apparently stops excecution. As the eMMC state is not reset now booting loops.
The last two years we hit on the problem ourselves (2 or 3 times). I’m very sure that the problem arises if the device is switched off shortly after switching it on.
Any idea how to solve the problem.