eMMC corruption?

We have received a RMA from one of our customers that his device is not booting any more. It seems like the flash is corrupted or at least cannot be mounted. Our kernel includes an initramfs which mounts all partitions to do some magic prior switching to the real rootfs. Please find the output below. There are a lot of “mmcblk0” errors which might point to the error itself. Thanks in advance.

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.9.144-rt94-2.8.5+g5758a8e (oe-user@oe-host) (gcc version 7.3.0 (GCC) ) #4 SMP PREEMPT RT Tue Nov 5 10:42:17 UTC 2019
[    0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt:Machine model: Konrad Technologies FXC-200
[    0.000000] Reserved memory: created CMA memory pool at 0x18000000, size 320 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] percpu: Embedded 14 pages/cpu @9fb12000 s27488 r8192 d21664 u57344
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 130048
[    0.000000] Kernel command line: vmalloc=400M user_debug=30 ip=off root=/dev/mmcblk0p2 ro rootfstype=ext4 rootwait fec_mac=00:14:2d:a1:d4:7f consoleblank=0 no_console_suspend=1 console=tty1 console=ttymxc0,115200n8 mxc_hdmi.only_cea=0 video=mxcfb0:dev=hdmi,1920x1080M@60,if=RGB24 video=mxcfb1:off video=mxcfb2:off video=mxcfb3:off fbmem=32M isolcpus=1
[    0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Memory: 163392K/524288K available (8192K kernel code, 397K rwdata, 2072K rodata, 15360K init, 431K bss, 33216K reserved, 327680K cma-reserved, 0K highmem)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
[    0.000000]     vmalloc : 0xa0800000 - 0xff800000   (1520 MB)
[    0.000000]     lowmem  : 0x80000000 - 0xa0000000   ( 512 MB)
[    0.000000]     pkmap   : 0x7fe00000 - 0x80000000   (   2 MB)
[    0.000000]     modules : 0x7f000000 - 0x7fe00000   (  14 MB)
[    0.000000]       .text : 0x80008000 - 0x80900000   (9184 kB)
[    0.000000]       .init : 0x80c00000 - 0x81b00000   (15360 kB)
[    0.000000]       .data : 0x81b00000 - 0x81b637e0   ( 398 kB)
[    0.000000]        .bss : 0x81b637e0 - 0x81bcf644   ( 432 kB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000]  Build-time adjustment of leaf fanout to 32.
[    0.000000]  RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=32, nr_cpu_ids=2
[    0.000000] NR_IRQS:16 nr_irqs:16 16
[    0.000000] L2C-310 errata 752271 769419 enabled
[    0.000000] L2C-310 enabling early BRESP for Cortex-A9
[    0.000000] L2C-310 full line of zeros enabled for Cortex-A9
[    0.000000] L2C-310 ID prefetch enabled, offset 16 lines
[    0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
[    0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
[    0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x76470001
[    0.000000] Switching to timer-based delay loop, resolution 333ns
[    0.000007] sched_clock: 32 bits at 3000kHz, resolution 333ns, wraps every 715827882841ns
[    0.000018] clocksource: mxc_timer1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 637086815595 ns
[    0.001541] Console: colour dummy device 80x30
[    0.002439] console [tty1] enabled
[    0.002457] Calibrating delay loop (skipped), value calculated using timer frequency.. 6.00 BogoMIPS (lpj=30000)
[    0.002461] pid_max: default: 32768 minimum: 301
[    0.002555] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.002563] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.003224] CPU: Testing write buffer coherency: ok
[    0.003233] CPU0: Spectre v2: using BPIALL workaround
[    0.003531] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.003788] Setting up static identity map for 0x10100000 - 0x10100058
[    0.269855] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.269860] CPU1: Spectre v2: using BPIALL workaround
[    0.270062] Brought up 2 CPUs
[    0.270148] SMP: Total of 2 processors activated (12.00 BogoMIPS).
[    0.270179] CPU: All CPU(s) started in SVC mode.
[    0.271066] devtmpfs: initialized
[    0.290010] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[    0.290401] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.290467] futex hash table entries: 512 (order: 3, 32768 bytes)
[    0.307052] pinctrl core: initialized pinctrl subsystem
[    0.308476] NET: Registered protocol family 16
[    0.324317] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.329656] cpuidle: using governor menu
[    0.330077] CPU identified as i.MX6Q, silicon rev 1.5
[    0.361907] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[    0.361964] hw-breakpoint: maximum watchpoint size is 4 bytes.
[    0.362513] imx6q-pinctrl 20e0000.iomuxc: Invalid fsl,pins property in node /soc/aips-bus@02000000/iomuxc@020e0000/gpio-bl-on
[    0.364643] imx6q-pinctrl 20e0000.iomuxc: initialized IMX pinctrl driver
[    0.366674] imx-gpc 20dc000.gpc: no fsl,ldo-bypass found!
[    0.366847] imx-gpc 20dc000.gpc: Registered imx-gpc
[    0.438357] usb_host_vbus: supplied by usb_host_vbus_hub
[    0.439076] vgaarb: loaded
[    0.440052] SCSI subsystem initialized
[    0.449992] usbcore: registered new interface driver usbfs
[    0.450103] usbcore: registered new interface driver hub
[    0.450233] usbcore: registered new device driver usb
[    0.451106] i2c-gpio i2c@0: using pins 80 (SDA) and 62 (SCL)
[    0.452558] i2c i2c-1: IMX I2C adapter registered
[    0.452607] i2c i2c-1: can't use DMA, using PIO instead.
[    0.460368] stmpe-i2c 2-0041: stmpe811 detected, chip id: 0x811
[    0.550855] i2c i2c-2: IMX I2C adapter registered
[    0.550904] i2c i2c-2: can't use DMA, using PIO instead.
[    0.551854] i2c i2c-3: IMX I2C adapter registered
[    0.551901] i2c i2c-3: can't use DMA, using PIO instead.
[    0.552122] Linux video capture interface: v2.00
[    0.552201] pps_core: LinuxPPS API ver. 1 registered
[    0.552232] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.552295] PTP clock support registered
[    0.579655] imx-ipuv3 2400000.ipu: IPU DMFC NORMAL mode: 1(0~1), 5B(4,5), 5F(6,7)
[    0.609651] imx-ipuv3 2800000.ipu: IPU DMFC NORMAL mode: 1(0~1), 5B(4,5), 5F(6,7)
[    0.610801] mxc_mipi_csi2 21dc000.mipi_csi: i.MX MIPI CSI2 driver probed ipu0 csi1
[    0.610862] mxc_mipi_csi2 21dc000.mipi_csi: i.MX MIPI CSI2 dphy version is 0x3130302a
[    0.611026] MIPI CSI2 driver module loaded
[    0.611286] Advanced Linux Sound Architecture Driver Initialized.
[    0.620082] clocksource: Switched to clocksource mxc_timer1
[    0.620260] VFS: Disk quotas dquot_6.6.0
[    0.620366] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    0.633565] NET: Registered protocol family 2
[    0.634324] TCP established hash table entries: 4096 (order: 2, 16384 bytes)
[    0.634414] TCP bind hash table entries: 4096 (order: 4, 114688 bytes)
[    0.634660] TCP: Hash tables configured (established 4096 bind 4096)
[    0.634759] UDP hash table entries: 256 (order: 2, 16384 bytes)
[    0.634828] UDP-Lite hash table entries: 256 (order: 2, 16384 bytes)
[    0.635071] NET: Registered protocol family 1
[    0.655670] RPC: Registered named UNIX socket transport module.
[    0.655712] RPC: Registered udp transport module.
[    0.655740] RPC: Registered tcp transport module.
[    0.655768] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.924138] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[    1.926056] Bus freq driver module loaded
[    1.938374] workingset: timestamp_bits=30 max_order=17 bucket_order=0
[    1.957014] NFS: Registering the id_resolver key type
[    1.957079] Key type id_resolver registered
[    1.957109] Key type id_legacy registered
[    1.957151] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    1.977958] ntfs: driver 2.1.32 [Flags: R/W].
[    1.978406] fuse init (API version 7.26)
[    1.988616] io scheduler noop registered
[    1.988661] io scheduler deadline registered
[    1.988887] io scheduler cfq registered (default)
[    1.989952] imx-pcie 1ffc000.pcie: no reserved region node.
[    1.990159] 1ffc000.pcie supply epdev_on not found, using dummy regulator
[    1.990552] OF: PCI: host bridge /soc/pcie@0x01000000 ranges:
[    1.990618] OF: PCI:    IO 0x01f80000..0x01f8ffff -> 0x00000000
[    1.990660] OF: PCI:   MEM 0x01000000..0x01efffff -> 0x01000000
[    2.046609] imx-pcie 1ffc000.pcie: Link: Gen2 disabled
[    2.046653] imx-pcie 1ffc000.pcie: Link up, Gen1
[    2.047157] imx-pcie 1ffc000.pcie: PCI host bridge to bus 0000:00
[    2.047203] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.047240] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    2.047274] pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]
[    2.047858] PCI: bus0: Fast back to back transfers disabled
[    2.070349] PCI: bus1: Fast back to back transfers disabled
[    2.070607] pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff]
[    2.070653] pci 0000:00:00.0: BAR 8: assigned [mem 0x01100000-0x011fffff]
[    2.070692] pci 0000:00:00.0: BAR 6: assigned [mem 0x01200000-0x0120ffff pref]
[    2.070738] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.070778] pci 0000:01:00.0: BAR 0: assigned [mem 0x01100000-0x0111ffff]
[    2.070837] pci 0000:01:00.0: BAR 3: assigned [mem 0x01120000-0x01123fff]
[    2.070893] pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x101f]
[    2.070949] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    2.070985] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.071024] pci 0000:00:00.0:   bridge window [mem 0x01100000-0x011fffff]
[    2.071561] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.071605] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    2.075643] 20e0000.hdmi_video supply HDMI not found, using dummy regulator
[    2.077067] mxc_hdmi 20e0000.hdmi_video: Detected HDMI controller 0x13:0xa:0xa0:0xc1
[    2.077138] fbcvt: 1920x1080@60: CVT Name - 2.073M9
[    2.077456] mxc_sdc_fb fb@0: registered mxc display driver hdmi
[    2.092836] mxc_sdc_fb fb@0: 1920x1080 h_sync,r,l: 44,88,148  v_sync,l,u: 5,4,36 pixclock=148500000 Hz
[    2.120279] imx-ipuv3 2400000.ipu: IPU DMFC DP HIGH RESOLUTION: 1(0,1), 5B(2~5), 5F(6,7)
[    2.154199] mxc_sdc_fb fb@0: 1920x1080 h_sync,r,l: 44,88,148  v_sync,l,u: 5,4,36 pixclock=148500000 Hz
[    2.202252] Console: switching to colour frame buffer device 240x67
[    2.237056] mxc_sdc_fb fb@1: mxcfb1 is turned off!
[    2.237233] mxc_sdc_fb fb@2: mxcfb2 is turned off!
[    2.237402] mxc_sdc_fb fb@3: mxcfb3 is turned off!
[    2.240004] imx-sdma 20ec000.sdma: no iram assigned, using external mem
[    2.240749] imx-sdma 20ec000.sdma: loaded firmware 3.3
[    2.246621] pfuze100-regulator 2-0008: Full layer: 2, Metal layer: 1
[    2.247534] pfuze100-regulator 2-0008: FAB: 0, FIN: 0
[    2.247645] pfuze100-regulator 2-0008: pfuze100 found.
[    2.275250] 2020000.serial: ttymxc0 at MMIO 0x2020000 (irq = 29, base_baud = 5000000) is a IMX
[    3.284115] console [ttymxc0] enabled
[    3.289091] 21e8000.serial: ttymxc1 at MMIO 0x21e8000 (irq = 74, base_baud = 5000000) is a IMX
[    3.298703] 21f0000.serial: ttymxc3 at MMIO 0x21f0000 (irq = 75, base_baud = 5000000) is a IMX
[    3.308295] 21f4000.serial: ttymxc4 at MMIO 0x21f4000 (irq = 76, base_baud = 5000000) is a IMX
[    3.318251] [drm] Initialized
[    3.321822] [drm] Initialized vivante 1.0.0 20120216 on minor 0
[    3.502230] brd: module loaded
[    3.593438] loop: module loaded
[    3.598056] ahci-imx 2200000.sata: fsl,transmit-level-mV not specified, using 00000024
[    3.606135] ahci-imx 2200000.sata: fsl,transmit-boost-mdB not specified, using 00000480
[    3.614296] ahci-imx 2200000.sata: fsl,transmit-atten-16ths not specified, using 00002000
[    3.622630] ahci-imx 2200000.sata: fsl,receive-eq-mdB not specified, using 05000000
[    3.633896] ahci-imx 2200000.sata: SSS flag set, parallel bus scan disabled
[    3.641024] ahci-imx 2200000.sata: AHCI 0001.0300 32 slots 1 ports 3 Gbps 0x1 impl platform mode
[    3.649977] ahci-imx 2200000.sata: flags: ncq sntf stag pm led clo only pmp pio slum part ccc apst
[    3.670990] scsi host0: ahci-imx
[    3.674682] ata1: SATA max UDMA/133 mmio [mem 0x02200000-0x02203fff] port 0x100 irq 83
[    3.684748] spi_imx 2008000.ecspi: dma setup error -19, use pio
[    3.691737] spi_imx 2008000.ecspi: probed
[    3.696452] spi_imx 200c000.ecspi: dma setup error -19, use pio
[    3.703402] spi_imx 200c000.ecspi: probed
[    3.708366] libphy: Fixed MDIO Bus: probed
[    3.717623] vcan: Virtual CAN interface driver
[    3.726723] CAN device driver interface
[    3.735504] 2090000.flexcan supply xceiver not found, using dummy regulator
[    3.748268] flexcan 2090000.flexcan: device registered (reg_base=a0b08000, irq=34)
[    3.760900] 2094000.flexcan supply xceiver not found, using dummy regulator
[    3.773673] flexcan 2094000.flexcan: device registered (reg_base=a0b10000, irq=35)
[    3.787507] 2188000.ethernet supply phy not found, using dummy regulator
[    3.810726] pps pps0: new PPS source ptp0
[    3.821674] libphy: fec_enet_mii_bus: probed
[    3.849019] fec 2188000.ethernet eth0: registered PHC device 0
[    3.860306] e100: Intel(R) PRO/100 Network Driver, 3.5.24-k2-NAPI
[    3.871447] e100: Copyright(c) 1999-2006 Intel Corporation
[    3.882133] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[    3.894337] e1000: Copyright (c) 1999-2006 Intel Corporation.
[    3.905313] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    3.916253] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    3.927428] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[    3.939606] igb: Copyright (c) 2007-2014 Intel Corporation.
[    3.950551] igb 0000:01:00.0: enabling device (0140 -> 0142)
[    3.992050] pps pps1: new PPS source ptp1
[    4.001283] igb 0000:01:00.0: added PHC on eth1
[    4.010998] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
[    4.023038] igb 0000:01:00.0: eth1: (PCIe:2.5Gb/s:Width x1) 70:b3:d5:57:50:14
[    4.030130] ata1: SATA link down (SStatus 0 SControl 300)
[    4.030168] ahci-imx 2200000.sata: no device found, disabling link.
[    4.030174] ahci-imx 2200000.sata: pass ahci_imx..hotplug=1 to enable hotplug
[    4.069369] igb 0000:01:00.0: eth1: PBA No: FFFFFF-0FF
[    4.079540] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[    4.092386] usbcore: registered new interface driver asix
[    4.102890] usbcore: registered new interface driver ax88179_178a
[    4.113985] usbcore: registered new interface driver cdc_ether
[    4.124759] usbcore: registered new interface driver net1080
[    4.135318] usbcore: registered new interface driver cdc_subset
[    4.146151] usbcore: registered new interface driver zaurus
[    4.156601] usbcore: registered new interface driver cdc_ncm
[    4.167013] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    4.178300] ehci-pci: EHCI PCI platform driver
[    4.187587] ehci-mxc: Freescale On-Chip EHCI Host driver
[    4.197984] usbcore: registered new interface driver usb-storage
[    4.208899] usbcore: registered new interface driver usbserial
[    4.219471] usbcore: registered new interface driver usbserial_generic
[    4.230708] usbserial: USB Serial support registered for generic
[    4.241473] usbcore: registered new interface driver ftdi_sio
[    4.252025] usbserial: USB Serial support registered for FTDI USB Serial Device
[    4.264143] usbcore: registered new interface driver pl2303
[    4.274546] usbserial: USB Serial support registered for pl2303
[    4.286799] 2184800.usbmisc supply vbus-wakeup not found, using dummy regulator
[    4.321602] ci_hdrc ci_hdrc.1: EHCI Host Controller
[    4.331360] ci_hdrc ci_hdrc.1: new USB bus registered, assigned bus number 1
[    4.370135] ci_hdrc ci_hdrc.1: USB 2.0 started, EHCI 1.00
[    4.381431] hub 1-0:1.0: USB hub found
[    4.390000] hub 1-0:1.0: 1 port detected
[    4.400238] mousedev: PS/2 mouse device common for all mice
[    4.422404] input: stmpe-ts as /devices/soc0/soc/2100000.aips-bus/21a4000.i2c/i2c-2/2-0041/stmpe-ts/input/input0
[    4.443429] rtc-ds1307 1-0068: rtc core: registered m41t0 as rtc0
[    4.455376] snvs_rtc 20cc000.snvs:snvs-rtc-lp: rtc core: registered 20cc000.snvs:snvs-r as rtc1
[    4.469144] i2c /dev entries driver
[    4.479121] mxc_v4l2_output v4l2_out: V4L2 device registered as video16
[    4.490877] mxc_v4l2_output v4l2_out: V4L2 device registered as video17
[    4.504476] imx2-wdt 20bc000.wdog: timeout 60 sec (nowayout=0)
[    4.516137] sdhci: Secure Digital Host Controller Interface driver
[    4.527229] sdhci: Copyright(c) Pierre Ossman
[    4.536474] sdhci-pltfm: SDHCI platform and OF driver helper
[    4.548099] sdhci-esdhc-imx 2194000.usdhc: could not get ultra high speed state, work on normal mode
[    4.562367] sdhci-esdhc-imx 2194000.usdhc: Got CD GPIO
[    4.630326] mmc2: SDHCI controller on 2194000.usdhc [2194000.usdhc] using ADMA
[    4.643332] sdhci-esdhc-imx 2198000.usdhc: could not get ultra high speed state, work on normal mode
[    4.720203] mmc0: SDHCI controller on 2198000.usdhc [2198000.usdhc] using ADMA
[    4.750151] usb 1-1: new high-speed USB device number 2 using ci_hdrc
[    4.760133] mxc_vpu 2040000.vpu_fsl: VPU initialized
[    4.760997] mxc_vdoa 21e4000.vdoa: i.MX Video Data Order Adapter(VDOA) driver probed
[    4.762019] Galcore version 6.2.4.150331
[    4.773814] random: fast init done
[    4.822392] mmc0: new DDR MMC card at address 0001
[    4.850349] mmcblk0: mmc0:0001 Q2J54A 3.64 GiB
[    4.870343] mmcblk0boot0: mmc0:0001 Q2J54A partition 1 2.00 MiB
[    4.890340] mmcblk0boot1: mmc0:0001 Q2J54A partition 2 2.00 MiB
[    4.891888]  mmcblk0: p1 p2 p3
[    5.141583] hub 1-1:1.0: USB hub found
[    5.154413] hub 1-1:1.0: 4 ports detected
[    5.331375] caam 2100000.caam: ERA source: CAAMVID.
[    5.341226] caam 2100000.caam: AXI pipeline throttling enabled.
[    5.353184] caam 2100000.caam: device ID = 0x0a16010000000000 (Era 4)
[    5.353184] job rings = 2, qi = 0
[    5.375434] caam_jr 2101000.jr0: Entropy delay = 3200
[    5.385333] caam_jr 2101000.jr0: Instantiated RNG4 SH0.
[    5.446097] caam_jr 2101000.jr0: Instantiated RNG4 SH1.
[    5.466077] caam algorithms registered in /proc/crypto
[    5.480569] caam_jr 2101000.jr0: registering rng-caam
[    5.491689] caam-snvs 20cc000.caam-snvs: can't get snvs clock
[    5.502377] caam-snvs 20cc000.caam-snvs: violation handlers armed - non-secure state
[    5.515189] hidraw: raw HID events driver (C) Jiri Kosina
[    5.525564] usbcore: registered new interface driver usbhid
[    5.535918] usbhid: USB HID core driver
[    5.552085] stmpe-adc stmpe-adc: Initialized
[    5.564381] sgtl5000 2-000a: sgtl5000 revision 0x11
[    5.607368] fsl-ssi-dai 2028000.ssi: No cache defaults, reading back from HW
[    5.640896] imx-sgtl5000 sound: sgtl5000 <-> 2028000.ssi mapping ok
[    5.657399] imx-audio-hdmi sound-hdmi: hdmi-hifi.0 <-> soc:hdmi_audio@00120000 mapping ok
[    5.672359] NET: Registered protocol family 10
[    5.693095] NET: Registered protocol family 17
[    5.702606] can: controller area network core (rev 20120528 abi 9)
[    5.713965] NET: Registered protocol family 29
[    5.723497] can: raw protocol (rev 20120528)
[    5.732805] can: broadcast manager protocol (rev 20161123 t)
[    5.743497] can: netlink gateway (rev 20130117) max_hops=1
[    5.754430] Key type dns_resolver registered
[    5.766541] cpu cpu0: Registered imx6q-cpufreq
[    5.774933] Registering SWP/SWPB emulation handler
[    5.802662] imx_thermal 2000000.aips-bus:tempmon: Extended Commercial CPU temperature grade - max:105C critical:115C passive:95C
[    5.819832] input: gpio-keys as /devices/soc0/gpio-keys/input/input1
[    5.832394] rtc-ds1307 1-0068: setting system clock to 2020-07-27 06:27:07 UTC (1595831227)
[    5.845607] usb_otg_vbus: disabling
[    5.853307] ALSA device list:
[    5.860389]   #0: imx6q-apalis-sgtl5000
[    5.868219]   #1: imx-hdmi-soc
[    5.875216] Warning: unable to open an initial console.
[    5.900968] Freeing unused kernel memory: 15360K
[    5.934717] random: python3: uninitialized urandom read (24 bytes read)
### hit any key within 3 seconds to enter debug mode ###
[   10.011975] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[   15.913601] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x400e00
[   16.353420] random: crng init done
[   20.960164] INFO: task python3:241 blocked for more than 10 seconds.
[   20.993030]       Not tainted 4.9.144-rt94-2.8.5+g5758a8e #4
[   21.036283] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   21.078746] python3         D    0   241      1 0x00000000
[   21.153552] Backtrace:
[   21.160031] [<808c8654>] (__schedule) from [<808c8a50>] (schedule+0x58/0xf4)
[   21.235191]  r10:81bac2cc r9:8338fb9c r8:808c94f8 r7:7fffffff r6:00000000 r5:ffffe000
[   21.277914]  r4:833d2080
[   21.317422] [<808c89f8>] (schedule) from [<808caeac>] (schedule_timeout+0x1b0/0x2d4)
[   21.359760]  r5:9fb18440 r4:7fffffff
[   21.398213] [<808cacfc>] (schedule_timeout) from [<808c861c>] (io_schedule_timeout+0x7c/0xb4)
[   21.474959]  r10:81bac2cc r9:8338fb9c r8:808c94f8 r7:7fffffff r6:00000000 r5:9fb18440
[   21.517768]  r4:9fb17f80
[   21.557771] [<808c85a0>] (io_schedule_timeout) from [<808c9510>] (bit_wait_io+0x18/0x64)
[   21.634085]  r7:81b024a0 r6:8338fb24 r5:00000002 r4:00000002
[   21.677193] [<808c94f8>] (bit_wait_io) from [<808c9154>] (__wait_on_bit+0x68/0xc0)
[   21.719816]  r5:00000002 r4:8338fb18
[   21.758520] [<808c90ec>] (__wait_on_bit) from [<801c8914>] (wait_on_page_bit+0x90/0x98)
[   21.834924]  r9:8338fb9c r8:84246a6c r7:0007ffff r6:00000000 r5:8338fb68 r4:0000000d
[   21.878025] [<801c8884>] (wait_on_page_bit) from [<801c89ec>] (__filemap_fdatawait_range+0xd0/0x104)
[   21.955693]  r4:9fbbf9e0
[   21.995998] [<801c891c>] (__filemap_fdatawait_range) from [<801c8a40>] (filemap_fdatawait_range+0x20/0x3c)
[   22.074213]  r9:00000000 r8:83484c00 r7:00000000 r6:7fffffff r5:84246a6c r4:7fffffff
[   22.118310] [<801c8a20>] (filemap_fdatawait_range) from [<801c8ab8>] (filemap_fdatawait+0x5c/0x68)
[   22.195395]  r7:00000000 r6:80000000 r5:00000000 r4:7fffffff
[   22.238120] [<801c8a5c>] (filemap_fdatawait) from [<801cad40>] (filemap_write_and_wait+0x54/0x7c)
[   22.315667]  r7:7fffffff r6:ffffffff r5:84246a6c r4:00000000
[   22.358436] [<801cacec>] (filemap_write_and_wait) from [<8024ba70>] (sync_blockdev.part.6+0x18/0x1c)
[   22.436527]  r7:8344f594 r6:00000000 r5:83484c00 r4:00000000
[   22.478429] [<8024ba58>] (sync_blockdev.part.6) from [<8024ba8c>] (sync_blockdev+0x18/0x24)
[   22.555808] [<8024ba74>] (sync_blockdev) from [<802d2f54>] (jbd2_journal_recover+0xf8/0x138)
[   22.600072] [<802d2e5c>] (jbd2_journal_recover) from [<802d6d64>] (jbd2_journal_load+0xb4/0x2f4)
[   22.678190]  r6:00000000 r5:83484c00 r4:00000002
[   22.719010] [<802d6cb0>] (jbd2_journal_load) from [<802a77e4>] (ext4_fill_super+0x2fd8/0x38fc)
[   22.797082]  r10:81bac2cc r9:833ec000 r8:833eb000 r7:8344f594 r6:00000000 r5:83484c00
[   22.874246]  r4:833c0500
[   22.915351] [<802a480c>] (ext4_fill_super) from [<802176d4>] (mount_bdev+0x164/0x190)
[   22.959100]  r10:00000000 r9:00000000 r8:00000000 r7:84246968 r6:00000083 r5:833eb000
[   23.036244]  r4:84246900
[   23.077308] [<80217570>] (mount_bdev) from [<8029f3e0>] (ext4_mount+0x20/0x28)
[   23.153821]  r9:00000020 r8:00000000 r7:81b1517c r6:81b1517c r5:8742a5c0 r4:8029f3c0
[   23.198696] [<8029f3c0>] (ext4_mount) from [<802180a0>] (mount_fs+0x1c/0xac)
[   23.275061] [<80218084>] (mount_fs) from [<80235168>] (vfs_kern_mount.part.2+0x50/0xe8)
[   23.319134]  r6:00000000 r5:8742a5c0 r4:83463b00
[   23.359879] [<80235118>] (vfs_kern_mount.part.2) from [<802377c4>] (do_mount+0x164/0xc1c)
[   23.437512]  r9:00000020 r8:8742a000 r7:8742a5c0 r6:81b1517c r5:00000000 r4:00000000
[   23.514835] [<80237660>] (do_mount) from [<802385f8>] (SyS_mount+0x5c/0xc8)
[   23.559022]  r10:00000015 r9:8338e000 r8:c0ed0000 r7:765c5bf0 r6:00000000 r5:8742a000
[   23.636107]  r4:8742a5c0
[   23.677334] [<8023859c>] (SyS_mount) from [<80107ca0>] (ret_fast_syscall+0x0/0x50)
[   23.754345]  r8:80107ea4 r7:00000015 r6:00000000 r5:00000001 r4:00000000
[   34.400147] INFO: task python3:241 blocked for more than 10 seconds.
[   34.434135]       Not tainted 4.9.144-rt94-2.8.5+g5758a8e #4
[   34.478719] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   34.556863] python3         D    0   241      1 0x00000000
[   34.599156] Backtrace:
[   34.639487] [<808c8654>] (__schedule) from [<808c8a50>] (schedule+0x58/0xf4)
[   34.715936]  r10:81bac2cc r9:8338fb9c r8:808c94f8 r7:7fffffff r6:00000000 r5:ffffe000
[   34.759897]  r4:833d2080
[   34.799313] [<808c89f8>] (schedule) from [<808caeac>] (schedule_timeout+0x1b0/0x2d4)
[   34.876505]  r5:9fb18440 r4:7fffffff
[   34.919081] [<808cacfc>] (schedule_timeout) from [<808c861c>] (io_schedule_timeout+0x7c/0xb4)
[   34.997111]  r10:81bac2cc r9:8338fb9c r8:808c94f8 r7:7fffffff r6:00000000 r5:9fb18440
[   35.074356]  r4:9fb17f80
[   35.115832] [<808c85a0>] (io_schedule_timeout) from [<808c9510>] (bit_wait_io+0x18/0x64)
[   35.193873]  r7:81b024a0 r6:8338fb24 r5:00000002 r4:00000002
[   35.238566] [<808c94f8>] (bit_wait_io) from [<808c9154>] (__wait_on_bit+0x68/0xc0)
[   35.315657]  r5:00000002 r4:8338fb18
[   35.358140] [<808c90ec>] (__wait_on_bit) from [<801c8914>] (wait_on_page_bit+0x90/0x98)
[   35.436103]  r9:8338fb9c r8:84246a6c r7:0007ffff r6:00000000 r5:8338fb68 r4:0000000d
[   35.514224] [<801c8884>] (wait_on_page_bit) from [<801c89ec>] (__filemap_fdatawait_range+0xd0/0x104)
[   35.559979]  r4:9fbbf9e0
[   35.599337] [<801c891c>] (__filemap_fdatawait_range) from [<801c8a40>] (filemap_fdatawait_range+0x20/0x3c)
[   35.679182]  r9:00000000 r8:83484c00 r7:00000000 r6:7fffffff r5:84246a6c r4:7fffffff
[   35.757151] [<801c8a20>] (filemap_fdatawait_range) from [<801c8ab8>] (filemap_fdatawait+0x5c/0x68)
[   35.836037]  r7:00000000 r6:80000000 r5:00000000 r4:7fffffff
[   35.879467] [<801c8a5c>] (filemap_fdatawait) from [<801cad40>] (filemap_write_and_wait+0x54/0x7c)
[   35.958499]  r7:7fffffff r6:ffffffff r5:84246a6c r4:00000000
[   36.033437] [<801cacec>] (filemap_write_and_wait) from [<8024ba70>] (sync_blockdev.part.6+0x18/0x1c)
[   36.079000]  r7:8344f594 r6:00000000 r5:83484c00 r4:00000000
[   36.154474] [<8024ba58>] (sync_blockdev.part.6) from [<8024ba8c>] (sync_blockdev+0x18/0x24)
[   36.198840] [<8024ba74>] (sync_blockdev) from [<802d2f54>] (jbd2_journal_recover+0xf8/0x138)
[   36.277569] [<802d2e5c>] (jbd2_journal_recover) from [<802d6d64>] (jbd2_journal_load+0xb4/0x2f4)
[   36.356189]  r6:00000000 r5:83484c00 r4:00000002
[   36.399214] [<802d6cb0>] (jbd2_journal_load) from [<802a77e4>] (ext4_fill_super+0x2fd8/0x38fc)
[   36.477725]  r10:81bac2cc r9:833ec000 r8:833eb000 r7:8344f594 r6:00000000 r5:83484c00
[   36.555657]  r4:833c0500
[   36.597406] [<802a480c>] (ext4_fill_super) from [<802176d4>] (mount_bdev+0x164/0x190)
[   36.674632]  r10:00000000 r9:00000000 r8:00000000 r7:84246968 r6:00000083 r5:833eb000
[   36.719361]  r4:84246900
[   36.758982] [<80217570>] (mount_bdev) from [<8029f3e0>] (ext4_mount+0x20/0x28)
[   36.836099]  r9:00000020 r8:00000000 r7:81b1517c r6:81b1517c r5:8742a5c0 r4:8029f3c0
[   36.913341] [<8029f3c0>] (ext4_mount) from [<802180a0>] (mount_fs+0x1c/0xac)
[   36.959145] [<80218084>] (mount_fs) from [<80235168>] (vfs_kern_mount.part.2+0x50/0xe8)
[   37.036907]  r6:00000000 r5:8742a5c0 r4:83463b00
[   37.079317] [<80235118>] (vfs_kern_mount.part.2) from [<802377c4>] (do_mount+0x164/0xc1c)
[   37.157442]  r9:00000020 r8:8742a000 r7:8742a5c0 r6:81b1517c r5:00000000 r4:00000000
[   37.235264] [<80237660>] (do_mount) from [<802385f8>] (SyS_mount+0x5c/0xc8)
[   37.278879]  r10:00000015 r9:8338e000 r8:c0ed0000 r7:765c5bf0 r6:00000000 r5:8742a000
[   37.356346]  r4:8742a5c0
[   37.397831] [<8023859c>] (SyS_mount) from [<80107ca0>] (ret_fast_syscall+0x0/0x50)
[   37.475180]  r8:80107ea4 r7:00000015 r6:00000000 r5:00000001 r4:00000000
[   47.840143] INFO: task python3:241 blocked for more than 10 seconds.
[   47.873968]       Not tainted 4.9.144-rt94-2.8.5+g5758a8e #4
[   47.918864] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   47.996401] python3         D    0   241      1 0x00000000
[   48.039262] Backtrace:
[   48.078828] [<808c8654>] (__schedule) from [<808c8a50>] (schedule+0x58/0xf4)
[   48.155162]  r10:81bac2cc r9:8338fb9c r8:808c94f8 r7:7fffffff r6:00000000 r5:ffffe000
[   48.199194]  r4:833d2080
[   48.238964] [<808c89f8>] (schedule) from [<808caeac>] (schedule_timeout+0x1b0/0x2d4)
[   48.315919]  r5:9fb18440 r4:7fffffff
[   48.358420] [<808cacfc>] (schedule_timeout) from [<808c861c>] (io_schedule_timeout+0x7c/0xb4)
[   48.436370]  r10:81bac2cc r9:8338fb9c r8:808c94f8 r7:7fffffff r6:00000000 r5:9fb18440
[   48.514007]  r4:9fb17f80
[   48.555270] [<808c85a0>] (io_schedule_timeout) from [<808c9510>] (bit_wait_io+0x18/0x64)
[   48.599237]  r7:81b024a0 r6:8338fb24 r5:00000002 r4:00000002
[   48.674489] [<808c94f8>] (bit_wait_io) from [<808c9154>] (__wait_on_bit+0x68/0xc0)
[   48.719250]  r5:00000002 r4:8338fb18
[   48.759183] [<808c90ec>] (__wait_on_bit) from [<801c8914>] (wait_on_page_bit+0x90/0x98)
[   48.836780]  r9:8338fb9c r8:84246a6c r7:0007ffff r6:00000000 r5:8338fb68 r4:0000000d
[   48.913871] [<801c8884>] (wait_on_page_bit) from [<801c89ec>] (__filemap_fdatawait_range+0xd0/0x104)
[   48.959214]  r4:9fbbf9e0
[   48.999283] [<801c891c>] (__filemap_fdatawait_range) from [<801c8a40>] (filemap_fdatawait_range+0x20/0x3c)
[   49.078609]  r9:00000000 r8:83484c00 r7:00000000 r6:7fffffff r5:84246a6c r4:7fffffff
[   49.156247] [<801c8a20>] (filemap_fdatawait_range) from [<801c8ab8>] (filemap_fdatawait+0x5c/0x68)
[   49.234706]  r7:00000000 r6:80000000 r5:00000000 r4:7fffffff
[   49.279158] [<801c8a5c>] (filemap_fdatawait) from [<801cad40>] (filemap_write_and_wait+0x54/0x7c)
[   49.357500]  r7:7fffffff r6:ffffffff r5:84246a6c r4:00000000
[   49.399738] [<801cacec>] (filemap_write_and_wait) from [<8024ba70>] (sync_blockdev.part.6+0x18/0x1c)
[   49.478611]  r7:8344f594 r6:00000000 r5:83484c00 r4:00000000
[   49.553714] [<8024ba58>] (sync_blockdev.part.6) from [<8024ba8c>] (sync_blockdev+0x18/0x24)
[   49.599112] [<8024ba74>] (sync_blockdev) from [<802d2f54>] (jbd2_journal_recover+0xf8/0x138)
[   49.677157] [<802d2e5c>] (jbd2_journal_recover) from [<802d6d64>] (jbd2_journal_load+0xb4/0x2f4)
[   49.755415]  r6:00000000 r5:83484c00 r4:00000002
[   49.798835] [<802d6cb0>] (jbd2_journal_load) from [<802a77e4>] (ext4_fill_super+0x2fd8/0x38fc)
[   49.877151]  r10:81bac2cc r9:833ec000 r8:833eb000 r7:8344f594 r6:00000000 r5:83484c00
[   49.955168]  r4:833c0500
[   49.996538] [<802a480c>] (ext4_fill_super) from [<802176d4>] (mount_bdev+0x164/0x190)
[   50.074189]  r10:00000000 r9:00000000 r8:00000000 r7:84246968 r6:00000083 r5:833eb000
[   50.119435]  r4:84246900
[   50.159206] [<80217570>] (mount_bdev) from [<8029f3e0>] (ext4_mount+0x20/0x28)
[   50.236336]  r9:00000020 r8:00000000 r7:81b1517c r6:81b1517c r5:8742a5c0 r4:8029f3c0
[   50.313805] [<8029f3c0>] (ext4_mount) from [<802180a0>] (mount_fs+0x1c/0xac)
[   50.358934] [<80218084>] (mount_fs) from [<80235168>] (vfs_kern_mount.part.2+0x50/0xe8)
[   50.436594]  r6:00000000 r5:8742a5c0 r4:83463b00
[   50.479074] [<80235118>] (vfs_kern_mount.part.2) from [<802377c4>] (do_mount+0x164/0xc1c)
[   50.556817]  r9:00000020 r8:8742a000 r7:8742a5c0 r6:81b1517c r5:00000000 r4:00000000
[   50.634613] [<80237660>] (do_mount) from [<802385f8>] (SyS_mount+0x5c/0xc8)
[   50.678871]  r10:00000015 r9:8338e000 r8:c0ed0000 r7:765c5bf0 r6:00000000 r5:8742a000
[   50.756645]  r4:8742a5c0
[   50.798061] [<8023859c>] (SyS_mount) from [<80107ca0>] (ret_fast_syscall+0x0/0x50)
[   50.875200]  r8:80107ea4 r7:00000015 r6:00000000 r5:00000001 r4:00000000

[  615.990154] mmc0: Card stuck in programming state! mmcblk0 card_busy_detect
[  616.002937] mmc0: cache flush error -110
[  619.070237] mmc0: tried to reset card, got error -110
[  619.089150] blk_update_request: I/O error, dev mmcblk0, sector 139264
[  619.101145] Buffer I/O error on dev mmcblk0p2, logical block 0, lost async page write
[  619.114583] blk_update_request: I/O error, dev mmcblk0, sector 139272
[  619.126501] Buffer I/O error on dev mmcblk0p2, logical block 1, lost async page write
[  629.600164] mmc0: Timeout waiting for hardware interrupt. retries left=0 opcode=19
[  629.621614] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[  629.641837] mmc0: sdhci: Sys addr:  0x2d9a2200 | Version:  0x00000002
[  629.661986] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000010
[  629.682032] mmc0: sdhci: Argument:  0x00022428 | Trn mode: 0x0000002b
[  629.702007] mmc0: sdhci: Present:   0x01fd8009 | Host ctl: 0x00000011
[  629.721869] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
[  629.741593] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x000010ff
[  629.761311] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
[  629.780919] mmc0: sdhci: Int enab:  0x107f100b | Sig enab: 0x107f100b
[  629.800437] mmc0: sdhci: AC12 err:  0x00000082 | Slot int: 0x00000003
[  629.819814] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x0000a007
[  629.839091] mmc0: sdhci: Cmd:       0x0000193a | Max curr: 0x00ffffff
[  629.858249] mmc0: sdhci: Resp[0]:   0x00ff8080 | Resp[1]:  0xffffffff
[  629.877274] mmc0: sdhci: Resp[2]:   0x320f5913 | Resp[3]:  0x00000900
[  629.896140] mmc0: sdhci: Host ctl2: 0x00000000
[  629.912866] mmc0: sdhci: ADMA Err:  0x00000003 | ADMA Ptr: 0x18078204
[  629.931575] mmc0: sdhci: ============================================
[  629.951174] mmcblk0: error -110 sending status command, retrying
[  629.962462] mmcblk0: error -110 sending status command, retrying
[  629.973668] mmcblk0: error -110 sending status command, aborting
[  629.984406] blk_update_request: I/O error, dev mmcblk0, sector 140328
[  629.995503] Buffer I/O error on dev mmcblk0p2, logical block 133, lost async page write
[  630.008197] blk_update_request: I/O error, dev mmcblk0, sector 140336
[  630.019272] Buffer I/O error on dev mmcblk0p2, logical block 134, lost async page write
[  640.480187] mmc0: Timeout waiting for hardware interrupt. retries left=0 opcode=19
[  640.499364] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[  640.517292] mmc0: sdhci: Sys addr:  0x2d98a200 | Version:  0x00000002
[  640.535113] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000008
[  640.552771] mmc0: sdhci: Argument:  0x00022488 | Trn mode: 0x0000002b
[  640.570314] mmc0: sdhci: Present:   0x01fd8009 | Host ctl: 0x00000011
[  640.587670] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
[  640.604868] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x000010ff
[  640.621919] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
[  640.638894] mmc0: sdhci: Int enab:  0x107f100b | Sig enab: 0x107f100b
[  640.655773] mmc0: sdhci: AC12 err:  0x00000082 | Slot int: 0x00000003
[  640.672522] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x0000a007
[  640.689128] mmc0: sdhci: Cmd:       0x0000193a | Max curr: 0x00ffffff
[  640.705592] mmc0: sdhci: Resp[0]:   0x00ff8080 | Resp[1]:  0xffffffff
[  640.721947] mmc0: sdhci: Resp[2]:   0x320f5913 | Resp[3]:  0x00000900
[  640.738177] mmc0: sdhci: Host ctl2: 0x00000000
[  640.752260] mmc0: sdhci: ADMA Err:  0x00000003 | ADMA Ptr: 0x18078204
[  640.768297] mmc0: sdhci: ============================================
[  640.785506] mmcblk0: error -110 sending status command, retrying
[  640.795696] mmcblk0: error -110 sending status command, retrying
[  640.805809] mmcblk0: error -110 sending status command, aborting
[  640.815461] blk_update_request: I/O error, dev mmcblk0, sector 140424
[  640.825526] Buffer I/O error on dev mmcblk0p2, logical block 145, lost async page write
[  651.360162] mmc0: Timeout waiting for hardware interrupt. retries left=0 opcode=19
[  651.376901] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[  651.392427] mmc0: sdhci: Sys addr:  0x2d984200 | Version:  0x00000002
[  651.407878] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000008
[  651.423205] mmc0: sdhci: Argument:  0x00022498 | Trn mode: 0x0000002b
[  651.438422] mmc0: sdhci: Present:   0x01fd8009 | Host ctl: 0x00000011
[  651.453529] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
[  651.468524] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x000010ff
[  651.483428] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
[  651.498304] mmc0: sdhci: Int enab:  0x107f100b | Sig enab: 0x107f100b
[  651.513120] mmc0: sdhci: AC12 err:  0x00000082 | Slot int: 0x00000003
[  651.527836] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x0000a007
[  651.542430] mmc0: sdhci: Cmd:       0x0000193a | Max curr: 0x00ffffff
[  651.556921] mmc0: sdhci: Resp[0]:   0x00ff8080 | Resp[1]:  0xffffffff
[  651.571282] mmc0: sdhci: Resp[2]:   0x320f5913 | Resp[3]:  0x00000900
[  651.585534] mmc0: sdhci: Host ctl2: 0x00000000
[  651.597725] mmc0: sdhci: ADMA Err:  0x00000003 | ADMA Ptr: 0x18078204
[  651.611997] mmc0: sdhci: ============================================
[  651.627304] mmcblk0: error -110 sending status command, retrying
[  651.636875] mmcblk0: error -110 sending status command, retrying
[  651.646415] mmcblk0: error -110 sending status command, aborting
[  651.655536] blk_update_request: I/O error, dev mmcblk0, sector 140440
[  651.665124] Buffer I/O error on dev mmcblk0p2, logical block 147, lost async page write
[  662.240161] mmc0: Timeout waiting for hardware interrupt. retries left=0 opcode=19
[  662.255941] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[  662.270595] mmc0: sdhci: Sys addr:  0x2d981200 | Version:  0x00000002
[  662.285295] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000008
[  662.299928] mmc0: sdhci: Argument:  0x00022a78 | Trn mode: 0x0000002b
[  662.314624] mmc0: sdhci: Present:   0x01fd8009 | Host ctl: 0x00000011
[  662.329244] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
[  662.343891] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x000010ff
[  662.358443] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
[  662.372963] mmc0: sdhci: Int enab:  0x107f100b | Sig enab: 0x107f100b
[  662.387559] mmc0: sdhci: AC12 err:  0x00000082 | Slot int: 0x00000003
[  662.402232] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x0000a007
[  662.416836] mmc0: sdhci: Cmd:       0x0000193a | Max curr: 0x00ffffff
[  662.431312] mmc0: sdhci: Resp[0]:   0x00ff8080 | Resp[1]:  0xffffffff
[  662.445658] mmc0: sdhci: Resp[2]:   0x320f5913 | Resp[3]:  0x00000900
[  662.459831] mmc0: sdhci: Host ctl2: 0x00000000
[  662.471896] mmc0: sdhci: ADMA Err:  0x00000003 | ADMA Ptr: 0x18078204
[  662.485939] mmc0: sdhci: ============================================
[  662.501008] mmcblk0: error -110 sending status command, retrying
[  662.510520] mmcblk0: error -110 sending status command, retrying
[  662.519960] mmcblk0: error -110 sending status command, aborting
[  662.528967] blk_update_request: I/O error, dev mmcblk0, sector 141944
[  662.538438] Buffer I/O error on dev mmcblk0p2, logical block 335, lost async page write
[  673.120160] mmc0: Timeout waiting for hardware interrupt. retries left=0 opcode=19
[  673.135702] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[  673.150116] mmc0: sdhci: Sys addr:  0x2d989200 | Version:  0x00000002
[  673.164554] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000008
[  673.178934] mmc0: sdhci: Argument:  0x00022be0 | Trn mode: 0x0000002b
[  673.193347] mmc0: sdhci: Present:   0x01fd8009 | Host ctl: 0x00000011
[  673.207748] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
[  673.222182] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x000010ff
[  673.236570] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
[  673.250934] mmc0: sdhci: Int enab:  0x107f100b | Sig enab: 0x107f100b
[  673.265266] mmc0: sdhci: AC12 err:  0x00000082 | Slot int: 0x00000003
[  673.279513] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x0000a007
[  673.293861] mmc0: sdhci: Cmd:       0x0000193a | Max curr: 0x00ffffff
[  673.308230] mmc0: sdhci: Resp[0]:   0x00ff8080 | Resp[1]:  0xffffffff
[  673.322570] mmc0: sdhci: Resp[2]:   0x320f5913 | Resp[3]:  0x00000900
[  673.336766] mmc0: sdhci: Host ctl2: 0x00000000
[  673.348771] mmc0: sdhci: ADMA Err:  0x00000003 | ADMA Ptr: 0x18078204
[  673.362795] mmc0: sdhci: ============================================
[  673.377818] mmcblk0: error -110 sending status command, retrying
[  673.387285] mmcblk0: error -110 sending status command, retrying
[  673.396713] mmcblk0: error -110 sending status command, aborting
[  673.405721] blk_update_request: I/O error, dev mmcblk0, sector 142304
[  673.415198] Buffer I/O error on dev mmcblk0p2, logical block 380, lost async page write

[  727.783815] mmcblk0: error -110 sending status command, retrying
[  727.793349] mmcblk0: error -110 sending status command, retrying
[  727.802851] mmcblk0: error -110 sending status command, aborting
[  727.811932] blk_update_request: I/O error, dev mmcblk0, sector 561176
[  727.821461] Buffer I/O error on dev mmcblk0p2, logical block 52739, lost async page write
[  727.834496] mmc0: cache flush error -110
[  727.841601] blk_update_request: I/O error, dev mmcblk0, sector 0
[  727.850850] JBD2: recovery failed
[  727.857322] EXT4-fs (mmcblk0p2): error loading journal
Traceback (most recent call last):
  File "init.py", line 3, in <module>
    kezu_core.main()
  File "/usr/lib/python3.5/site-packages/kezu_core/__init__.py", line 1487, in main
    mount_everything()
  File "/usr/lib/python3.5/site-packages/kezu_core/__init__.py", line 564, in mount_everything
    mount_parted_partition(p_partition)
  File "/usr/lib/python3.5/site-packages/kezu_core/__init__.py", line 465, in mount_parted_partition
    libc.mount(source=path, target=mp, fs_type=fs)
  File "/usr/lib/python3.5/site-packages/kezu_core/libc.py", line 57, in mount
    raise OSError(ctypes.get_errno())
OSError: 0
[  727.979438] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[  727.979438]
[  727.979451] CPU1: stopping
[  727.979458] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.9.144-rt94-2.8.5+g5758a8e #4
[  727.979461] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[  727.979466] Backtrace:
[  727.979488] [<8010b864>] (dump_backtrace) from [<8010bb3c>] (show_stack+0x18/0x1c)
[  727.979494]  r7:20000193 r6:81b1d578 r5:00000000 r4:81b1d578
[  727.979505] [<8010bb24>] (show_stack) from [<80403598>] (dump_stack+0x80/0x94)
[  727.979515] [<80403518>] (dump_stack) from [<8010ed88>] (handle_IPI+0x1a8/0x204)
[  727.979520]  r7:00000000 r6:00000004 r5:81a3d1d0 r4:81b63ac8
[  727.979527] [<8010ebe0>] (handle_IPI) from [<8010152c>] (gic_handle_irq+0x90/0x94)
[  727.979533]  r9:f4a01100 r8:86091f08 r7:f4a00100 r6:f4a0010c r5:81b1d7c0 r4:81b04890
[  727.979540] [<8010149c>] (gic_handle_irq) from [<8010c68c>] (__irq_svc+0x6c/0xbc)
[  727.979543] Exception stack(0x86091f08 to 0x86091f50)
[  727.979550] 1f00:                   00000001 80a2cb10 86091ef8 1e0e7000 7ef4c1f7 000000a9
[  727.979557] 1f20: 9fb25430 00000000 7ef2fcd7 000000a9 00000000 86091f8c 86090000 86091f58
[  727.979561] 1f40: 80421328 806b647c 20000013 ffffffff
[  727.979567]  r9:86090000 r8:7ef2fcd7 r7:86091f3c r6:ffffffff r5:20000013 r4:806b647c
[  727.979581] [<806b6334>] (cpuidle_enter_state) from [<806b6658>] (cpuidle_enter+0x1c/0x20)
[  727.979588]  r10:81a3e428 r9:81b09ad8 r8:9fb25430 r7:81b04580 r6:00000002 r5:81b0452c
[  727.979590]  r4:ffffe000
[  727.979599] [<806b663c>] (cpuidle_enter) from [<80168894>] (call_cpuidle+0x38/0x50)
[  727.979606] [<8016885c>] (call_cpuidle) from [<80168b6c>] (cpu_startup_entry+0x180/0x23c)
[  727.979614] [<801689ec>] (cpu_startup_entry) from [<8010e964>] (secondary_start_kernel+0x15c/0x168)
[  727.979616]  r7:81b63ad8
[  727.979623] [<8010e808>] (secondary_start_kernel) from [<101015cc>] (0x101015cc)
[  727.979627]  r5:00000051 r4:1606806a
[  728.255471] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100

Dear @qojote

Could you run mmc extcsd read /dev/mmcblk0 and send us the output? And could you try to install our reference image Toradex Embedded Linux Console Demo (3.0b4.254) with the Toradex Easy Installer and boot it? If your image doesn’t contain the mmc-utils, you could either post-install it or run it within our reference image (in our image it’s probably a different device number).

Do you know if your application does a lot of write cycles to the eMMC? Is this the first kind of this issue or do you experience similar things more often?

Best regards
Diego

Hi @diego_b.tx ,
Please find the output of “mmc extcsd read /dev/mmcblk0” below. According to your website the eMMC health status is okay, right? U-boot was still able to read and access the mmc 0:1 partition so i have replaced the zImage (via USB drive). After that the system booted normally… Is this to be expected?

Do you know if your application does a lot of write cycles to the eMMC?

I have investigated the application in terms of write actions to the eMMC. The application seems to write data at a rate of 1MB/min.

Is this the first kind of this issue or do you experience similar things more often?

This is the first issue with an Apalis iMX6 module.

mmc extcsd read /dev/mmcblk0
=============================================
  Extended CSD rev 1.7 (MMC 5.0)
=============================================

Card Supported Command sets [S_CMD_SET: 0x01]
HPI Features [HPI_FEATURE: 0x01]: implementation based on CMD13
Background operations support [BKOPS_SUPPORT: 0x01]
Max Packet Read Cmd [MAX_PACKED_READS: 0x3f]
Max Packet Write Cmd [MAX_PACKED_WRITES: 0x3f]
Data TAG support [DATA_TAG_SUPPORT: 0x01]
Data TAG Unit Size [TAG_UNIT_SIZE: 0x03]
Tag Resources Size [TAG_RES_SIZE: 0x00]
Context Management Capabilities [CONTEXT_CAPABILITIES: 0x05]
Large Unit Size [LARGE_UNIT_SIZE_M1: 0x01]
Extended partition attribute support [EXT_SUPPORT: 0x03]
Supported modes [SUPPORTED_MODES: 0x03]
FFU features [FFU_FEATURES: 0x00]
Operation codes timeout [OPERATION_CODE_TIMEOUT: 0x00]
FFU Argument [FFU_ARG: 0x0000ffff]
Number of FW sectors correctly programmed [NUMBER_OF_FW_SECTORS_CORRECTLY_PROGRAMMED: 0]
Vendor proprietary health report:
[VENDOR_PROPRIETARY_HEALTH_REPORT[301]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[300]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[299]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[298]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[297]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[296]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[295]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[294]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[293]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[292]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[291]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[290]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[289]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[288]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[287]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[286]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[285]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[284]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[283]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[282]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[281]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[280]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[279]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[278]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[277]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[276]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[275]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[274]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[273]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[272]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[271]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[270]]: 0x00
Device life time estimation type B [DEVICE_LIFE_TIME_EST_TYP_B: 0x01]
i.e. 0% - 10% device life time used
Device life time estimation type A [DEVICE_LIFE_TIME_EST_TYP_A: 0x01]
i.e. 0% - 10% device life time used
Pre EOL information [PRE_EOL_INFO: 0x01]
i.e. Normal
Optimal read size [OPTIMAL_READ_SIZE: 0x01]
Optimal write size [OPTIMAL_WRITE_SIZE: 0x04]
Optimal trim unit size [OPTIMAL_TRIM_UNIT_SIZE: 0x01]
Device version [DEVICE_VERSION: 0x00 - 0x00]
Firmware version:
[FIRMWARE_VERSION[261]]: 0x00
[FIRMWARE_VERSION[260]]: 0x00
[FIRMWARE_VERSION[259]]: 0x00
[FIRMWARE_VERSION[258]]: 0x00
[FIRMWARE_VERSION[257]]: 0x00
[FIRMWARE_VERSION[256]]: 0x00
[FIRMWARE_VERSION[255]]: 0x00
[FIRMWARE_VERSION[254]]: 0x05
Power class for 200MHz, DDR at VCC= 3.6V [PWR_CL_DDR_200_360: 0x00]
Generic CMD6 Timer [GENERIC_CMD6_TIME: 0x19]
Power off notification [POWER_OFF_LONG_TIME: 0xff]
Cache Size [CACHE_SIZE] is 512 KiB
Background operations status [BKOPS_STATUS: 0x00]
1st Initialisation Time after programmed sector [INI_TIMEOUT_AP: 0x64]
Power class for 52MHz, DDR at 3.6V [PWR_CL_DDR_52_360: 0x00]
Power class for 52MHz, DDR at 1.95V [PWR_CL_DDR_52_195: 0x00]
Power class for 200MHz at 3.6V [PWR_CL_200_360: 0x00]
Power class for 200MHz, at 1.95V [PWR_CL_200_195: 0x00]
Minimum Performance for 8bit at 52MHz in DDR mode:
[MIN_PERF_DDR_W_8_52: 0x0b]
[MIN_PERF_DDR_R_8_52: 0x00]
TRIM Multiplier [TRIM_MULT: 0x11]
Secure Feature support [SEC_FEATURE_SUPPORT: 0x55]
Boot Information [BOOT_INFO: 0x07]
Device supports alternative boot method
Device supports dual data rate during boot
Device supports high speed timing during boot
Boot partition size [BOOT_SIZE_MULTI: 0x10]
Access size [ACC_SIZE: 0x06]
High-capacity erase unit size [HC_ERASE_GRP_SIZE: 0x01]
i.e. 512 KiB
High-capacity erase timeout [ERASE_TIMEOUT_MULT: 0x11]
Reliable write sector count [REL_WR_SEC_C: 0x01]
High-capacity W protect group size [HC_WP_GRP_SIZE: 0x10]
i.e. 8192 KiB
Sleep current (VCC) [S_C_VCC: 0x08]
Sleep current (VCCQ) [S_C_VCCQ: 0x08]
Sleep/awake timeout [S_A_TIMEOUT: 0x13]
Sector Count [SEC_COUNT: 0x00748000]
Device is block-addressed
Minimum Write Performance for 8bit:
[MIN_PERF_W_8_52: 0x0b]
[MIN_PERF_R_8_52: 0x08]
[MIN_PERF_W_8_26_4_52: 0x0c]
[MIN_PERF_R_8_26_4_52: 0x08]
Minimum Write Performance for 4bit:
[MIN_PERF_W_4_26: 0x0b]
[MIN_PERF_R_4_26: 0x08]
Power classes registers:
[PWR_CL_26_360: 0x00]
[PWR_CL_52_360: 0x00]
[PWR_CL_26_195: 0x00]
[PWR_CL_52_195: 0x00]
Partition switching timing [PARTITION_SWITCH_TIME: 0x03]
Out-of-interrupt busy timing [OUT_OF_INTERRUPT_TIME: 0x0a]
I/O Driver Strength [DRIVER_STRENGTH: 0x1f]
Card Type [CARD_TYPE: 0x57 - 00]
HS400 Dual Data Rate eMMC @200MHz 1.8VI/O
HS200 Single Data Rate eMMC @200MHz 1.8VI/O
HS Dual Data Rate eMMC @52MHz 1.8V or 3VI/O
HS eMMC @52MHz - at rated device voltage(s)
HS eMMC @26MHz - at rated device voltage(s)
CSD structure version [CSD_STRUCTURE: 0x02]
Command set [CMD_SET: 0x00]
Command set revision [CMD_SET_REV: 0x00]
Power class [POWER_CLASS: 0x00]
High-speed interface timing [HS_TIMING: 0x01]
Erased memory content [ERASED_MEM_CONT: 0x00]
Boot configuration bytes [PARTITION_CONFIG: 0x48]
Boot Partition 1 enabled
No access to boot partition
Boot config protection [BOOT_CONFIG_PROT: 0x00]
Boot bus Conditions [BOOT_BUS_CONDITIONS: 0x0a]
High-density erase group definition [ERASE_GROUP_DEF: 0x00]
Boot write protection status registers [BOOT_WP_STATUS]: 0x00
Boot Area Write protection [BOOT_WP]: 0x00
Power ro locking: possible
Permanent ro locking: possible
ro lock status: not locked
User area write protection register [USER_WP]: 0x00
FW configuration [FW_CONFIG]: 0x00
RPMB Size [RPMB_SIZE_MULT]: 0x04
Write reliability setting register [WR_REL_SET]: 0x1f
user area: the device protects existing data if a power failure occurs during a write operation
partition 1: the device protects existing data if a power failure occurs during a write operation
partition 2: the device protects existing data if a power failure occurs during a write operation
partition 3: the device protects existing data if a power failure occurs during a write operation
partition 4: the device protects existing data if a power failure occurs during a write operation
Write reliability parameter register [WR_REL_PARAM]: 0x15
Device supports writing EXT_CSD_WR_REL_SET
Device supports the enhanced def. of reliable write
Enable background operations handshake [BKOPS_EN]: 0x00
H/W reset function [RST_N_FUNCTION]: 0x01
HPI management [HPI_MGMT]: 0x01
Partitioning Support [PARTITIONING_SUPPORT]: 0x07
Device support partitioning feature
Device can have enhanced tech.
Max Enhanced Area Size [MAX_ENH_SIZE_MULT]: 0x0000e9
i.e. 1908736 KiB
Partitions attribute [PARTITIONS_ATTRIBUTE]: 0x00
Partitioning Setting [PARTITION_SETTING_COMPLETED]: 0x00
Device partition setting NOT complete
General Purpose Partition Size
[GP_SIZE_MULT_4]: 0x000000
[GP_SIZE_MULT_3]: 0x000000
[GP_SIZE_MULT_2]: 0x000000
[GP_SIZE_MULT_1]: 0x000000
Enhanced User Data Area Size [ENH_SIZE_MULT]: 0x000000
i.e. 0 KiB
Enhanced User Data Start Address [ENH_START_ADDR]: 0x000000
i.e. 0 bytes offset
Bad Block Management mode [SEC_BAD_BLK_MGMNT]: 0x00
Periodic Wake-up [PERIODIC_WAKEUP]: 0x00
Program CID/CSD in DDR mode support [PROGRAM_CID_CSD_DDR_SUPPORT]: 0x01
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[127]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[126]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[125]]: 0x10
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[124]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[123]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[122]]: 0x04
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[121]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[120]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[119]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[118]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[117]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[116]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[115]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[114]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[113]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[112]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[111]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[110]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[109]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[108]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[107]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[106]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[105]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[104]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[103]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[102]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[101]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[100]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[99]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[98]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[97]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[96]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[95]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[94]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[93]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[92]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[91]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[90]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[89]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[88]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[87]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[86]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[85]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[84]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[83]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[82]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[81]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[80]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[79]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[78]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[77]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[76]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[75]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[74]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[73]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[72]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[71]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[70]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[69]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[68]]: 0x01
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[67]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[66]]: 0x01
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[65]]: 0xaf
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[64]]: 0x03
Native sector size [NATIVE_SECTOR_SIZE]: 0x00
i.e. 512 B
Sector size emulation [USE_NATIVE_SECTOR]: 0x00
Sector size [DATA_SECTOR_SIZE]: 0x00
i.e. 512 B
1st initialization after disabling sector size emulation [INI_TIMEOUT_EMU]: 0x00
Class 6 commands control [CLASS_6_CTRL]: 0x00
Number of addressed group to be Released[DYNCAP_NEEDED]: 0x00
Exception events control [EXCEPTION_EVENTS_CTRL]: 0x0000
Exception events status[EXCEPTION_EVENTS_STATUS]: 0x0000
Extended Partitions Attribute [EXT_PARTITIONS_ATTRIBUTE]: 0x0000
Context configuration [CONTEXT_CONF[51]]: 0x00
Context configuration [CONTEXT_CONF[50]]: 0x00
Context configuration [CONTEXT_CONF[49]]: 0x00
Context configuration [CONTEXT_CONF[48]]: 0x00
Context configuration [CONTEXT_CONF[47]]: 0x00
Context configuration [CONTEXT_CONF[46]]: 0x00
Context configuration [CONTEXT_CONF[45]]: 0x00
Context configuration [CONTEXT_CONF[44]]: 0x00
Context configuration [CONTEXT_CONF[43]]: 0x00
Context configuration [CONTEXT_CONF[42]]: 0x00
Context configuration [CONTEXT_CONF[41]]: 0x00
Context configuration [CONTEXT_CONF[40]]: 0x00
Context configuration [CONTEXT_CONF[39]]: 0x00
Context configuration [CONTEXT_CONF[38]]: 0x00
Context configuration [CONTEXT_CONF[37]]: 0x00
Packed command status [PACKED_COMMAND_STATUS]: 0x00
Packed command failure index [PACKED_FAILURE_INDEX]: 0x00
Power Off Notification [POWER_OFF_NOTIFICATION]: 0x01
Control to turn the Cache ON/OFF [CACHE_CTRL]: 0x01
Mode config [MODE_CONFIG: 0x00]
Mode operation codes [MODE_OPERATION_CODES: 0x00]
FFU status [FFU_STATUS: 0x00]
Success
Pre loading data size [PRE_LOADING_DATA_SIZE] is 0 sector size
Max pre loading data size [MAX_PRE_LOADING_DATA_SIZE] is 3784704 sector size
Product state awareness enablement [PRODUCT_STATE_AWARENESS_ENABLEMENT: 0x01]
Secure Removal Type [SECURE_REMOVAL_TYPE: 0x01]

Dear @qojote

As far as I can see the health status seems to be okay.

U-boot was still able to read and access the mmc 0:1 partition so i have replaced the zImage (via USB drive). After that the system booted normally… Is this to be expected?

Yes, usually this is a possible way. Did you try to flash our Toradex Embedded Linux Console Demo image and see how this performs? Could you try to flash your custom image completely from scratch and check if the error still persists?

Best regards
Diego

Hi @diego_b.tx ,

Could you try to flash your custom image completely from scratch and check if the error still persists?

After flashing my custom image via TEZI the systems works like expected. However, what might have caused this issue?

Although the device is running again, i still need to know why this problem occurred or at least how to prevent from it. The device will be sent back to the customer but you can image what problems arise when the same problem pops up again in the field.

Hi @qojote

Good that flashing the image newly works!

It’s hard to say what the root cause could have been. In your initial post with the log, I see some Python3 application which accesses the filesystem. And it seems like at some point your root file system got corrupt. So one possible conclusion could be, that something happened during the filesystem access with the Python3 application and that this access wasn’t finished properly. It’s important to always close file streams and synchronize the filesystem, even if the application fails (with proper exception handling).

This is just one guess and from my position it’s hard to make any other guess. Just observe the situation and check if it repeats, may even with other devices with the same image.

Best regards
Diego

Hi,
Unfortunately another device “died” in field with the same issue… We were able to get the iMX6 module back for investigation. What we could check is that the device won’t recover by switching it off and on again. Trying to mount the mmc0 partition via bash fails with the same error. However, after the next power cycle the same command succeeded stating that “recovery completed”. Now the error is gone and we cannot reproduce it. We did backup the whole flash via dd command but when writing the flash content back the error still does not occur… Any suggestions?

Just to get a better understanding, some of our devices are used 24/7 for years now without any issues. The “broken” modules are quite young, e.g. one module broke during the production test.

Could you share the serial numbers of the “young” modules?

I can provide you two serial numbers at the moment: 10605731 , 10605695

Thanks for the serial numbers. We checked internally these modules passed the tests before they were sent to you.

It does not look like a hardware issue.

Ok. Thanks for your efforts. We will futher investigate this issue. I will update this ticket if there are new findings.

What exact hardware (module and carrier board) and software versions of things are you talking about?

Apalis iMX6D V1.1B (512MB RAM), BSP is 2.8.7 (we moved to latest in the mean time) with RT kernel patch. Carrier board is custom but highly inspired by the Ixora.

Perfect. Thanks for your Input.

Another device shows up the same issue. Serial number is 10605720

We checked internally, this module passed all the tests before it was sent out.

Thanks for checking. We are still investigating the issue. Right now we are able to reproduce the “bad state” within some minutes (<5min) altough this process only “works” for the mentioned modules. Older modules with the same image are not showing the issue (not once in a day with the same setup).

This is the output when we try to check the rootfs partition before mounting:

    /kezu/debug # fsck.ext4 /dev/mmcblk0p2 -n
    e2fsck 1.43.5 (04-Aug-2017)
    Warning: skipping journal recovery because doing a read-only filesystem check.
    RFS: clean, 17982/131072 files, 195504/524288 blocks
    /kezu/debug # fsck.ext4 /dev/mmcblk0p2
    e2fsck 1.43.5 (04-Aug-2017)
    RFS: recovering journal
    [  662.455888] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x400e00
    [  662.854998] random: crng init done
    [  676.320135] INFO: task fsck.ext4:272 blocked for more than 10 seconds.
    [  676.352625]       Not tainted 4.9.220-rt143-2.8.7+g5722926 #4
    [  676.395178] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    [  676.437094] fsck.ext4       D    0   272    264 0x00000000
    [  676.476641] Backtrace:
    [  676.515701] [<808dbd6c>] (__schedule) from [<808dc168>] (schedule+0x58/0xf4)
    [  676.556775]  r10:00000076 r9:82f6be7c r8:808dcc10 r7:7fffffff r6:00000000 r5:ffffe000
    [  676.598508]  r4:833e6800
    [  676.636597] [<808dc110>] (schedule) from [<808de5b4>] (schedule_timeout+0x1b0/0x2d4)
    [  676.678297]  r5:9fb19440 r4:7fffffff
    [  676.716958] [<808de404>] (schedule_timeout) from [<808dbd34>] (io_schedule_timeout+0x7c/0xb4)
    [  676.759656]  r10:00000076 r9:82f6be7c r8:808dcc10 r7:7fffffff r6:00000000 r5:9fb19440
    [  676.834312]  r4:9fb18f80
    [  676.874147] [<808dbcb8>] (io_schedule_timeout) from [<808dcc28>] (bit_wait_io+0x18/0x64)
    [  676.917148]  r7:81b02c80 r6:82f6be04 r5:00000002 r4:00000002
    [  676.957285] [<808dcc10>] (bit_wait_io) from [<808dc86c>] (__wait_on_bit+0x68/0xc0)
    [  676.998884]  r5:00000002 r4:82f6bdf8
    [  677.036778] [<808dc804>] (__wait_on_bit) from [<801c8adc>] (wait_on_page_bit+0x90/0x98)
    [  677.079219]  r9:82f6be7c r8:8424a5f4 r7:ffffffff r6:00000000 r5:82f6be48 r4:0000000d
    [  677.153317] [<801c8a4c>] (wait_on_page_bit) from [<801c8bb4>] (__filemap_fdatawait_range+0xd0/0x104)
    [  677.197424]  r4:9fbc0ec0
    [  677.236929] [<801c8ae4>] (__filemap_fdatawait_range) from [<801c8c08>] (filemap_fdatawait_range+0x20/0x3c)
    [  677.313324]  r9:7fffffff r8:ffffffff r7:7fffffff r6:ffffffff r5:8424a5f4 r4:00000000
    [  677.357097] [<801c8be8>] (filemap_fdatawait_range) from [<801cafd4>] (filemap_write_and_wait_range+0x60/0x88)
    [  677.433638]  r7:00000000 r6:00000000 r5:8424a5f4 r4:00000000
    [  677.476514] [<801caf74>] (filemap_write_and_wait_range) from [<8024b170>] (blkdev_fsync+0x24/0x54)
    [  677.552318]  r9:82f6a000 r8:80107e24 r7:7fffffff r6:8424a500 r5:7fffffff r4:ffffffff
    [  677.597787] [<8024b14c>] (blkdev_fsync) from [<80243940>] (vfs_fsync_range+0x44/0xa8)
    [  677.672526]  r6:ffffffff r5:00000000 r4:833eb400
    [  677.714295] [<802438fc>] (vfs_fsync_range) from [<80243a1c>] (do_fsync+0x44/0x6c)
    [  677.757473]  r7:00000076 r6:00000000 r5:833eb400 r4:833eb400
    [  677.797866] [<802439d8>] (do_fsync) from [<80243cb8>] (SyS_fsync+0x14/0x18)
    [  677.839645]  r6:76eed2cc r5:00a108b8 r4:00000000
    [  677.878992] [<80243ca4>] (SyS_fsync) from [<80107c20>] (ret_fast_syscall+0x0/0x50)
    
    
    / # mount /dev/mmcblk0p2 /mnt
    [   84.238343] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities
    [   84.258376] EXT4-fs (mmcblk0p2): couldn't mount as ext2 due to feature incompatibilities
    [   84.658079] EXT4-fs (mmcblk0p2): 1 orphan inode deleted
    [   84.674265] EXT4-fs (mmcblk0p2): recovery complete
    [   84.706975] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)