Apalis iMX6 : Unable to read sectors from e-mmc?

fred.rol · November 5, 2018, 1:40pm

Hi all,

Recently we received 3 embedded systems from our RMA service team. These 3 systems, built with Apalis iMX6 and Boot2Qt were unable to boot the software. On the last one, I see that the main software partition is unable to be mounted, causing the system not to reach its operational condition.

Further investigation showed me that the ext4 driver was experiencing read errors when trying to get metadata needed to mount this partition. When trying to ‘dd’ the entire e-mmc to another storage, I experienced similar read errors.

Someone at our RMA service team would have been reporting that full rewrite of the internal flash storage with a clean disk image will get the system back fully fonctional, but I can’t myself confirm this information.

Have you ever heard a similar problem?

Regards,
Fred.

fred.rol · November 5, 2018, 1:45pm

Here are the log files (boot messages and ‘dd’ output)

link text

jaski.tx · November 5, 2018, 4:06pm

Hi @fred.rol

We did not hear about similar Problem till now.

Could you provide the software version of your module (uname -r)?
Could you flash the latest Bsp release 2.8b4 (LXDE or Console Image) on the module using Toradex Easy Installer and check if the issue is still there? If yes, then please share the complete dmesg log in a file (dmesg > log.txt).

Thanks and best regards, Jaski

fred.rol · November 7, 2018, 8:31am

Hi,

I’m going to install the BSP 2.8b4 as you requested.
Currently, we run :

Give root password for maintenance
(or press Control-D to continue): 
sh-4.3# uname -a
Linux b2qt-apalis-imx6 4.1.39-2.7.2+gba2f75f #1 SMP Wed Apr 11 11:54:04 UTC 2018 armv7l armv7l armv7l GNU/Linux

Regards,
Fred.

jaski.tx · November 7, 2018, 9:33am

Perfect. We will wait for your feedback.

fred.rol · November 8, 2018, 10:11am

Hello Jaski,

Here we are.
I used a USB stick to boot on the linux BSP release 2.8b4, just after some minor changes into the live U-boot env vars to deal with ‘zImage’ format of the new kernel (our current BSP 2.7.2 is still using ‘uImage’).
We first decided not to rewrite the internal emmc with the BSP 2.8b4, as we were scared to loose all the precious forensics data …

I let the system boot until the console prompt and check first that it was really the new BSP

.---O---.                                           
|       |                  .-.           o o        
|   |   |-----.-----.-----.| |   .----..-----.-----.
|       |     | __  |  ---'| '--.|  .-'|     |     |
|   |   |  |  |     |---  ||  --'|  |  |  '  | | | |
'---'---'--'--'--.  |-----''----''--'  '-----'-'-'-'
                -'  |
                '---'

The Angstrom Distribution apalis-imx6 ttymxc0

Angstrom v2017.12 - Kernel 

Apalis-iMX6_LXDE-Image 2.8b4.129 20181005

apalis-imx6 login: [   87.196066] mmcblk0: error -110 transferring data, sector 3417648, nr 192, cmd response 0x900, card status 0x200b00
[   92.630255] mmcblk0: error -110 transferring data, sector 3417648, nr 192, cmd response 0x900, card status 0x0
[   92.632762] INFO: task jbd2/mmcblk0p2-:678 blocked for more than 10 seconds.
[   92.632777]       Tainted: G           O    4.9.87-2.8.4+g3bb6e3284a1b #1
[   92.632782] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   92.633364] INFO: task mount:961 blocked for more than 10 seconds.
[   92.633373]       Tainted: G           O    4.9.87-2.8.4+g3bb6e3284a1b #1
[   92.633377] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   92.682974] blk_update_request: I/O error, dev mmcblk0, sector 3417648
[   98.112685] mmcblk0: error -110 transferring data, sector 3417649, nr 191, cmd response 0x900, card status 0x0
[   98.122813] blk_update_request: I/O error, dev mmcblk0, sector 3417649
[  102.872766] INFO: task jbd2/mmcblk0p2-:678 blocked for more than 10 seconds.
[  102.879847]       Tainted: G           O    4.9.87-2.8.4+g3bb6e3284a1b #1
[  102.886710] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  102.895276] INFO: task mount:961 blocked for more than 10 seconds.
[  102.901479]       Tainted: G           O    4.9.87-2.8.4+g3bb6e3284a1b #1
[  102.908330] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  103.552561] mmcblk0: error -110 transferring data, sector 3417650, nr 190, cmd response 0x900, card status 0x0
[  103.562662] blk_update_request: I/O error, dev mmcblk0, sector 3417650
[  108.992408] mmcblk0: error -110 transferring data, sector 3417651, nr 189, cmd response 0x900, card status 0x0
[  109.002522] blk_update_request: I/O error, dev mmcblk0, sector 3417651
[  114.432324] mmcblk0: error -110 transferring data, sector 3417652, nr 188, cmd response 0x900, card status 0x0
[  114.442430] blk_update_request: I/O error, dev mmcblk0, sector 3417652

.---O---.                                           
|       |                  .-.           o o        
|   |   |-----.-----.-----.| |   .----..-----.-----.
|       |     | __  |  ---'| '--.|  .-'|     |     |
|   |   |  |  |     |---  ||  --'|  |  |  '  | | | |
'---'---'--'--'--.  |-----''----''--'  '-----'-'-'-'
                -'  |
                '---'

The Angstrom Distribution apalis-imx6 ttymxc0

Angstrom v2017.12 - Kernel 

Apalis-iMX6_LXDE-Image 2.8b4.129 20181005

apalis-imx6 login: root

[  119.872214] mmcblk0: error -110 transferring data, sector 3417653, nr 187, cmd response 0x900, card status 0x0
[  119.882316] blk_update_request: I/O error, dev mmcblk0, sector 3417653

root@apalis-imx6:~# uname -a
Linux apalis-imx6 4.9.87-2.8.4+g3bb6e3284a1b #1 SMP Fri Oct 5 16:02:39 UTC 2018 armv7l GNU/Linux

As you see now, I get identical error messages :

...
[   34.097366] blk_update_request: I/O error, dev mmcblk0, sector 3417649
[   39.527145] mmcblk0: error -110 transferring data, sector 3417650, nr 190, cmd response 0x900, card status 0x0
...

I give you the complete log of the boot process (my commands under Uboot, then the Angstrom boot) and the ‘dmesg’ output you have requested.

link text

Regards,
Fred

jaski.tx · November 9, 2018, 8:23am

hi Fred

Thanks for the the console output and the bootlog files.

Here we are. I used a USB stick to boot on the linux BSP release 2.8b4, just after some minor changes into the live U-boot env vars to deal with ‘zImage’ format of the new kernel (our current BSP 2.7.2 is still using ‘uImage’).

Yeah, we switched from uImage to zImage, when updating to Bsp2.8.

We first decided not to rewrite the internal emmc with the BSP 2.8b4, as we were scared to loose all the precious forensics data …

You should backup this data somewhere else if this is precious. As any storage device, the eMMC can get corrupt and you will also loose data. Further for the moment, there is no stable update procedure. If you flash a new Image, then the partitions on the Flash are remade and the files are installed from scratch.

So if I understand you correct, you are booting from USB stick and you are still having eMMC errors with Bsp 2.8b4. Then best will be really to flash a regular toradex image on the eMMC and then check if the errors are still there.

Best regards, Jaski

fred.rol · November 14, 2018, 9:41am

hi Jaski,

I’m a little embarrassed : I took this emmc read “problem” on behalf of someone else and I’m not cleared to write something into the flash memory that could make the consequences of the “problem” disappear… and consequently loose information of WHY it has happened (a misuse of ext4, an hardware damage, a design problem …). They would prefer return the iMX6 module back to Toradex for analysis.

Do I have to contact someone else in particular ?

Regards,
Fred.

jaski.tx · November 14, 2018, 1:55pm

hi fred.rol

I am sorry, it was not my intention to embarrass you. You are right since some sectors seems to be corrupted, you should file up a RMA. We are sorry that you had this experience with our Products.

Further for future, you can always check the Health Status using the mmc-utils as described here. Please send us also the report of the Health Status of eMMC Flash when you file up the RMA.

Thanks and best regards, Jaski

fred.rol · November 15, 2018, 10:13am

Thanks a lot, Jaski
We will launch the RMA procedure.
BTW, as you proposed I built the mmc-utils (the one from OE and the one from toradex-demos) and ran them on the faulty card.
I’m not sure I got the “Health Status” you will need…
Here are the logs :
mmc-utils logfiles

Thank you very much for your help Jaski and the whole time you spent !
Best Regards,
Fred.

jaski.tx · November 15, 2018, 10:22am

You are welcome.

Thanks for the files and the feedback.