emmc high erase counts

Dear Support team,

we have an issue with a long running test using an Colibri iMX8 regarding high erase counts for specific regions of the emmc.
The issue was detected after some files were corrupted.

I found the lifetime is close to 100% (in another device we reached about 120% and detected similar issues)

# mmc extcsd read /dev/mmcblk0 | grep "life" -C 5
[VENDOR_PROPRIETARY_HEALTH_REPORT[274]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[273]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[272]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[271]]: 0x00
[VENDOR_PROPRIETARY_HEALTH_REPORT[270]]: 0x00
Device life time estimation type B [DEVICE_LIFE_TIME_EST_TYP_B: 0x02]
 i.e. 10% - 20% device life time used
Device life time estimation type A [DEVICE_LIFE_TIME_EST_TYP_A: 0x09]
 **i.e. 80% - 90% device life time used**
Pre EOL information [PRE_EOL_INFO: 0x01]
 i.e. Normal
Optimal read size [OPTIMAL_READ_SIZE: 0x01]
Optimal write size [OPTIMAL_WRITE_SIZE: 0x04]
Optimal trim unit size [OPTIMAL_TRIM_UNIT_SIZE: 0x01]

Also I used emmcparm from micron to get further information and found out that some blocks have around 9000 erase counts where others have only 2600.

  • Is there an issue with not correctly behaving wear leveling (not enabled? / how to check?)
  • How can I find out which files / parts of the filesystem are stored at these blocks to get a clue why they have been written so frequently?
root@colibri-imx8x-07202839:/opt# ./emmcparm_arm_64bit --erase-count
./emmcparm_arm_64bit: unrecognized option '--erase-count'
root@colibri-imx8x-07202839:/opt# ./emmcparm_arm_64bit --erase_count
emmcparm v6.2
Device file: /dev/mmcblk0
EXT_CSD revision [192]: 1.7 (for MMC v5.0)
Product name (PNM): Q2J54A

Erase Count                       Min   |         Max   |         Avg
Global:                             1   |        8952   |        2842
Enhanced area (SLC):                1   |        2678   |        2538
Normal area (MLC):                  1   |        8952   |        7630

root@colibri-imx8x-07202839:/opt# ./emmcparm_arm_64bit --bad_block
emmcparm v6.2
Device file: /dev/mmcblk0
EXT_CSD revision [192]: 1.7 (for MMC v5.0)
Product name (PNM): Q2J54A

Feature name: Bad Block

Total initial bad block count: 0
Total later bad block count  : 0

root@colibri-imx8x-07202839:/opt# ./emmcparm_arm_64bit --sect_count
emmcparm v6.2
Device file: /dev/mmcblk0
EXT_CSD revision [192]: 1.7 (for MMC v5.0)
Product name (PNM): Q2J54A

Feature name: Block Erase Count
Block#  Erase#
52      8952
88      8949
42      27
77      27
87      2
57      8952
95      8952
89      8952
82      2
834     2615
83      2
227     2430
46      8934
90      2
55      8951
51      8952
79      8946
69      8952
96      8952
61      8952
601     2406
593     2406
704     2416
556     2405
558     2416
372     2416
256     2417
586     2404
724     2427
804     2428
563     2420
417     2440
110     2441
535     2425
245     2434
378     2436
1030    2438
312     2428
273     2430
131     2423
448     2439
517     2451
773     2454
388     2435
266     2427
291     2434
250     2430
327     2434
299     2448
796     2432
440     2434
429     2420
293     2412
408     2410
404     2419
201     2429
881     2439
998     2439
468     2423
134     2444
150     2439
286     2456
261     2457
373     2441
1002    2455
871     2455
977     2450
394     2454
741     2445
894     2436
233     2435
990     2448
697     2437
1063    2431
500     2431
600     2423
164     2423
220     2426
774     2421
342     2436
615     2427
232     2430
978     2440
470     2441
407     2435
732     2444
126     2454
287     2451
625     2436
922     2438
453     2426
124     2433
579     2436
338     2444
727     2436
915     2450
612     2432
515     2439
1054    2440
216     2430
680     2440
561     2432
722     2444
646     2451
1062    2452
961     2460
173     2461
311     2452
138     2457
170     2462
113     2449
146     2452
147     2467
102     2451
889     2474
114     2475
278     2469
1032    2474
341     2474
656     2450
636     2442
258     2454
736     2453
191     2448
664     2444
239     2455
382     2457
984     2444
264     2464
493     2465
465     2462
941     2462
283     2466
439     2454
944     2445
836     2465
161     2470
935     2470
776     2470
466     2470
334     2469
157     2459
938     2459
363     2461
422     2459
129     2461
620     2454
759     2466
380     2467
304     2466
609     2419
545     2460
721     2446
1023    2462
913     2451
818     2471
911     2472
958     2473
931     2463
1050    2478
513     2471
703     2477
802     2487
1008    2466
214     2470
761     2468
132     2468
787     2478
933     2485
140     2462
479     2471
177     2473
494     2471
763     2478
221     2483
442     2484
1035    2486
1027    2487
728     2487
238     2485
725     2490
651     2487
679     2491
1043    2490
392     2487
638     2496
436     2487
1013    2492
415     2494
499     2497
623     2489
427     2478
744     2478
681     2481
873     2482
857     2485
564     2479
670     2488
882     2474
143     2479
194     2475
929     2467
149     2482
581     2505
401     2508
520     2516
860     2488
224     2493
446     2516
812     2496
162     2505
562     2505
766     2506
969     2486
409     2469
306     2489
980     2500
503     2493
1042    2503
672     2489
475     2496
852     2497
1015    2497
154     2462
966     2508
735     2456
634     2469
462     2464
457     2502
838     2501
764     2417
618     2496
999     2495
269     2503
717     2502
366     2494
537     2509
904     2515
885     2519
210     2520
690     2493
767     2495
810     2495
308     2516
716     2434
992     2524
707     2525
1058    2511
280     2532
762     2524
748     2455
570     2523
912     2466
979     2476
310     2534
497     2479
989     2534
368     2531
498     2526
592     2501
906     2492
897     2456
808     2534
142     2516
662     2532
799     2508
483     2509
840     2521
649     2524
459     2525
295     2532
673     2507
330     2506
383     2524
887     2529
940     2517
598     2521
249     2504
951     2438
839     2530
469     2494
778     2532
878     2529
691     2536
695     2555
490     2555
219     2542
874     2538
1034    2531
1003    2491
729     2539
702     2529
292     2530
524     2487
1007    2497
353     2523
352     2461
653     2540
526     2528
698     2540
790     2544
166     2541
505     2543
360     2557
758     2565
988     2549
666     2561
737     2543
696     2545
658     2559
181     2543
485     2550
986     2560
428     2546
144     2552
397     2542
1061    2553
688     2551
797     2528
867     2547
1005    2550
976     2533
869     2540
1039    2548
223     2547
301     2532
303     2530
321     2431
228     2456
481     2446
467     2551
813     2540
947     2537
182     2540
892     2543
798     2545
648     2539
1004    2546
964     2539
252     2539
268     2528
831     2528
251     2428
925     2541
671     2548
805     2501
188     2517
1044    2530
760     2549
884     2555
347     2531
156     2568
611     2514
305     2547
546     2522
864     2543
1020    2555
888     2562
927     2562
215     2543
956     2537
385     2543
827     2563
473     2537
137     2537
180     2551
324     2551
1018    2533
313     2553
845     2540
829     2544
738     2538
329     2552
333     2567
807     2427
315     2564
398     2535
854     2558
825     2557
905     2542
769     2535
1047    2536
108     2541
632     2542
1006    2533
172     2537
496     2548
202     2561
782     2562
753     2548
982     2546
675     2546
326     2554
606     2555
879     2542
997     2573
484     2559
525     2556
554     2558
712     2556
1040    2551
530     2530
617     2560
814     2564
851     2560
1026    2507
595     2552
425     2531
908     2547
514     2553
424     2556
544     2496
640     2565
510     2569
125     2516
743     2554
285     2545
960     2454
275     2569
578     2572
179     2573
868     2568
423     2552
346     2543
253     2568
880     2564
1024    2550
536     2519
552     2552
693     2557
501     2562
923     2567
430     2565
122     2551
928     2546
487     2441
478     2541
198     2557
816     2560
953     2565
207     2578
111     2591
1036    2587
507     2557
669     2533
320     2533
290     2537
654     2498
145     2548
597     2549
559     2546
584     2545
340     2560
112     2561
786     2566
1001    2571
955     2562
272     2576
740     2472
265     2498
699     2523
824     2536
613     2445
351     2582
298     2582
551     2580
588     2524
367     2580
185     2573
683     2585
859     2585
932     2572
700     2584
445     2585
1041    2587
637     2554
1014    2569
784     2585
959     2583
1022    2565
846     2573
104     2573
921     2569
375     2576
502     2581
850     2589
441     2579
849     2583
176     2583
650     2583
713     2579
538     2590
359     2489
495     2590
187     2480
370     2590
231     2592
454     2476
734     2576
954     2430
573     2584
274     2589
677     2596
254     2584
386     2591
848     2597
619     2587
792     2588
872     2589
821     2576
212     2559
577     2567
987     2575
543     2453
328     2573
1033    2575
504     2577
795     2577
815     2579
621     2596
532     2598
374     2574
571     2458
781     2519
898     2567
186     2439
1016    2471
348     2439
247     2604
1012    2559
631     2562
115     2568
153     2576
452     2553
297     2554
755     2576
463     2582
949     2569
1025    2580
336     2590
946     2583
723     2589
572     2589
685     2571
155     2591
720     2592
518     2592
862     2563
747     2419
460     2585
432     2586
389     2587
317     2588
222     2585
847     2594
434     2595
745     2597
136     2599
770     2603
416     2609
203     2609
876     2603
169     2610
883     2606
464     2615
674     2621
948     2631
371     2631
919     2611
565     2611
523     2602
241     2613
865     2614
418     2601
963     2598
920     2595
800     2609
433     2601
785     2579
337     2546
458     2556
772     2595
830     2599
509     2589
714     2609
557     2526
886     2595
508     2581
910     2600
175     2600
875     2599
403     2602
701     2603
511     2595
165     2475
891     2602
844     2600
568     2600
899     2601
726     2606
120     2609
550     2579
116     2609
455     2603
451     2612
692     2621
820     2623
668     2623
107     2635
877     2617
208     2612
319     2636
709     2633
841     2640
534     2628
817     2627
1052    2595
171     2614
607     2590
1048    2604
235     2641
657     2558
527     2619
754     2548
396     2615
1049    2579
395     2594
302     2602
610     2602
248     2591
384     2601
694     2604
279     2441
362     2548
602     2619
270     2601
257     2613
343     2616
189     2619
431     2605
779     2620
687     2613
118     2624
823     2625
706     2620
281     2597
531     2612
939     2599
1053    2608
902     2623
344     2624
916     2605
803     2611
981     2618
192     2626
242     2626
318     2632
907     2603
605     2626
213     2635
411     2620
930     2623
350     2547
684     2498
356     2631
863     2626
555     2639
765     2639
832     2461
934     2573
861     2576
676     2641
775     2642
234     2642
339     2639
826     2649
566     2644
943     2607
419     2649
1011    2616
652     2632
393     2633
580     2601
1038    2600
168     2605
757     2600
1060    2621
553     2613
357     2602
300     2603
660     2583
991     2599
243     2602
381     2598
414     2605
974     2452
890     2614
119     2623
965     2615
159     2630
678     2630
926     2642
569     2646
788     2650
410     2666
390     2627
893     2470
1019    2610
811     2604
1051    2617
183     2624
603     2611
582     2647
477     2641
973     2641
205     2638
204     2642
364     2545
809     2562
361     2525
449     2537
521     2640
663     2639
616     2641
549     2623
349     2624
909     2621
190     2631
896     2628
583     2630
482     2631
211     2627
718     2629
975     2635
639     2635
945     2641
644     2646
294     2646
806     2645
917     2652
438     2653
843     2656
529     2665
924     2665
489     2659
626     2656
1031    2664
614     2665
480     2664
756     2619
777     2638
399     2580
708     2672
199     2482
574     2522
288     2429
267     2663
719     2429
837     2440
447     2667
661     2669
1000    2533
901     2658
1028    2538
866     2603
608     2659
426     2670
325     2595
750     2677
833     2566
822     2453
289     2520
587     2426
413     2636
488     2578
471     2522
1045    2500
271     2490
733     2657
405     2594
365     2525
109     2476
742     2645
665     2577
996     2482
435     2550
596     2620
354     2536
749     2525
542     2420
148     2677
591     2678
218     2590
659     2619
1046    2445
406     2486
167     2506
801     2438
794     2503
391     2647
193     2675
783     2476
645     2587
983     2618
633     2659
870     2595
127     2433
263     2659
993     2597
197     2646
121     2629
456     2475
1029    2646
512     2469
355     2642
630     2643
139     2597
994     2476
585     2484
1057    2644
151     2459
332     2588
895     2456
400     2564
412     2477
667     2590
402     2520
950     2461
237     2677
345     2552
635     2447
682     2429
985     2570
276     2450
541     2438
624     2630
226     2605
918     2588
594     2645
376     2510
307     2538
768     2645
771     2646
533     2519
195     2673
262     2518
421     2655
474     2432
106     2563
643     2638
519     2597
117     2656
103     2642
942     2639
1059    2497
229     2524
105     2626
705     2550
540     2611
1055    2646
358     2532
590     2444
420     2544
225     2569
547     2613
952     2629
842     2451
196     2630
819     2465
486     2645
133     2494
900     2623
903     2638
331     2504
604     2428
715     2634
472     2645
461     2646
236     2640
443     2640
627     2566
711     2534
476     2650
246     2614
730     2637
971     2589
970     2654
492     2423
835     2654
178     2443
128     2576
791     2474
314     2654
1056    2514
450     2657
209     2657
914     2497
642     2655
995     2514
506     2426
746     2662
335     2663
599     2407
731     2430
793     2422
789     2637
629     2622
174     2640
123     2426
560     2534
491     2597
962     2642
141     2593
282     2484
260     2433
316     2532
968     2572
576     2534
160     2650
135     2422
1010    2609
622     2564
322     2542
516     2524
856     2542
296     2617
957     2645
828     2448
230     2446
1021    2444
369     2573
641     2451
710     2553
255     2642
548     2479
200     2472
751     2629
217     2568
444     2572
937     2675
655     2468
387     2412
277     2424
575     2590
184     2566
284     2521
323     2585
853     2533
689     2546
1009    2483
528     2580
158     2513
780     2622
539     2510
686     2464
1017    2605
972     2561
152     2514
647     2465
522     2565
967     2449
309     2490
567     2536
752     2574
379     2618
855     2641
130     2647
240     2644
628     2648
259     2523
244     2614
1037    2605
206     2463
163     2525
67      8951
49      8951
62      8951
58      8951
81      8951
59      8951
41      8951
93      8951
63      8951
92      8951
64      8951
65      8951
45      8951
86      8951
94      8952
76      8951
54      8951
68      8951
43      8949
72      8948
53      8952
44      8952
97      8950
48      8952
73      8951
50      8951
85      8952
40      8952
75      8952
71      8952
80      8952
74      8952
56      8952
91      8952
47      8890
66      8952
99      8942
60      8951
70      8952
739     2650
589     2650
936     2506
377     2468
858     2615
101     1
100     1
78      2
84      2
437     2474
98      8952
# tdx-info

Software summary
------------------------------------------------------------
Bootloader:               U-Boot
Kernel version:           5.15.129-6.5.0-devel+git.5ee7c6180bd5 #1 SMP PREEMPT Wed Nov 8 10:43:06 UTC 2023
Kernel command line:      root=PARTUUID=b2ce6e87-02 ro rootwait console=ttyLP3,115200 consoleblank=0 vt.global_cursor_default=0 earlycon video=imxdpufb5:off video=imxdpufb6:off video=imxdpufb7:off
Distro name:              NAME="TDX Wayland with XWayland"
Distro version:           VERSION_ID=6.5.0-devel-20231115212442-build.0
Hostname:                 colibri-imx8x-07202839
------------------------------------------------------------

Hardware info
------------------------------------------------------------
HW model:                 Toradex Colibri iMX8DX on XXX SoM Carrier Board
Toradex version:          0052 V1.0D
Serial number:            07202839
Processor arch:           aarch64
------------------------------------------------------------

Hi, I’m the Toradex AI Assistant. The Toradex Support team will follow up with you shortly!

Did you like my answer?

:+1: :-1:


Hello,

Thank you for reaching out with your concern about high erase counts on your eMMC, which can indeed lead to file corruption. This situation often relates to wear leveling, a process managed by the eMMC’s internal controller to distribute write and erase cycles evenly across the memory.

Wear Leveling

Wear leveling should automatically balance the wear across the memory. If certain blocks are experiencing significantly higher erase counts, it might indicate an issue with this process.

Checking Wear Leveling

To verify if wear leveling is functioning correctly, you can use the mmc-utils tool, which is included in Toradex BSPs. This tool allows you to query the eMMC for health status and configuration details. You can use commands like mmc extcsd read /dev/mmcblk0 to check the device’s health and wear leveling status.

Mapping Files to Blocks

Unfortunately, there is no straightforward method to map specific files to physical blocks on the eMMC. The file system abstracts this layer, and tools to directly correlate files with specific blocks are not typically available in standard Linux distributions. However, you can monitor file system activity to identify which files are frequently accessed or modified, which might give you clues about which files are contributing to high wear.

Managing eMMC Wear

  1. Monitor eMMC Health: Regularly check the eMMC’s health status using mmc-utils to track wear levels and lifetime estimation.
  2. Optimize File System Usage: Consider using file systems optimized for flash memory, such as ext4 with journaling disabled, to reduce write amplification.
  3. Adjust Application Behavior: If possible, modify applications to reduce frequent writes to the eMMC, especially in areas showing high wear.
  4. Enhanced User Area: If supported, configure parts of the eMMC as an Enhanced User Area to improve reliability and performance.

By taking these steps, you can better manage the wear on your eMMC and potentially extend its lifespan.

Best regards,
Toradex AI Assistant

Please, also check if these links can help you solve your problem:

https://community.nxp.com/t5/-/-/m-p/960558

https://developer.toradex.com/software/linux-resources/linux-features/emmc-linux/.

The mmc-utils tool doesn’t provide me information if wear-leveling is enabled or correctly configured.