Chances are high that the failure is related to the eMMC chip itself (either the physical flash, or the built-in eMMC firmware).
We used 3 different eMMC types on the modules, which can be distinguished by the manufacturer’s label:
- V1.1a : Toshiba / Kingston
- V1.1b : Micron
It would be interesting to see whether the problem is evenly spread between all eMMC types.
Your application seems to use the flash quite intensively. This might possibly lead to wear-out of the flash. I didn’t check the precise datasheets, and I can’t know how exactly the eMMC firmwares behave, but please have a look at the very rough estimations below:
- Writing data to the flash (no matter how small they are) lead to a write of several pages (the data itself, file allocation tables, lower layer NAND mapping tables). Let’s assume it is 5 pages in average.
- A flash page can bare 1000 write cycles
- A page is 8kB in size
- Let’s assume there is almost no static data on your flash, so all 4GB can be used for dynamically written data.
- For the calculation, I assume your application writes once every second
How long can your application run before the flash’s lifecycle is over?
- In total you are allowed to write 1000 x 4GB = 4TB
- Each write consumes 5 x 8kB = 40kB
- ==> You can do 4TB/40kB = 100 million writes.
- One year is 32 million seconds
- ==> With 1 write every second your flash would last for about (100 mio / 32mio) = 3 years
However, there are effects which are not taken into account:
- The calculation assumes that your devices runs 24/7
- There are additional maintenance operations which the eMMC firmware probably does, such as static / dynamic wear leveling, defragmentation etc.
- Writes which occur shortly after each other are cached and can be combined.
Again: all the assumptions above are very rough. I just wanted to show you that you might get in a critically high use of the flash.
What to try next
There are additional effects which could degrade the flash content, for example writes to adjacent cells (which probably also happens more often in your use case compared to our average customers).
If your modules are working fine again after re-flashing them, you might consider to update the bootloader, OS, config block and registry of your modules on a regular base, in order to refresh the flash contents.