Getting USB mass storage to work on T20 using USB on-the-go

jrjab · December 4, 2017, 9:38am

No, our updater.exe is just a stub that checks if an update is available for our application and if so, does an update. If not, it launches our application. So in it’s most basic form it’s a ShellExecute of our application.

I’m not sure what you mean by “Are you also able to see the files diapering and appearing in different folders in our image?”. What files or folders are you refering to? The ones in eg the Windows folder?

And how can I verify that it’s TFAT? I checked the registry but I’m not sure what to check. I’ll include our generated reginit.ini here for you to check: link text

samuel.tx · December 5, 2017, 3:45pm

You wrote in a previous message “but I still have the problem that, all of a sudden, directories turn up empty…or other directories suddenly have files that resided in another directory before.” . So my point was if this may could be related to a difference in your BSP.

And how can I verify that it’s TFAT?
I shared a little tool with you to check how the partition is formatted. According to your reginit.ini you should already use TFAT.

I reviewed your reginit.ini: Please compare once the following keys with our default registry, there are some differneces:

[HKEY_LOCAL_MACHINE\Init\BootVars] : “MasterKeysInRegistry”=dword:1 (Has an impact on loading registry and loading ecrypted keys. Please do not touch this.)
[HKEY_LOCAL_MACHINE\System\StorageManager\Profiles\FlashDisk\FATFS] (There are two different keys).

jrjab · December 6, 2017, 1:02pm

I ran the TFATCheck tool and it reports back with the message that it is in face TFAT!

To clarify, we’re on a 2.0 CE6 image.

I checked the registry keys. What I see is this:

*the project.reg (the one I got from your site for CE6) sets MasterKeysInRegistry on 1.
*the platform.reg then removes the entire [HKEY_LOCAL_MACHINE\Init\BootVars] and does not set the MasterKeysInRegistry anymore. In platform.reg it is set in comments:
;“MasterKeysInRegistry”=dword:1 ; On CE6 we could store masterkeys in registry, but CE7 does not support this anymore. NOTE: Does not seem to work on CE6 either!

So basically, this MasterKeysInRegistry is NOT set at all in our image. We’re not changing anything ourselves to this registry key, so if there is an error it must be in project or platform.reg.

The [HKEY_LOCAL_MACHINE\System\StorageManager\Profiles\FlashDisk\FATFS] has the exact content as stated in platform.reg. Again, we’re not changing these keys. This is what is in platform.reg.

[HKEY_LOCAL_MACHINE\System\StorageManager\Profiles\FlashDisk\FATFS]
“EnableCache”=dword:0
“FormatExfat”=dword:0
“FormatTfat”=dword:1
“TransactData”=dword:1
“CheckForFormat”=dword:1

What settings should we have for MasterKeysInRegistry and
[HKEY_LOCAL_MACHINE\System\StorageManager\Profiles\FlashDisk\FATFS]??

The phenomenon is that, after a few resets, Flashdisk seems to lose (some of) its content. Directories suddenly show no more files or files seem to have moved in a different directory. It looks like the filetables are messed up…

samuel.tx · December 7, 2017, 8:21am

Hello @jrjab

Before doing further check on the registry details please check as already mentioned earlier, if you see the same with our standard image (get it from here).

If this also happens there, it looks as if this issue is a due to a broken module. If not, there must be a difference in the BSP.

jrjab · January 29, 2018, 10:53am

Hello Samuel,

I’m returning to this issue. I have done many many tests and have finally found a way to consistently have the problem on YOUR CE6 image for T20. So I used Toradex Ce6 image to reproduce the problem. Below you will find the steps I did. As long as I don’t create files on the FlashDisk on the device itself (eg, an application that creates files on the FlashDisk), it seems ok. So just copying files and not changing anything on the device side, seems to work. I made a test program that does nothing else than create 5 simple txt files on the root of the FlashDisk. (CreateFile - WriteFile - CloseHandle). If I use this program, things get messed up sooner or later. Can you please check this on your side and hopefully find a solution? I have lost a lot of time figuring this out as I thought the problem was in our image.

Code for this simple testtool is below.

These are the steps:

set Toradex image to mass storage
connect to pc using USB cable
copy 150MB data (this data is structured in several directories and subdirectories) on E:\ (the drive that is linked to the device)
several reboots of device and check files on E:\ drive - all ok
new reboot
start toradextest.exe - 5 files are created
connect to pc
check files on E:\ drive - all ok
disconnect
reboot
connect to pc
check files on E:\ drive - all ok
disconnect
start toradextest.exe - 5 files are re-created
connect to pc
check files on E:\ drive - messed up directories

Code testtool:

#include <windows.h>



int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPTSTR lpCmdLine, int nCmdShow)
{
	TCHAR szFileName[128];

	for (int i=0; i < 5; i++)
	{
		_stprintf(szFileName,TEXT("\\FlashDisk\\tstfilelng%d.txt"),i);

		HANDLE theFile = CreateFile( szFileName, GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ, 0, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL );
		if( theFile!= INVALID_HANDLE_VALUE)
		{
			char szSample[128];
			strcpy(szSample,"voorbeeldje");
			DWORD theDummy;
			WriteFile( theFile, szSample, strlen(szSample)*sizeof(char), &theDummy,NULL );
			CloseHandle( theFile );
		}
	}
	
	return 0;
}

samuel.tx · February 1, 2018, 7:08am

@jrjab: Thanks for the report. I will further investigate this issue and will get back to you.

samuel.tx · February 1, 2018, 8:12am

@jrjab: We tested the scenario on different ways but have not been able to reproduce this issue:

Some details about our setup, may be we can reproduce it if we do all the same as you did.

We setup the devices (Once tested with a Colibri T20 256 MB and once with a Colibri T30 1GB) using NVFlash.
We flashed an image 2.0 and also tried with 2.1b4 on both
We copied around 160MB to the device. 13 Elements have been folders with subfolders and 51 have been files in the root directory.
We run the test as you described. The device we connected to was a Windows 10 device.

Some questions:

On the tests above did you see some issues on the debug output?
Could you share a print screen of the “broken” flashdisk?
In case it is depending on the kind of data that is on the flashdisk you could also put your data once to share.toradex.com so we could test with your data. If you don’t want to share this in the public, send us a mail to support@toradex.com.

jrjab · February 1, 2018, 9:59am

I have been using a Colibri T20 512MB and flashed it with YOUR image 2.0 (CE6).
I did not see anything on debug output
I don’t think it’s dependent on the data. I have tried with different data and the results were the same.
I’m testing on a Windows 7 machine but I highly doubt that would matter…
What mostly happens is that, after it happens, certain directories are empty where originally there were several files and directories. Other directories suddenly have content that originally resided in a different directory. It seems to me that the filetable gets messed up somehow. It’s as if writing on the flashdisk can mess up the filetables (if it was connected to the PC before). I don’t see incorrect file or directory names. It’s now mostly files disappearing or jumping directories.
What I did to test this is programmatically compare the original content with the content on the removable drive so it will tell me immediately if some files are missing. (I did this with a simple batch file)
Should I share my test tool? Maybe you need to run it several times to see if you see the same behavior.

jrjab · February 1, 2018, 10:51am

Hi Samuel,

I can reproduce the problem as follows (less steps):

set Toradex image to Mass storage and Mass storage DeviceName to DSK1:
connect to PC
copy 150 MB of data (2 directories- 1directory just holds 17 files, other directory holds 2 subfolders and 16 files, the subfolders have 473 file and 16 files respectively)
copy toradextest.exe as well
disconnect from PC
run toradextest.exe on the device (so, the exe is located on FlashDisk)
connect to PC
compare files: all ok
disconnect from PC
run toradextest.exe
connect to PC
compare files: the directory with 2 subfolders is completely empty now

Nothing specific on debug output.

I will send a link to support@toradex.com with my test directories, with the batch I use to do the comparison and with the toradextest.exe…

samuel.tx · February 2, 2018, 2:06pm

Hello @jrjab,

We have been able to reproduce this issue. Thank you for the nice step by step guidance.

Issue

After investigating this we are pretty sure the issue is related to the disk formatting. The FlashDisk is formatted with TexFAT as you already know. On the Desktop PC side, Windows does not support TexFAT but only ExFAT. Basically TFAT or TexFAT has two synchronized FAT tables, where as on the Desktop only one of those is getting used. When switching back and forth the two tables some get messed up and some files are getting lost.

Solution

One option would be to format the FlashDisk with exFAT. I quickly tried this and this resolved the issue. However I would not recommend to do this as you would also loose the transaction safe features of TFAT. The other option would be to enable a RAM Disk. I have prepared for you a reg file, that you can import to enable the RAM Disk. Import this registry change and modify the masstorage driver to mount DKS3:. Save the registry andt the next time you boot you will have a RAM Disk there. Instead of sharing the flashdisk you share now the RAM disk.

Of course this has somehow a different behavior as before. If you need this to upgrade your application you will have to listen to a event that tells you when the RAM Disk gets mounted and then copy the stuff to your flashdisk (RequestDeviceNotifications) or even easier just copy everything into an AutoCopy folder into the RAMDisk, so that the files are copied automatically to the right place once the RamDisk gets mounted on the device.
On the other side the RAMDisk has the advantage that you are sure that the user has no direct access to the FlashDisk and everything that is copied there is drooped after a device reboot.

Please let us know if you need further assistance on this solution.

jrjab · February 5, 2018, 9:31am

Hello Samuel,

I’m glad you were able to reproduce the issue. I have several additional questions to your proposed solution:

-How large is the capacity of the RAMDISK? I suppose it’s just as big as RAM allows? So, in theory a maximum of 512MB on my module? And of course, in practice this would be at most 450MB or so? (the problem in our case is that we use the mass storage solution in production to program our device. The data would be typically more than 500MB).

-On another device we have exposed the MMC Card. Now, the MMC Card is not formatted as TFAT. If I get it right, this would mean that a sudden power loss while writing to the MMC Card could corrupt all data on the MMC Card? Right?

-What is the biggest danger of formatting the FlashDisk as exFat instead of TFAT? Power loss corrupts file tables? (and would that be only when writing?)

-Is there by any chance another more permanent solution that could be found to solve this problem? (eg having TFAT on FlashDisk and exposing FlashDisk safely as mass storage?)

jrjab · February 5, 2018, 3:10pm

Also, while looking into this further, I was wondering if it would be an option to format FlashDisk as FAT32? (by setting FormatTFat and FormatExFat in the registry HKLM/System/StorageManager/Profiles/XYZ/FATFS to 0).

FAT32 has 2 FAT tables, so there is some recovery if a FAT becomes toast, exFAT only has ONE FAT table, and you don’t get recovery unless you use Transactional ExFAT and then you get 2 FAT tables (but Windows seems to handle this wrong). The downside of FAT32 (max file size 4GB, only 65K directories, …) are not an issue for us. I feel like FAT32 would be the way to go: faster saving of files (up to 3 times as fast apparently), and better recovery because two file tables (and I suppose WIndows has better support for FAT32 file tables than TFAT file tables).

So, I guess my question would be: is FAT32 an option for FlashDisk? And is my reasoning correct?

jrjab · February 8, 2018, 10:01am

Hi again,

I return to this question.

Can I format FlashDisk as FAT32 (by setting FormatTFat and FormatExFat in the registry HKLM/System/StorageManager/Profiles/XYZ/FATFS to 0)?

If so, am I correct to say that FAT32 is a safer option than exfat (as far as I can tell from the info I found online: specifically one of the comments in the article here: FAT32 vs. exFAT: What's the Difference and Which One's Better?)?

TexFat and RamDisk are two solutions that won’t work for our application…

samuel.tx · February 15, 2018, 9:41am

Size of Ramdisk

You are right, the max size would be less the the RAM size of the specific module.

MMC not TFAT formated

Potentially it is true, that a non TFAT formated MMC card could be corrupted on a sudden power loss. May be your MMC card already has kind of a protection agains power loss (industrial graded cards often have that). But it the underlaying MMC hardware does not know about the Filesystem. It could happen, that may be a file gets written but the FAT Table does not get updated or the other way arround. So you could lose some files here of course.

Dankger on formating FlashDisk with FAT

You potentially could see the same effects as above.

Other solutions

There are may be some troublesome workarounds like move the data from the RAM disk to the flashdisk once the disk is full. One other option of course is active sync or FTP over RNDIS or native network device?

samuel.tx · February 15, 2018, 9:59am

Is FAT32 really having 2 redundant FAT Table? This would be new to me. May be using TFAT instead of TexFAT could change the situation, but this would have to be tested in detail. And yes, you could modify this in the registry.

samuel.tx · February 15, 2018, 10:10am

Hello @jrjab

As written in the previous post: The information that FAT32 has two fail tables that should help agains power loss is new to me. If this would be the case, yes you could change this in the registry. But you would have to reformat the disk at least once. If you use our standard image and boot the device it will be formated with TexFAT. If you change that after, I am not sure what happens. If you formated the disk once after changing this registry setting, you can be sure the change has applied.

hotmaillogin · May 19, 2018, 3:13am

Thanks for sharing these useful information! This is really interesting information for me.
hotmaillogin