I want to quickly delete a directory containing a large number of files

Technical Support
, , , ,
1

Apalis T30 + WEC7 + .NET CF 3.5

An application that starts up at all times creates a directory for each date in the USB memory and stores the image files captured by the camera.
Another application, which is started only once at power on, deletes the directory older than one month among the directories for each date stored in the USB memory.

However, the deletion process takes a lot of time.
In addition, during the deletion process, it also affects the operation of the application that is always activated, the display is delayed, and the process of storing in USB memory takes time.

So we are thinking about reviewing the deletion process.

Currently, it is deleted by Directory.Delete (dirPath, True).

Is there any other way to make the deletion process more efficient?

We get the information that it will be faster if we use rmdir /s /q dirPath, but is there a RMDIR command in WEC7?

2

Dear @kyas

I actually never did any performance measurements for deleting files.

There is no rmdir command in WEC7, but you can use the same parameters with the delcommand:

del /s /q dirPath

Another option is to write the file delete function in a separate application in native C. I don’t know if this improves the deletion speed significantly, but at least it should be able to run in parallel to your main application and thus avoid the impact on the overall application performance.

Regards, Andy

3

HI,
you can use the double commander s\w

4

Dear @andy.tx

Thank you for your reply.

I tested using the DEL command but stopped it because it was even slower.
So I changed it to delete using WinAPI’s DeleteFile and RemoveDirectory.

I delete about 20 files in one folder in order, and finally delete the folder itself.
Because there are a large number of folders, repeat the same process.

Normally, it takes less than 20 ms to delete a single file.
However, if you continue processing continuously, it may take more than 5000ms to delete a single file.
After deleting three or four files, it takes more than 5000ms and can be deleted in less than 20ms.
It was found that this state occurs periodically for about one minute.

I think this is also a problem with USB memory controllers, but from another point of view, may it be cache settings for accessing USB or problems with USB drivers? I also think.

Are there any causes and approaches to solving these symptoms?

5

Dear @kyas

I’m afraid I cannot reproduce the problem. I was using the following code to measure deletion speed:

int _tmain(int argc, _TCHAR* argv[])
{
    HANDLE hFile;
    WIN32_FIND_DATA fd;
    WCHAR path[] = L"\\USB HD\\kill\\";
    WCHAR allFiles[] = L"*.*";

    WCHAR findFilter[256];
    WCHAR fullPath[256];
    DWORD t;

    swprintf_s(findFilter, sizeof(findFilter), L"%s%s", path, allFiles);

	hFile = FindFirstFile(findFilter, &fd);
    if (hFile != INVALID_HANDLE_VALUE)
    {
        do
        {
            swprintf_s(fullPath, sizeof(fullPath), L"%s%s", path, fd.cFileName);
            NKDbgPrintfW(fullPath);
           
            t = GetTickCount();
            ASSERT( DeleteFile(fullPath) );
            NKDbgPrintfW(L" (%d ms)\r\n", GetTickCount() - t);

            if (!FindNextFile(hFile, &fd))
                NKDbgPrintfW(L"-- Done --\r\n\r\n");
           
        } while (GetLastError() != ERROR_NO_MORE_FILES);
        ASSERT( FindClose(hFile) );
    }
}

The folder \USB HD\Kill\ contained 100 files, I measured deletion times of 5-436 ms (average 25ms) for each file. This time was almost independent of the file size. I tested with file sizes of 8Bytes, 256kB and 1MB.
You can refer to the attached log for more details.

Regards,
Andy

6

Dear @andy.tx

Thank you for doing the operation test.
I also tried the operation test in more detail using the source code I received.
I will attach a log.

log

Then I write down what I found through the test.

(1) Difference by area of ​​partition

The USB memory I am using this time has a capacity of 128GB, but the problem did not occur if the same test was performed after securing only 32GB of partition space.
The problem occurs if the partition area is about 64GB or more.

(2) Difference by USB memory

I tried using another manufacturer’s 1GB and 32GB USB memory, but the problem did not occur.

From this, it was found that the problem that is unique to the USB memory used this time is highly likely.

However, with the USB memory used this time, is there anything I can do to solve it?
Of course, we are considering using a USB memory from another manufacturer.

7

You could delete the files in a background thread with low priority.
Delete each file separately like in the code from andy.tx, but add a Sleep(x) after each DeleteFile() call so that your main application is not affected by the deletion thread.
Deletion of a directory will take longer, but this shouldn’t matter as long as you are deleting files faster than you are generating new ones.

8

Dear @kyas

After your measurements I highly suspect that the delay is depending on the USB drive’s firmware. Deleting a file on a drive can basically trigger any kind of cleanup / defrag operation inside the drive.

There’s only a few smaller manufacturers of USB drives which would give you any information about what’s going on in their firmware, one of them could be Hyperstone. If you are willing to spend additional money for more expensive drives, and possible engineering hours for them to analyze your use case on their drives, this could be a good option.

If you want to stay with lower cost drives, I would recommend to split them in multiple smaller partitions, if applicable. The hope is that a smaller drive is easier to clean up and therefore uses less time for this maintenance operation.

And of course @haide’s suggestion to use a background thread is a good solution if it fits your requirements.

Regards, Andy

9

Dear @haide
Dear @andy.tx

Thank you for your reply.
It will be very helpful.

Because there is a cost issue, changing USB memory is considered as a last resort.

I tried to reduce the priority of the proposed thread and put it to sleep, but there was no change in locking the UI.

So, I’m thinking about trying it out by dividing the partition size, but I have some questions.

(1) Can we read extended partitions with WEC7?
I divided the 128GB USB memory into 10GB, 32GB, 32GB, 32GB, but the last partition is not displayed.
Only the last partition is an extended partition, the others are primary partitions.

(2) Can you change the partition from the command or program?
If you can do by command, when upgrading our product, it can be handled without replacing the USB memory.

(3) Can you format the drive from a command or program?
In addition to the method of deleting files one by one with DeleteFile(), I thought that it would be faster if formatting was done for each divided partition.

Added questions to 4/3.

(4) In the Apalis T30 + Iris + WEC7 environment, I recognized that the USB 3.0 port of Iris can not be used. Is this correct?

(5) Although this is an alternative method, can you use storage connected to mSATA instead of USB memory?

Added questions to 4/4.

(6) State of being divided into a plurality of partitions.
I want to format the partition instead of deleting files and folders sequentially, but when I check cmd.exe help, I can not find the formatting command.
Is it possible to format from cmd.exe or .NET application?

10

Another option could be writing those many files to an uncompressed zip file instead of a directory. If the data has a good compression ratio writing a compressed zip file could be even faster when you can spend the cpu cycles.
But the deletion should be much faster because you are deleting just a single zip file.

11

Dear @kyas,

(1) Can we read extended partitions
with WEC7? I divided the 128GB USB
memory into 10GB, 32GB, 32GB, 32GB,
but the last partition is not
displayed. Only the last partition is
an extended partition, the others are
primary partitions.

Quickly I formatted USB pen driver with primary and extended partition and that is mounting on WEC7 using minitool partition wizard. But I have yet to understand this topic better and I would like to get back you.

(2) Can you change the partition from
the command or program? If you can do
by command, when upgrading our
product, it can be handled without
replacing the USB memory.

Yes, this can be done through Storage manger APIs, refer here :

(3) Can you format the drive from a
command or program? In addition to the
method of deleting files one by one
with DeleteFile(), I thought that it
would be faster if formatting was done
for each divided partition.

Please refer below links and sample code attached here

or

Please use Storage manager library APIs(\wincelib_bin_V2_0bis_2868\inc\StoreMgrLib.h) from our libraries

or

Use Windows Embedded Compact Storage Manager APIs(Storage Manager Functions (Compact 2013)) to format SD card

Added questions to 4/3.

(4) In the Apalis T30 + Iris +
WEC7environment, I recognized that the
USB 3.0 port of Iris can not be used.
Is this correct?

USB 3.0 stack not supported by WEC7 hence it won’t work.

(5) Although this is an alternative
method, can you use storage connected
to mSATA instead of USB memory?

SATA interface not supported by our WinCE BSP. Please refer here
Added questions to 4/4.

(6) State of being divided into a
plurality of partitions. I want to
format the partition instead of
deleting files and folders
sequentially, but when I check cmd.exe
help, I can not find the formatting
command. Is it possible to format from
cmd.exe or .NET application?

Please check answer for 3rd question.

12

Dear @haide Dear @raja.tx

Thank you for your reply. It will be very helpful.

As we proceeded with this survey, we found that not only deletion of files on USB memory, but also copying of files (as a matter of course can be considered) causes the same phenomenon.
It does not make sense to speed up deletion of files using drive format.

So I would like to ask some more questions.

This is an issue with an application that runs on .NET Compact Framework 3.5.
When copying files to USB memory or deleting files, there is time for the UI to freeze.
Of course, file processing is executed in a dedicated thread.
The UI freezes regardless of whether the thread’s priority is Highest, Normal or Lowest.

The time taken for file processing is usually about 1000 ms to 2000 ms.
However, if you process files regularly, it may take a long time.
It may take 20000ms to 50000ms.

Just as with this processing time, the UI doesn’t freeze.
The UI can only be measured with the appearance of the clock displayed on the screen, but it has been confirmed that the movement of the clock has stopped for about 3 to 15 seconds.

As a question, in principle, while file processing is executed by a dedicated thread, does it affect the operation of the UI?
Or is it possible that my program has a problem and is not properly processing it?

I am very sorry for repeating the long and ugly questions, but thank you for helping me solve the problem.

13

Dear @kyas,

Thank you for your patience.

It is very difficult to suggest a specific answer without knowing much details about the issue. These kinds of issue can be debugged using kernel tracker, please refer this forum question for more information : https://www.toradex.com/community/questions/35633/diagnose-a-cpu-load-on-wince.html

Maybe the problem related to .NET 3.5 doesn’t support multi-core execution., please refer this question : https://www.toradex.com/community/questions/21936/smp-support-in-wec7.html for more information and you may try to execute CPU hogging task on a particluar processor and leave other processors for other tasks. Please refer to this documentation : https://docs.microsoft.com/en-us/previous-versions/windows/embedded/ee488202%28v=winembedded.70%29 for API details and examples here.

Please let us know if you have any other questions.