Hello, we are seeing intermittently poor disk (both ram and sd card )performance on our colibri T30 WEC7 BSP 2.3 devices.
Our scenario is that we are receiving many small payloads, around 220 Bytes from a Lora transceiver and trying to send those out as files via Ftp to one or possibly two servers.
Our devices are mostly connected via modem to the Ftp servers in the field so we try and anticipate communications disruptions, to this end, we developed software that will
take the incoming Lora payloads as files and make a series of copies of the files to time stamped directories
send those files via Ftp to our servers
delete the files in the stamped directories to indicate that they have been sent
Consequently, we have quite a number of reads and writes of small files to the disk going on and sometimes we see hiccups or long times (sometimes on the order of a few minutes to make 100 copies and 50 deletes of 220 files) to move OR read these files.
We can see that this is happening if we if read and write the files from RamDisk OR
SDHC (16 Gig, 1 Partition, exFAT formatted, sector size 512 Bytes ) Card.
One possibility is the high CPU usage of core 1 (out of 4 cores) that is starving the threads necessary to perform the disk IO. Or is it the case that the system will use all 4 cores and be able to avoid this problem? We use quite a few apps that use the compact framework and sometimes they are cpu heavy on core 1
We are under a tight deadline to get these times under a minute in the worst case. Do you have any advice on tracking “hiccups” in IO performance?
One thing I didnt want to qualify , is that RAM Disk is an order of magnitude faster, but we still seem to have some “hiccups” writing to RAM Disk and RAM Disk is not persistent so we wanted to be sure there was not some other way to increase the performance of our writes to SD.
Additionally, is there a way to specify how much memory a ram disk can use (WINCE) ? Can you please clarify the registry key values for the RAM disk on WEC 7? Your documentation seems very sparse
File creating and deletion are relatively expensive by default. Situation became significantly worse when physical media is flash. When you create 220 bytes files you need to write at least 2 new flash blocks (usually 4K each) and re-format 2 old ones. Same is happened during deletion.
How many 220byte payloads your getting per second (or minute) ?
Since you are trying to use RAM disk I assume you have no concerns about data loos in case of power failure. In that case it’s much more efficient just to keep your data in an array inside your app together with a timestamps.
If you need a persistence you can use an SQLlight or another database solution.
Hi Alex, thanks for the answer and the information.
We are writing approx 30 payloads per minute or one every two seconds.
We do have concerns about data loss on failure, some of the payloads might indicate dangerous environmental readings, but we were more concerned about the massive delays in transmitting data and hence we had tried the ObjectStore (I was mistaken I believe, because I think that the Temp directory setup on the default image is actually in the ObjectStore)
keep your data in an array inside your app together with a timestamps.
That is a good idea and so is SQLLite for persistence. This system really only currently uses Files to communicate between processes, it would be a good idea to use memory mapped files/or even sockets (since the data flows one way) or something to accomplish this.
What we have discovered is that one of the Compact Framework processes was performing very expensive operations in a loop and was delaying everything in the entire system! We run windows ce 7 , so all of the managed code apps are fighting for time on the first core.
We also discovered that logic we were using to iterate the temp directories that we created was O(N^3) and so we simplified that some.
Another concern - if you doing frequent file creating/deletion on SD card or eMMC storage it will cause a fast flash wearing. Please check this article for details.
