How to verify that no data is written to flash memory between operations

Technical Support
, , , , ,
1

I power-on an Apalis iMX8 QuadMax 4GB IT, collect data from a USB device, dump that data on an SSD, then power-off. My code only copies the data from USB to RAM before writing it to the file on the SSD. My code is run within an Ubuntu docker container using TorizonCore.

I need to provide a process/script/anything to guarantee that none of the data that was received from the USB device is ever written anywhere in nonvolatile memory (aka the eMMC) besides the SSD of course. I have to verify this before powering down the device, but after the data collection is finished.

Is there a way to read the entire eMMC user area contents before running my code and compare it to the contents after running my code to show that nothing changed and prove no data was written to flash? Would making the eMMC Read-only be possible instead? Is there a Letter of Volatility for the iMX8?

Not sure what the best path forward would be depending on how TorizonCore handles the container’s read/write access to the flash eMMC and how TorizonCore logs data to the eMMC.

2

Forgot to include:

Apalis iMX8QM 4GB IT V1.1C found here.

Custom Carrier board based on Ixora V1.1B

TorizonCore 5.6.0+build.13, but version can be updated as needed, using PREMPT Patch.

Possibly related posts here and here.

3

Greetings @jonk4m,

This is a very interesting requirement. I assume this is due to some security/compliance for your product?

But in general let me say way I know. So a running container does store some small information in the non-volatile storage. This is related to state information regarding the container so that it can restart where it left off if it was stopped. However, if your application code is truly writing the data from USB to RAM to SSD. In theory it should never be written as part of this state information

On a minor note, one thing you should probably do is prevent your container from using swap memory: Runtime options with Memory, CPUs, and GPUs | Docker Docs

I need to provide a process/script/anything to guarantee that none of the data that was received from the USB device is ever written anywhere in nonvolatile memory (aka the eMMC) besides the SSD of course. I have to verify this before powering down the device, but after the data collection is finished.

As I said in theory I would think then that your data never touches the eMMC. However, proving/guaranteeing this is a different matter. I guess a good question is, for your purposes what would suffice as proof?

As for possible methods and next steps to look at. There are generic mmc-utils that can be used. These offer various features like monitoring/configuration of the eMMC. You might be able to monitor the eMMC before and after using these utils. Or, you could configure the eMMC temporarily to be write protected. Though I’m not too sure myself of the extent of these tools myself.

There are also various proprietary features and function of the eMMC depending on the manufacturer. Some manufacturer’s even provide tools to access these features and configurations. This could be another place to look.

We briefly touch on some of these eMMC monitoring/configuration methods in our article here: eMMC (Linux) | Toradex Developer Center

Best Regards,
Jeremias

4

Jeremias,

Thanks for responding! I completely agree that in theory the data should never touch the eMMC, but you’re also correct that this is motivated by security requirements. I will definitely look more thoroughly into the mmc-utils.

On a minor note, one thing you should probably do is prevent your container from using swap memory

Absolutely.

So a running container does store some small information in the non-volatile storage. This is related to state information regarding the container so that it can restart where it left off if it was stopped.

Can you direct me towards where you found this information? If I can understand exactly how TorizonCore/docker/containers effect the non-volatile memory, I can then investigate if I can either prevent those changes or at least identify and document them for a letter of volatility (LoV) document.

5

Can you direct me towards where you found this information?

Information on how Docker stores it’s information can be found here: https://docs.docker.com/storage/storagedriver/

For further reference we use the default overlay2 storage driver back in Docker. I’m not sure if this will contain the information you’re looking for, but it explains how container images and containers are stored in non-volatile storage on a system.

Best Regards,
Jeremias

6

After reading your references and a lot more, it seems the following are my options:

  • Erase the entire eMMC after each use → This will slowing destroy the eMMC as well as being a huge hassle.
  • Make the container the code runs in --read-only → This does not solve the problem because volumes like “/dev/bus/usb” will still need to be mounted to the container to access the USB device. Also this flag applies to the container’s filesystem, not the host’s.
  • Make the container the code runs in --tmpfs → Then I can’t write to the SSD since “/mnt/ssd” must be a volume mount, not tmpfs.
  • Make the eMMC read-only using mmc-utils → this seems to brick the system everytime (recoverable via tezi) since docker relies on having write access to the host.
  • Read entire contents of the eMMC after each operation to confirm nothing changed → something will change every time due to docker journals/logs. Also reading the contents of the eMMC, while running code that’s on the eMMC causes issues and unreliable data for obvious reasons. Maybe just doing a checksum would solve this but as said before, the contents will change regardless.
  • Run TorizonCore from the SSD instead of flash → As far as I can tell that’s not possible. Not to mention the performance hit I would take.
  • Ditch TorizonCore and add a partition to the SSD to run the OS from the SSD → This would be my last option if I cannot find a better methodology of proving beyond any doubt that no data from the USB device is written anywhere in the eMMC flash memory.

Please correct me where I’m wrong! Thanks!

7

I have the following comments regarding what you’ve said:

something will change every time due to docker journals/logs.

You should be able to disable/turn-off docker logging I believe: Configure logging drivers | Docker Docs

Also this flag applies to the container’s filesystem, not the host’s.

If everything will be run and processed within the container does the host filesystem matter in this case?

As for new comments:

  • I’m not very knowledgeable about such security requirements/compliance regarding this, so forgive the questions. But, what is the usual method/processes to guarantee that data never hits the eMMC? Is the “proof beyond any doubt” requirement that strict?
  • Another idea, I just had what if the container/docker data is stored on a USB/SD card medium rather than the internal eMMC? (See: How to Store Docker Data on an External Storage Device (USB/SD Card) | Toradex Developer Center). If all docker/container data is isolated to a USB/SD card it might be easier for you to manage, was my thinking at least.

Best Regards,
Jeremias

8

You should be able to disable/turn-off docker logging I believe.

If I read the entire contents of the eMMC on boot-up, then ran my container with logging turned off, then finished by reading the entire contents of the eMMC again, would any data other than logging cause those two readings to be different? If no other data would cause the eMMC contents to change at all, then this would certainly solve my use-case.

If everything will be run and processed within the container does the host filesystem matter in this case?

The --read-only flag mounts the container’s root filesystem as read only but doesn’t change the r/w permissions of the host. Although everything will run and be processed within the container, the host filesystem matters in this case because I have mounts and privilege flags that would allow the container to write to the host’s filesystem. If the container can write to the host filesystem, I once again need a methodology of guaranteeing that none of the data was written to the eMMC.

For reference, my flags:

docker run --restart always -itd --network host --cap-add=ALL -v /run/systemd/system:/run/systemd/system -v /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket -v /sys/fs/cgroup:/sys/fs/cgroup -v /dev/bus/usb:/dev/bus/usb -v /mnt/ssd:/media/ssd -v /dev:/dev --privileged=true --name NAME my-imported-image:v1 /bin/bash -c "<command to run code>"

what is the usual method/processes to guarantee that data never hits the eMMC? Is the “proof beyond any doubt” requirement that strict?

Typically, a letter of volatility document (LoV) is written for a device, and approval to use the device is based on information in that document. If we can “prove beyond any doubt” that no data from the USB device ever contaminates memory other than the SSD, then we can write a similar document for approval. Unfortunately, the requirements really are that strict.

Another idea, I just had what if the container/docker data is stored on a USB/SD card medium rather than the internal eMMC?

Am I understanding it correctly that docker will then use the external media to store all its process data, but TorizonCore itself will still be running on the eMMC? Aka this method would not allow me to completely remove/de-solder the eMMC?

Thank you again for all the help with this Jeremias!

9

would any data other than logging cause those two readings to be different? If no other data would cause the eMMC contents to change at all, then this would certainly solve my use-case.

Well no, because the logging isn’t everything that gets written to eMMC from a container. There’s the thin r/w layer of a container that has state information about a container as well to consider. Though if your application keeps the sensitive data in memory only then this shouldn’t get written to this r/w layer of your container. Well at least in theory, again I don’t know how you’d go about “proving” this.

the host filesystem matters in this case because I have mounts and privilege flags that would allow the container to write to the host’s filesystem.

But if you are so concerned about writes to the filesystem then why do you have such mounts to begin with? Is every bind-mount really necessary? Also can some of these be set as read-only bind-mounts instead?

Furthermore, looking at your docker run flags I find it odd that you’re concerned about the security of the system to this degree. Yet you’re running your container with --network=host and --privileged, flags which give the container increased privileges.

Am I understanding it correctly that docker will then use the external media to store all its process data, but TorizonCore itself will still be running on the eMMC? Aka this method would not allow me to completely remove/de-solder the eMMC?

Wait do you want to remove the eMMC from the module? As for the method I outlined, you got the right gist. The OS and everything related to it will still be on the eMMC. However, all data related to containers and such will instead come from this external media. Therefore all state information from your container will be written to this media rather than the eMMC. The only way data from the container would get written to eMMC is if you specifically have a bind-mount to the filesystem on the eMMC and write to this bind-mount. But, I imagine this would be a very conscious and avoidable action.

Best Regards,
Jeremias

10

The OS and everything related to it will still be on the eMMC. However, all data related to containers and such will instead come from this external media. Therefore all state information from your container will be written to this media rather than the eMMC. The only way data from the container would get written to eMMC is if you specifically have a bind-mount to the filesystem on the eMMC and write to this bind-mount.

I am going to attempt to U-Boot in Flash, with TorizonCore Kernel and Rootfs in SATA drive to meet our security requirements. Nonetheless, I am marking this as the answer to this thread since I think this is the best solution available without a provable means of verifying no data has been written to the eMMC.

11

Apologies for not being able to find a solution/method that satisfies your use-case here.