Docker error after OS image update

Technical Support
, , , , ,
1

Hello,

we have caused a Docker error and we do not understand how it occurred and how it cleared itself.

We have a few machines running which are powered by imX6ULL with an Iris boards. We build images with torizoncore-builder, with which we only add Docker credentials onto a Torizon 7.5.x/7.6.x image. We deploy apps with Docker Compose, which have specific permissions for some /dev paths, e.g. for the UART and I2C interfaces. Applications are deployed via Torizon OS. We haven’t experienced any Docker issues yet, even after OS image updates.

Yesterday we pushed a new OS image version based on Torizon OS 7.6.1 to one of our test machines at the office. A few minutes later we pushed a new application version. Some time after pushing the new OS image, we noticed that the application wouldn’t start. (Unfortunately, we are not sure if the previous application was able to start after the OS image, because we didn’t pay close enough attention to that.) After viewing Docker logs with journalctl we were able to see that Docker repeatedly produced an error level=warning msg="error locating sandbox id ... not found .

In an attempt to fix this, we ran e.g. docker network prune -a and docker system prune -a --volumes -f. Afterwards, we restarted the Docker service with sudo systemctl restart docker. At first, Docker kept on producing the “sandbox not found” errors, but after a few minutes Docker was able to start the Docker Compose application without any issues.

This behavior is very puzzling to us and we neither know how the error was caused nor how it was resolved. Do you have any insights on why this might have happened and do you know how to resolve these types of Docker errors?

Best regards,

Tom

2

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.

Hello Tom,

Based on the scenario you described, the “sandbox not found” error typically occurs when Docker’s internal network state becomes inconsistent, often during or after system updates. While I don’t have specific documentation about this exact error pattern in the sources provided, your troubleshooting approach was correct.

What Likely Happened

The timing suggests the issue arose during the OS image update process. When you updated from Torizon OS 7.5.x/7.6.x to 7.6.1, Docker’s network subsystem may have retained stale references to network sandboxes that no longer existed after the update.

Your Resolution Steps Were Correct

The commands you ran are the standard Docker cleanup procedures(1):

  1. docker network prune -a - removes unused networks
  2. docker system prune -a --volumes -f - removes stopped containers, unused images, networks, and build cache(1)
  3. sudo systemctl restart docker - restarts the Docker service

The fact that the error persisted briefly after cleanup but then resolved itself suggests Docker needed time to reconcile its internal state after the pruning operations.

Recommendations Going Forward

To minimize the risk of similar issues:

  1. Monitor application status immediately after OS updates - This helps identify whether issues stem from the OS update itself or subsequent application deployments
  2. Consider a systematic approach: After pushing an OS update, wait for it to complete and verify the previous application version starts correctly before deploying a new application version
  3. Document the state: Use docker ps -a to check container status and journalctl -u docker to review Docker service logs(1)

Prevention Strategy

If you encounter this again, the cleanup sequence you used is appropriate. You might also consider adding docker network ls before and after updates to track network state changes.

Since you’re using TorizonCore Builder to customize images and deploying via Docker Compose, ensure your compose files don’t create persistent network configurations that might conflict across OS versions.

Best regards,
Toradex AI Assistant.

Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

3

Greetings @trunner,

We’ve never seen reports of a similar issue from other customers in the past, so we don’t have much prior knowledge to go off of. Some quick google searches suggest this “sandbox” error is related to cache/state issues in Docker, but it seems to be a pretty generic error.

First of all, if you try to reproduce the issue following the same procedure, do you get the same issue reliably? Or does this only happen sometimes/rarely?

Another test that might be helpful is whether you can get this error to happen with some generic container images (i.e hello-world). Or, if it specifically happens with your set of containers. If that’s the case we may need to analyze your container images and how they’re configured.

we are not sure if the previous application was able to start after the OS image, because we didn’t pay close enough attention to that

In your reproduction tests, please check this. It could be important to know whether the issue happened cause of the OS or application update specifically.

Best Regards,
Jeremias

4

Hey Jeremias,

thank you for your response. It took me quite a while to follow up with some more information, because we haven’t been able to recreate this Docker error reliably. As you say, it seems to be a very generic error. But we think that we have - for our case - found the root cause of the error:

We are working with I2C devices via the Rust crate “i2cdev”, which is a FFI wrapper for I2C interactions via the Linux Kernel. The foreign functions are used in “unsafe” code blocks, meaning they potentially cause application panics. We are speculating that invalid I2C interactions might cause a panic which potentially leave the Docker container in an unrecoverable state. The Docker service then has to be restarted.

We haven’t been able to verify this definitively, but using I2C addresses that are not inside the required 7 bit address range have resulted in this Docker error. That’s why we think that an invalid Kernel interaction causes the error in our case.

We probably won’t investigate this further, because we believe that our code (which was in the development phase) plainly was faulty and that we wouldn’t have triggered this error if we had correctly implemented the I2C Kernel interactions.

Best,

Tom

5

Interesting theory. I’ve never heard of “faulty code” running in a container leaving Docker in such a state. I suppose it’s plausible, though proving it is another matter.

Well in any case if this does solve your issue I’m glad to see you found a way. I know you said you won’t investigate this further, but if you happen to find further proof that this theory you proposed is true it would be appreciated if you could update the thread with this information.

Best Regards,
Jeremias