Docker 29.x on Ubuntu causes "no matching manifest for linux/arm64/v8" and Compose v5.0.0 crash

Environment

  • Host OS: Ubuntu 22.04.5 LTS

  • VS Code: 1.107.1

  • Torizon IDE Extension: 3.2.6

  • Docker: Engine (not Desktop, as recommended by the guide)

  • Target: Verdin iMX8M Mini

  • Guide followed: Set Up the IDE Ext2 Environment

Problem Description

After a fresh Docker Engine installation on Ubuntu following the official guide (which recommends Docker Engine over Docker Desktop for Linux), I encountered two issues when trying to debug a Python project created with the Torizon IDE Extension.

Issue 1: Manifest architecture mismatch

When running debug, the pull command failed:

Executing task: DOCKER_HOST=192.168.15.36:2375 docker pull localhost:5002/python3-docker-debug:arm64 
no matching manifest for linux/arm64/v8 in the manifest list entries

The image was being built correctly as arm64 locally (docker inspect confirmed this), but when pushed to the local registry, the manifest showed amd64 instead.

Issue 2: Docker Compose crash

After fixing Issue 1, the extension failed with a panic error:

Executing task: DOCKER_HOST= docker compose push python3-docker-debug 
panic: runtime error: index out of range [19] with length 9
goroutine 16 [running]:
github.com/docker/compose/v5/cmd/display.(*ttyWriter).lineText(...)

Root Cause Analysis

I identified the problems by comparing with a working Windows 11 + WSL2 setup that followed the same guide successfully.

Component Ubuntu (broken) Windows/WSL2 (working)
Docker 29.1.3 28.4.0
containerd 2.2.1 1.7.27
Storage Driver io.containerd.snapshotter.v1 overlay2
Docker Compose v5.0.0 v2.39.2

Issue 1 was caused by Docker 29.x enabling the containerd-snapshotter by default, which corrupts the manifest when pushing multi-arch images to a local registry.

Issue 2 is a bug in Docker Compose v5.0.0.

Workaround Applied

Fix 1: Disable containerd-snapshotter

Create or edit /etc/docker/daemon.json:

json

{
  "features": {
    "containerd-snapshotter": false
  }
}

Then restart Docker:

bash

sudo systemctl restart docker

Verify with:

bash

docker info | grep "Storage Driver"
# Should show: Storage Driver: overlay2

Fix 2: Downgrade Docker Compose to v2.x

bash

sudo curl -L "https://github.com/docker/compose/releases/download/v2.32.4/docker-compose-linux-x86_64" \
  -o /usr/libexec/docker/cli-plugins/docker-compose
sudo chmod +x /usr/libexec/docker/cli-plugins/docker-compose

After both fixes, I cleaned up the environment and recreated the project:

bash

docker system prune -a --volumes
docker stop registry && docker rm registry
docker run -d -p 5002:5000 --restart=always --name registry registry:2

Then created a new project in VS Code with the Torizon extension. The debug workflow now works correctly.

Questions for Toradex Team

  1. Did I miss something in my initial configuration, or is this a known incompatibility with Docker 29.x and Compose v5.0.0?

  2. Is the workaround I applied (disabling containerd-snapshotter and downgrading Compose) the recommended solution, or is there a better approach?

  3. Are there plans to support these newer Docker versions in future releases of the Torizon IDE Extension?

Thank you!

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.


Docker Compatibility Issues with Torizon IDE Extension

Thank you for the detailed analysis of the Docker compatibility issues with the Torizon IDE Extension. Your investigation and workaround demonstrate excellent troubleshooting skills.

Understanding the Issues

The problems you’ve identified align with known compatibility challenges between newer Docker versions and the Torizon development workflow:

  1. Containerd-snapshotter Issue: The manifest architecture mismatch occurs because Docker 29.x enables the containerd-snapshotter by default, which can interfere with multi-architecture image handling in local registries.

  2. Docker Compose v5.0.0 Bug: The panic error you encountered is indeed a known issue with Docker Compose v5.0.0.

Recommended Solutions

Your workaround approach is sound and represents the current best practice for resolving these compatibility issues:

Solution 1: Disable Containerd-Snapshotter

Your configuration in /etc/docker/daemon.json is correct:

{
  "features": {
    "containerd-snapshotter": false
  }
}

This ensures the storage driver returns to overlay2, which is compatible with the Torizon development workflow.

Solution 2: Use Compatible Docker Compose Version

Downgrading to Docker Compose v2.x (like v2.32.4) resolves the panic error and maintains compatibility with the Torizon IDE Extension.

Development Workflow Verification

After applying your fixes, the standard Python development workflow should work correctly(1). The extension creates the necessary Docker configurations and manages container deployment for debugging on your Verdin iMX8M Mini.

Answers to Your Questions

  1. Configuration: You didn’t miss anything in your initial setup. These are known incompatibilities with Docker 29.x and Compose v5.0.0.

  2. Workaround Validity: Your workaround (disabling containerd-snapshotter and using Docker Compose v2.x) is currently the recommended approach for maintaining compatibility.

  3. Future Support: While I don’t have specific timelines for Docker 29.x and Compose v5.0.0 support, Toradex continues to update the IDE Extension to support newer toolchain versions.

Additional Recommendations

  • Monitor the Torizon IDE Extension release notes for updates regarding Docker compatibility
  • Consider documenting your Docker configuration for team members who might encounter similar issues
  • Test thoroughly after any Docker updates to ensure continued compatibility

Your systematic approach to identifying and resolving these issues will help other developers facing similar challenges with the Torizon development environment.

Best regards, Toradex AI Assistant.


Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

@uilter I filed a bug report in torizon templates repo for the incompatibility with containerd snapshotter: Unable to run projects with docker "containerd snapshotter" backend (docker >=29) · Issue #427 · torizon/vscode-torizon-templates · GitHub

1 Like

Thank you for filing the GitHub issue for the containerd snapshotter incompatibility. This is an important step for tracking and resolving the Docker 29.x compatibility issues with the Torizon IDE Extension.

Issue Documentation and Community Support

Your bug report will help the Toradex development team prioritize this compatibility issue and potentially implement a proper fix in future releases of the VS Code extension templates. The containerd snapshotter problem affects the multi-architecture image handling that’s essential for the Torizon development workflow(1).

Related Community Issues

Similar Docker compatibility issues have been reported in the community. For example, there have been cases where TorizonCore Builder encounters Docker-related problems(2), and architecture detection issues in VS Code environments(3). Your detailed analysis and workaround documentation will be valuable for other developers encountering similar challenges.

Development Environment Considerations

While your workaround (disabling containerd-snapshotter and using Docker Compose v2.x) resolves the immediate issues, tracking this through the official GitHub repository ensures that future versions of the Torizon IDE Extension can properly support newer Docker versions without requiring manual configuration changes.

The Torizon platform is designed to simplify embedded development workflows(1), and maintaining compatibility with the latest Docker toolchain versions is important for the developer experience.

Next Steps

The GitHub issue will allow the development team to:

  • Investigate the root cause of the containerd snapshotter incompatibility
  • Develop a proper fix or alternative approach
  • Update the extension to handle Docker 29.x configurations automatically
  • Document any necessary configuration changes for users

Your contribution to identifying and documenting this issue helps improve the Torizon development ecosystem for the entire community.

Best regards, Toradex AI Assistant.


Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

Hello @uilter !

In regards to Issue 1:
This error normally occurs when ARM emulation is not enabled on your x86 host machine. To enable it, you can simply follow this guide here: Configure Build Environment for Torizon Containers | Toradex Developer Center
Just note that ARM emulation is not persistent and must be enabled at every boot.

After enabling ARM emulation, your project should work correctly with the Torizon IDE Extension.

In regards to Issue 2:
The error you’re havving is related to a known issue with certain versions of Docker Compose. This appears to be a bug in the Docker Compose CLI itself, not specifically with the Torizon IDE Extension. Simply downgrade Docker compose and it should work, for now your approach is the way to do so, but we will investigate this issue further.

1 Like

Thanks for your comments and considerations!

@uilter Thank you for your detailed description and workaround. I have been fighting this issue for 2 weeks now and was finally moving towards versions being the problem. But you got there quicker and saved me some time.
Regards
Lloyd

1 Like

Glad to know, I was in the same situation.