Bitbake failures with Aquila AM69

Technical Support
, ,
1

For the past week, I’ve noticed recurring build failures when trying to build images for the Aquila AM69. “bitbake torizon-minimal” always seems to fail with messages similar to the following:

NOTE: No reply from server in 30s (for command ping at 03:14:09.847634)
Timeout while waiting for a reply from the bitbake server (60s at 03:14:39.906421)
Command exited with non-zero status 1

This has happened across two systems, with fresh checkouts, trying both scarthgap-7.x.y (now identifying as 7.3.0), and 7.2.0. At first I thought it might be memory issues, because the build triggered the OOM killer about half a dozen times… but after limiting build threads to mitigate this, the problem keeps recurring. Switching to a build server with fewer cores and more physical memory avoided the out-of-memory errors, but the build still failed with similar timeout messages. Sometimes the build gets far enough to produce an ostree_repo for deployment, but usually not.

FWIW, I can build the Verdin iMX8M Plus image without issue, on both the same build servers. So this appears to be an Aquila-specific build problem?

EDIT: basic details of the build servers: one is a Xeon E5-2620 v3 (6 cores+6 threads) with 32 GB RAM; the other is a Xeon X5647 (4 cores+4 threads) with 48GB RAM. Both started with over 200 GB of disk space for the build area.

2

Hi, I’m the Toradex AI Assistant.

I might make mistakes. Our support team will step in if needed. If I answered your question, please tick the “Solution” checkbox. For follow-up questions to me, please use @ToradexAI in your post.

Did you like my answer? :+1: :-1:

Hello! Thank you for reaching out with your detailed description of the build failures for the Aquila AM69. I understand how frustrating intermittent build problems can be, especially with timeouts and OOM events.

Here are some recommendations to address your issue:

1. System Requirements & Resources

The Aquila AM69 Yocto build is known to be quite resource-intensive. Toradex recommends:

  • At least 8 GB of RAM (preferably more) for complex builds.
  • 120 GB or more of free disk space on the host machine.
  • Reference system requirements

If builds are triggering the OOM killer, limiting threads (using BB_NUMBER_THREADS and PARALLEL_MAKE in your local.conf) helps, but persistent timeouts point at either:

  • System resource exhaustion (CPU, RAM, or Disk I/O).
  • Underlying issues with your build environment or machine setup.

2. Bitbake Server Timeouts

The errors you’ve reported:

NOTE: No reply from server in 30s (for command ping at 03:14:09.847634)
Timeout while waiting for a reply from the bitbake server...

are typical if BitBake or the bitbake server process is overwhelmed, starved of RAM, or heavily swapping. This can also happen if your machine has intensive disk or CPU contention.

  • Check your swap configuration: If you are frequently running out of RAM, ensure you have sufficient swap enabled.
  • Monitor system resources: Run tools like htop, top, or iotop during the build to watch for any bottlenecks.

3. Build Environment Consistency

  • Double-check that all required dependencies are installed for your host OS: Yocto build dependencies
  • Use a supported Linux distribution as documented for Yocto builds.

4. Retrying Fetches/Builds

Sometimes, fetch errors or transient failures can occur due to network issues or server unavailability. Retrying the build is recommended in these cases:

  • Run bitbake torizon-minimal again after failure to see if it progresses further.
  • If the build consistently fails at the same task, investigate that recipe’s logs for more details.

5. Aquila-Specific Dependencies or Issues

Given that your Verdin iMX8M Plus images build correctly on the same machines:

  • There may be recipe dependencies or resource needs unique to the Aquila AM69 that are not present with Verdin.
  • Review any machine-specific settings in the Yocto conf/machine files that could influence resources used or server settings.

6. Clean Build

If you have switched Yocto branches/tags or made significant changes, it’s best to clean your build:

bitbake -c cleansstate torizon-minimal
bitbake torizon-minimal

or even a full rm -rf tmp/ before rebuilding.

If the above steps do not resolve the timeouts, please share which Linux distribution and version you are using on your build server, as well as your BB_NUMBER_THREADS and PARALLEL_MAKE values.

For additional details, please consult the following documentation:

Let me know if you need more targeted troubleshooting help!

Best regards,
Toradex AI Assistant.

3

Hey @jarndraugr,

Correction Torizon OS is available sense the 7.1.0 release. Apologies,
I have one going now to see if I can reproduce your error.

-Eric

5

Hey @jarndraugr,

Just ran a new torizon-mimimal build and it built as expected. What OS are you using for your build? Doesn’t it comply with yocto’s requirements? Ubuntu 24.04

https://docs.yoctoproject.org/ref-manual/system-requirements.html

-Eric

6

/etc/os-release says:

PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

At the last build on this system, I didn’t see it trigger the OOM killer, I didn’t see it thrashing the swap partition, and it still had about 90GB disk space free after the last failure. The tail end of the build log suggested it was wrapping up deployment packaging when it failed:

NOTE: recipe torizon-minimal-1.0-r0: task do_image_ostreecommit: Started
NOTE: recipe torizon-minimal-1.0-r0: task do_image_tar: Succeeded
NOTE: recipe torizon-minimal-1.0-r0: task do_image_ostreecommit: Succeeded
NOTE: Running task 11269 of 11276 (/home/fschaefer/mc11-c1-linux/build/build-torizon-aquila-am69/conf/../../layers/meta-toradex-torizon/recipes-images/images/torizon-minimal.bb:do_image_ota)
NOTE: Running task 11270 of 11276 (/home/fschaefer/mc11-c1-linux/build/build-torizon-aquila-am69/conf/../../layers/meta-toradex-torizon/recipes-images/images/torizon-minimal.bb:do_image_ostreepush)
NOTE: recipe torizon-minimal-1.0-r0: task do_image_ota: Started
NOTE: recipe torizon-minimal-1.0-r0: task do_image_ostreepush: Started
WARNING: torizon-minimal-1.0-r0 do_image_ostreepush: SOTA_PACKED_CREDENTIALS not set. Please add SOTA_PACKED_CREDENTIALS.
NOTE: recipe torizon-minimal-1.0-r0: task do_image_ostreepush: Succeeded
NOTE: Running task 11271 of 11276 (/home/fschaefer/mc11-c1-linux/build/build-torizon-aquila-am69/conf/../../layers/meta-toradex-torizon/recipes-images/images/torizon-minimal.bb:do_image_garagesign)
NOTE: No reply from server in 30s (for command ping at 03:14:09.847634)
Timeout while waiting for a reply from the bitbake server (60s at 03:14:39.906421)
Command exited with non-zero status 1
503.31user 15.65system 11:03:50elapsed 1%CPU (0avgtext+0avgdata 68984maxresident)k
84624inputs+17496outputs (5426major+22933minor)pagefaults 0swaps
7

Hey @jarndraugr,

Can you try building the image within a container such as described here:

-Eric