After bitmake tdx-reference-multimedia-image command processes are stuck

Hi there,
I want to build tdx-reference-multimedia-image for Colibri iMX7 Dual 1GB. I have followed this link. After running bitbake tdx-reference-multimedia-image progress was stuck.
I attached 2 picture of the progress.

By the way, I have built tdx-reference-minimal-image. There was no problem. I deployed image. Attached images are below.

Thank you,


Just a general rule, you need to have 8GB of RAM for each CPU core one allows the build process to use. Your screen shots are a bit tiny to read on this laptop, but 16 concurrent tasks would need 128GB of RAM to be safe.

I don’t remember the switch off the top of my head and I’m not at my office, just surfing through email with breakfast. I just know there is one. Use it to reduce the number of concurrent tasks to RAM / 8GB.

People get lulled into thinking their system can yocto with the best of them building the less resource intensive images, then they go for gusto and things like this happen. You have a deadly embrace going on where each process is waiting for one of the others to complete and free up its RAM for them to use. This routinely happens with some of the build processes but is a show stopper when it is all of them.

That would be the first thing to look at. Yes, your build will take longer.

1 Like

Thank you,
I know first build takes too much time. However, i can not see any changes in terminal, i think i had an error. Therefore i want to ask. Thank you again :slight_smile:

My system has Intel® Core™ i7-10700 CPU and 16 GB Ram. Have you any assumption that how long the tdx-reference-multimedia-image build process takes?

Thank you

Blockquote
My system has Intel® Core™ i7-10700 CPU and 16 GB Ram. Have you any assumption that how long the tdx-reference-multimedia-image build process takes?

No. With only 16GB of RAM, you can realistically only use 2 core though. Your build was running 16. I believe the 10700 has 8 physical core with virtualization which gives you 16 total virtual core. To run 16 virtual core flat out with a yocto/bitbake build you need 128GB of RAM. The page/swap file can let you cheat a bit, but disk I/O slower than RAM.

You must have an amazing swap file, but, alas, even the fake memory of a swap file couldn’t save this.

I have seen non-Toradex embedded linux builds on the best of the best quad-core Intel processors of the day take over 24 hours. This was quite a few years ago.

On the Dell T5500 with 12-core and 48GB or RAM I created a Ubuntu VM with 10 core and 40GB of RAM. The Torizon-Core build (not multi-media) took 3.5 hours.

On my HP Z820 with 20 physical core and 128GB of RAM I created a Ubuntu VM with 36864 MB and 16 core. Same build took 3 hours or just under.

“How long will it take?” is a complex question.

SSD drives are great at read speed but they suck at write speed once you exceed the write cache. Physically storing in them takes longer than charging rust on a spinning platter.

Western Digital Black spinning disks have much higher performance and life span than their Blue or base consumer models. When you get used to the “performance” of the sub $50 1TB consumer grade drives (probably what you have even if bigger capacity, probably consumer grade) it is a night and day difference dropping in a Black.

Gospel truth, I used to have a lot of consumer grade NAS scattered around my office.

I only archived stuff to them because they were just slow. Thankfully they all hit end of life as far as manufacturer support was concerned. I junked them all and installed a TerraMaster NAS.

I stuck four 4TB WD RED in it. Waited a day and a half for it to get done prepping the drives, and now it is blindingly fast. I can copy an entire ISO to/from it limited only by the Gigabit speed of my network. Writing to it across the network is faster than writing to a consumer grade disk in the computer I’m using. Can be a tiny bit slower to get the write started if the thing is idled down, but amazingly fast after that.

I took that detour so you understand the following.

Building any Linux kernel involves thousands of temporary object files being written. It also involves the same several hundred header files being read by the compiler hundreds of times each.

The “How long will it take” crowd tend to skimp on RAM and I/O speed.

Yes, you have one of the big bad new i7 processors. It’s probably got a consumer grade disk or SSD that isn’t going to perform well getting 3000+ temporary object files forced onto it in a rush.

The HP Z820 I’m typing this on is a loooooong way from new. It’s new enough to have USB 3.0 and not one bit newer. In fact, I had to replace the motherboard battery just last week this thing is so old. It’s got 128GB of RAM, 20 physical core, and the VM is using a 6TB WD Black drive. It gets a Torizon-Core yocto build done in around 3 hours.

A Linux kernel build is a long process with a lot of moving parts. The processor speed isn’t as important as the rest of your I/O including RAM for disk cache.

Nobody can answer your “How long will it take?” question with any accuracy.

Write a shell script to perform your clean yocto build. At the very beginning have the script print out the current time down to the second then write that down on a piece of paper.

Go out to supper with friends. Go to bed. Run errands the following morning. Some time after lunch see if it is done. If it is write down that time value and calculate the difference.

Plus or minus 10 minutes, that’s how long it’s going to take on your machine. Save that on a piece of paper.

As you upgrade components re-run said experiment again. This will tell you what gave you the best bang for the buck.

1 Like

Hi @bossman48,

Did you check if you have all dependencies needed for Yocto? You can check it here: Yocto Project Quick Build — The Yocto Project ® 4.0.1 documentation

Also, apparently, you are trying to add a custom meta-layer called meta-asel, am I correct? Did you try to build your image without any custom layers to see if it works?

Finally, the build gets stuck and doesn’t show anything? Does your CPU stay in 100% of usage while it’s stuck?

Best regards,
Hiago.

1 Like

Hi Hiago,
Thank you for your interest,

Actually, i have followed this link. After your comment, I checked your document and i run the the
command. One package was installed.

I have build tdx-reference-minimal-image before in my computer. After that, i wanted to add custom layer. The custom layer is built from my BSP team.Our BSP team were build the custom image in their computer. I wanted to build this image in my computer. All configurations were updated for my computer. In the building custom image, progress was stuck. Then, i wanted to build tdx-reference-multimedia-image to confirm my computer could build image or not. Therefore, i wanted to build the reference image.

I have changed the configuration file for tdx-reference-multimedia-image. In this build, the progress have reached to %89 in 15 minutes. You can see the image of the process is mentioned below.

My CPU usage is very low. I don’t understand exactly. The top command output from the computer is mentioned below.

Have you any suggestion for me? I am waiting the end of the process. Thank you,
Osman

Hi,

Thank you for your informative comment,

Actually, i want to ask to you your estimation of image building time in my computer. But you have written your comment informatively. Thank you so much :slight_smile:

Osman

Hi @bossman48,

This is strange, if you could build your image without this custom meta-layer and now you added it and Yocto stopped working, perhaps there is something wrong with your custom layer?
Could you please share with us your bblayers.conf? Also, how is the folder structure of your custom layer?

Please take a look at this guide on how to create your custom Yocto layer: Custom meta layers, recipes and images in Yocto Project (hello-world examples) | Toradex Developer Center

Best regards,
Hiago.

Hi Hiago,
After adding custom layer, progress was stuck. I wanted to build reference image that is tdx-reference-multimedia-image. Therefore i have changed the local.conf and bblayers.conf for tdx-reference-multimedia-image. I have removed custom layer. You might want to see the files. Because of that i am sharing. If there are some problems, could you inform me?

bblayers.conf (1.1 KB)
local.conf (11.5 KB)

Thank you,
Osman

Hi @bossman48,

Both of your files look fine. What Ubuntu version are you running?

Best regards,
Hiago.

Hi Hiago,
My ubuntu verison is 20.04 LTS. I was searching in log files in /home/user1/oe-core/build/tmp/log/cooker/colibri-imx7-emmc file. In console-latest.log file i found some problem. The problem is about alsa/alsa-utils_1.2.1.bb. I attached the log file.
20220628052628.log (6.9 KB)

In the log file, the last row is that “NOTE: Running noexec task 7942 of 8876 (/home/user1/oe-core/build/…/layers/openembedded-core/meta/recipes-multimedia/alsa/alsa-utils_1.2.1.bb:do_build)”. I think there is a package problem. Am i right?

Thank you,
Osman

Hi @bossman48,

I tried to bake the exact image that you’re trying, and it worked fine for me. Here is the output of my bitbake:

yocto@linux:~/oe-core/build$ bitbake tdx-reference-minimal-image                                                                                                                                                                 [21/47283]
Loading cache: 100% |#################################################################################| Time: 0:00:00                                                                                                                      
Loaded 3918 entries from dependency cache.                                                                                                                                                                                                 
NOTE: Resolving any missing task queue dependencies                                                                                                                                                                                        
                                                                                                                                                                                                                                           
Build Configuration:                                                                                                                                                                                                                       
BB_VERSION           = "1.46.0"                                                                                                                                                                                                            
BUILD_SYS            = "x86_64-linux"                                                                                                                                                                                                      
NATIVELSBSTRING      = "universal"                                                                                                                                                                                                         
TARGET_SYS           = "arm-tdx-linux-gnueabi"                                                                                                                                                                                             
MACHINE              = "colibri-imx7-emmc"                                                                                                                                                                                                 
DISTRO               = "tdx-xwayland"                                                                                                                                                                                                      
DISTRO_VERSION       = "5.7.0-devel-20220630140727+build.0"                                                                                                                                                                                
TUNE_FEATURES        = "arm armv7a vfp thumb neon callconvention-hard"                                                                                                                                                                     
TARGET_FPU           = "hard"                                                                                                                                                                                                              
meta-toradex-nxp     = "HEAD:cc4048ba326cb86cbf2631d5b31b415eb318ce82"                                                                                                                                                                     
meta-freescale       = "HEAD:3cb29cff92568ea835ef070490f185349d712837"                                                                                                                                                                     
meta-freescale-3rdparty = "HEAD:c52f64973cd4043a5e8be1c7e29bb9690eb4c3e5"                                                                                                                                                                  
meta-toradex-tegra   = "HEAD:16fc6147d6a145000693ebd695abb202da808499"                                                                                                                                                                     
meta-toradex-bsp-common = "HEAD:54c48da8942137b75c35fa6eb850f62fbbeb6bc9"                                                                                                                                                                  
meta-oe                                                                                                                                                                                                                                    
meta-filesystems                                                                                                                                                                                                                           
meta-gnome                                                
meta-xfce                                                 
meta-initramfs                                            
meta-networking                                           
meta-multimedia                                           
meta-python          = "HEAD:8ff12bfffcf0840d5518788a53d88d708ad3aae0"
meta-freescale-distro = "HEAD:5d882cdf079b3bde0bd9869ce3ca3db411acbf3b"
meta-toradex-demos   = "HEAD:6d6951d053ff1a7347af0ac04fcbb72564a28954"
meta-qt5             = "HEAD:5ef3a0ffd3324937252790266e2b2e64d33ef34f"
meta-toradex-distro  = "HEAD:cbde0286cb85bc445e70210b8df38f29b4784c08"
meta-poky            = "HEAD:7e0063a8546250c4c5b9454cfa89fff451a280ee"
meta                 = "HEAD:add860e1a69f848097bbc511137a62d5746e5019"

Initialising tasks: 100% |############################################################################| Time: 0:00:04
Sstate summary: Wanted 1702 Found 28 Missed 1674 Current 406 (1% match, 20% complete)
NOTE: Executing Tasks                                     
Currently 12 running tasks (4707 of 5811)  80% |#######################################################             | 
Currently 12 running tasks (4707 of 5811)  80% |#######################################################             | 
Currently 11 running tasks (4707 of 5811)  80% |#######################################################             | 
Currently 11 running tasks (4707 of 5811)  80% |#######################################################             | 
NOTE: Tasks Summary: Attempted 5811 tasks of which 1710 didn't need to be rerun and all succeeded.
NOTE: Writing buildhistory
NOTE: Writing buildhistory took: 3 seconds
yocto@linux:~/oe-core/build$ 

I’m running Ubuntu 22.04. The only difference is that I ran my bitbake inside a Docker container with ubuntu 20.04. So it should be the same as your setup.|

One thing that I’ve noticed is that your meta layers didn’t display the git commit. I will suggest you remove your oe-core folder and run your repo init and repo sync again, following this guide: Build a Reference Image with Yocto Project/OpenEmbedded | Toradex Developer Center

Also, make sure you have git installed and it’s working. I can’t see exactly what your problem is.

If you already tried those things and it still not working, you could try to run your bitbake inside a Docker container with an Ubuntu image. Let me know if you want to try this, I can help you with this setup.

Best regards,
Hiago.

Hi Hiago,
I have build the tdx-reference-minimal-image succesfully before. You have sent tdx-reference-minimal-image log. I want to build tdx-reference-multimedia-image. I think, this is not a package problem. I have researched bitbake log file. This file is not detailed version. Is there any detailed log file about bitbake processes? Have you any information? In case you want to see the last build log, you i am sending the lasest log file.
console-latest.log (3.2 KB)

Thank you,
Osman

Hi @bossman48,

I’m sorry, here is the correct log for tdx-reference-multimedia-image:

yocto@linux:~/oe-core/build$ bitbake tdx-reference-multimedia-image                                                                               
Loading cache: 100% |##############################################################################################################| Time: 0:00:00
Loaded 3918 entries from dependency cache.                                                                                                        
NOTE: Resolving any missing task queue dependencies                                                                                               
                                                                                                                                                  
Build Configuration:                                                                                                                              
BB_VERSION           = "1.46.0"                                                                                                                   
BUILD_SYS            = "x86_64-linux"                                                                                                             
NATIVELSBSTRING      = "universal"                                                                                                                
TARGET_SYS           = "arm-tdx-linux-gnueabi"                                                                                                    
MACHINE              = "colibri-imx7-emmc"                                                                                                        
DISTRO               = "tdx-xwayland"                                                                                                             
DISTRO_VERSION       = "5.7.0-devel-20220701123548+build.0"                                                                                       
TUNE_FEATURES        = "arm armv7a vfp thumb neon callconvention-hard"                                                                            
TARGET_FPU           = "hard"                                                                                                                     
meta-toradex-nxp     = "HEAD:cc4048ba326cb86cbf2631d5b31b415eb318ce82"                                                                            
meta-freescale       = "HEAD:3cb29cff92568ea835ef070490f185349d712837"
meta-freescale-3rdparty = "HEAD:c52f64973cd4043a5e8be1c7e29bb9690eb4c3e5"
meta-toradex-tegra   = "HEAD:16fc6147d6a145000693ebd695abb202da808499"
meta-toradex-bsp-common = "HEAD:54c48da8942137b75c35fa6eb850f62fbbeb6bc9"
meta-oe              
meta-filesystems     
meta-gnome           
meta-xfce            
meta-initramfs       
meta-networking      
meta-multimedia      
meta-python          = "HEAD:8ff12bfffcf0840d5518788a53d88d708ad3aae0"
meta-freescale-distro = "HEAD:5d882cdf079b3bde0bd9869ce3ca3db411acbf3b"
meta-toradex-demos   = "HEAD:6d6951d053ff1a7347af0ac04fcbb72564a28954"
meta-qt5             = "HEAD:5ef3a0ffd3324937252790266e2b2e64d33ef34f"
meta-toradex-distro  = "HEAD:cbde0286cb85bc445e70210b8df38f29b4784c08"
meta-poky            = "HEAD:7e0063a8546250c4c5b9454cfa89fff451a280ee"
meta                 = "HEAD:add860e1a69f848097bbc511137a62d5746e5019"

Initialising tasks: 100% |################################| Time: 0:00:07Sstate summary: Wanted 1597 Found 483 Missed 1114 Current 1622 (30% match, 65% complete)
NOTE: Tasks Summary: Attempted 8876 tasks of which 5777 didn't need to be rerun and all succeeded.
NOTE: Writing buildhistory
NOTE: Writing buildhistory took: 4 seconds
yocto@linux:~/oe-core/build$

As you can see, everything is normal here on my side. My CPU was 100% all the time and up to 12GB of RAM was being used.

I’m not aware of any other logs that you can check for errors. Did you try to remove the oe-core folder and start from scratch?

Also, there is one command that you can tun to clean your cache:

bitbake tdx-reference-multimedia-image -c cleanall

Then you can run your bitbake again. Please, see if this resolves your problem.

The image took some hours for me to complete.

Best regards,
Hiago.

Hi Hiago,
İ have started build process from scratch in this morning. The screenshot of the process is attached. Could you check the process?

Thank you,
Osman

Hi @bossman48,

It seems like it’s correct. I can see it running without errors, but I don’t know why your CPU usage is so low. Yocto is running 16 tasks at the same time, so it should be using more CPU.

It is stuck again? Were you able to complete your build?
Do you have a different computer that you could try this build?

Best regards,
Hiago.

Hi Hiago,
In my case, there is no error. I am confusing that i made something wrong. Therefore, i want to work on this link.

It is stuck. I can not complete the build. İ can try another computer later. ::frowning:
I cannot see any errors.

Thank you,
Osman

Hi @bossman48,

I made the exact same step that you did and I’ve never seen this kind of error before, that’s why I suspect there is something wrong with your system. If you could test it on a different computer, that would be good for us to investigate this problem.

While you can’t test on a different computer, do you want to try this inside a Docker container? We could try to isolate your system through a container.

I don’t know if you’re familiar with containers, but I could send you some instructions on how to do it.

Best regards,
Hiago.