Debugging custom layer in bitbake

I’m attempting to add a custom layer to the TorizonCore build process as documented at https://developer.toradex.com/knowledge-base/custom-meta-layers-recipes-images-in-yocto-project

So far I have successfully:

  • Configured the build host
  • Built the standard TorizonCore images (for 5.0.0), which took hours and hours
  • Added my local layer using bitbake-layers create-layer
  • Added my layer to conf/bblayers.conf
  • Created a “hello world” recipe in my layer
  • Run bitbake-layers show-recipes and confirmed my recipes are present
  • Run bitbake my-target and build my target successfully with no error messages

However, when I then attempt to build the TorizonCore images with my component added by running bitbake torizon-core-docker, I get one or more error messages as below (exact number of times I see this does vary somewhat from run to run):

NOTE: Executing Tasks
WARNING: Failed to get layers information. Exception: <class 'bb.process.ExecutionError'>
WARNING: torizon-core-docker-1.0-r0 do_rootfs: Failed to get layers information. Exception: <class 'bb.process.ExecutionError'>
WARNING: Failed to get layers information. Exception: <class 'bb.process.ExecutionError'>
WARNING: Failed to get layers information. Exception: <class 'bb.process.ExecutionError'>

and the build process torizon-core-docker.bb:do_image_ostreecommit task fails with numerous subsequent errors.

If I remove my layer from bblayers.conf, then the build succeeds every time, so it is clearly something wrong with my layer.

How can I debug this? How can I see what the underlying bb.process.ExecutionError exception is showing?

After several hours of adding asserts to the bitbake python libraries, it seems the failing command is running git rev-parse --abbrev-ref HEAD on the module directory. This fails in the demo case, as the layer is not a GIT module, it’s just a directory. This is called from layers/openembedded-core/meta/classes/metadata_scm.bbclass

Workaround is easy enough in this demo case, set the module up as a fake GIT project

git init && git add README && git commit -m 'fake GIT project for bitbake'

However, for a real build our local code will likely be in SVN not GIT. And in a Docker-ised build, a layer might not look like it’s in SCM at all, it might be copied from the Docker context.

Is subversion (or any SCM other than GIT, or no SCM at all) supported for bitbake layers? Looking at the code, calls to git are spread all through the libraries so I suspect the answer is NO.

Hi @gregb,

It seems that something is wrong with your layer.

You mentioned the create-layer command but the correct one is add-layer.

Have you checked this yocto documentation?

Best regards,

Daniel Morais

Hi Daniel,

I’ve read as much documentation as I can find, but I have seen nothing that addresses this issue. Plenty about using SVN in recipes, but not about using SVN (or anything other than git, including plain directories) in layers.

As I read it, create-layer builds the layer framework including the conf/layer.conf file. add-layer just inserts the path in bblayers.conf, which I have done by hand (as recommended by Torazon here)

Even if my layer is totally empty of recipes, the build fails unless the layer looks like a GIT repo. For now, I can just create a sham git repo in the layer directory, as outlined above. This confuses the heck out of the IDE but otherwise seems to work.

Hi @gregb,

Just so I understand on my side, so the issue is that your meta-layer itself is an SVN? This is interesting, while I don’t use SVN myself I wouldn’t think this a problem, though I may be wrong. Also I don’t quite understand your comment on not being able to use plain directories. For quick tests I’ve created meta-layers out of just plain directories before and it has never been a problem. Apologies if I misunderstand the situation.

So it seems everything works until you add your “component” to your layer correct? Could you maybe describe this component if possible? Like how you are adding it to your layer and such?

Best Regards,
Jeremias

Jeremias,

As far as I can see, the problem is that the bitbake framework requires that the layer directory itself look like a GIT repo, on which git rev-parse HEAD can run successfully. If the layer directory is an SVN project, or not in any SCM at all (which is what I mean by “plain directory”), then the build fails.

Here’s how I can replicate the problem (which basically follows the Torizon document):

  1. Set up a dev environment until bitbake torizon-core-docker succeeds

  2. bitbake-layers create-layer layers/mylayer

  3. bitbake-layers add-layer layers/mylayer

  4. bitbake torizon-core-docker now rebuilds the kernel (due to new recipe), and fails. Initially, with many warnings like

    WARNING: /workdir/build-torizon/conf/…/…/layers/openembedded-core/meta/recipes-core/images/core-image-minimal-dev.bb: Failed to get layers information. Exception: <class ‘bb.process.ExecutionError’>

as the recipes are re-parsed, then the actual build fails with

WARNING: Failed to get layers information. Exception: <class 'bb.process.ExecutionError'>
WARNING: torizon-core-docker-1.0-r0 do_image_ostreecommit: Failed to get layers information. Exception: <class 'bb.process.ExecutionError'>
ERROR: torizon-core-docker-1.0-r0 do_image_ostreecommit: Execution of '/workdir/build-torizon/tmp-torizon/work/apalis_imx8-tdx-linux/torizon-core-docker/1.0-r0/temp/run.do_image_ostreecommit.290621' failed with exit code 1:
error: Parsing oe.layers=None : 0:expected value
WARNING: exit code 1 from a shell command.

If I turn the layers/mylayer directory into a git repo:

(cd layers/mylayer && git init && git add README && git commit -m 'fake GIT repo for bitbake')

then run bitbake torizon-core-docker it works.

After some internal discussion, I believe your best course of action would just be work-around this limitation as you already are. For the time being we choose to support solely git as the versioning system with OSTree.

Though like I said you can continue to use SVN, but will need to work around this. Other than what you’ve already done, you can also try mirrioring your SVN repo to git for future development. Then like I said previously if you don’t plan to use or care about the OTA features of Torizon then it doesn’t matter too much how you work around this.

Best Regards,
Jeremias

Thanks Jeremias. We do not need OTA updates, but for now we will proceed as above by using a fake GIT repo. Hopefully we can migrate to GIT at some point.

You might want to ask your documentation colleagues to update the tech note with this information.

I apologize for the slight inconvenience but I’ll work with our documentation team to make sure this note is included.

I appreciate you reporting this issue.

Best Regards,
Jeremias

Oh now that I see the full error I know what’s going on now. So actually this has nothing to do with the bitbake framework. Rather the issue is sort of our fault.

To explain, we include this package “OSTree” into our Torizon image. This package serves as the framework for our OTA update features. Whenever a Yocto build is done we create an OSTree update package out of that build for compatibility. However in this update package includes information about the build, namely information about the each layer revision, which we parse with git as you can tell.

Let me bring this up to the team and see if there are any thoughts on making our OTA update framework compatible with other version system than git. Worst-case scenario you could also just remove the OSTree components from the Yocto build. However I can’t really say how this might end up as this package is fairly interwoven into our build and other tooling. Really it might be more trouble than it’s worth, but still a possibility. In any case let me get back to you after speaking with the team internally.

Best Regards,
Jeremias

I can add another, perhaps more manageable solution to this problem.

Install the git-svn package and then use git svn clone <SVNURL> <layerdir> command. This makes the layer directory look like a git repo, but the actual data is still in the SVN repo.

Thank you for sharing this alternative solution! This will be helpful to other customers who use SVN or maybe another version control framework.