Help to integrate a custom kernel into a Torizon Core Image

[kevin.tx]

Hi @gerko ,

Sorry for the lengthy delay.

So I try to forward the message from the team about this issue here.

→ Step 1: Clone the kernel with git clone
→ Step 2: Modify the value
→ Step 3: Run git format-patch to create a patch
→ Step 4: Add that patch with a .bbappend to the kernel recipe in a custom layer.

If its a kernel config then you would need to do this with kernel fragments.
https://wiki.koansoftware.com/index.php/Modify_the_linux_kernel_with_configuration_fragments_in_Yocto

For both situations the recommended way is to do this all in another layer or maintain a rebased version of meta-toradex-torizon

Hope this helps.

Best Regards
Kevin

[gerko]

Thank you,
we’ll let you know if it’s successful or if we have any further questions.

Best Regards
Gerald

[kevin.tx]

Hi @gerko ,

Sure! Let us know.

Have a great day and weekend!

Best Regards
Kevin

[matthijs]

I’ve also been struggling with a similar thing as @gerko - I’m trying to fix a bug in the kernel, for which I need to make changes to the code. From what I understand, the most reliable way to do that is to build a custom TorizonCore image using Yocto, adding a kernel patch as a custom layer (I might be messing up terminology here), I think I’ll figure that out from the docs at Build with Yocto Documentation Overview | Toradex Developer Center

However, I have two more questions:

1. Originally, I have tried compiling just the kernel itself (using the linux-toradex git tree and plain make and these instructions, hoping I could integrate that self-compiled kernel either into a pre-built TC image, or maybe just deploying the kernel and modules directly to the target. In particular, this section suggests that you can add a custom kernel to a TC core image/tarball by modifying to bootfs tarball inside, but this answer by @rudhi.tx suggests this only applies to “Reference BSP” images, not “TorizonCore” images. If I understand that correctly, maybe the docs at that page could make this a bit more explicit, pointing people to the custom Yocto build if they are building on top of TC images?
2. Provided I manage to make a custom Yocto image and deploy it, is there any way to more quickly iterate? I expect I might need a couple of tries before I get this right, and doing a full Yocto build everytime is probably going to be slow (unless its caching is really good?). I can image that I can just replace e.g. /boot/ostree/torizon-[hash]/vmlinuz-5.15.77-6.2.0+git.aa0ff7e3554e or a .ko file in /lib/modules with a new version (if Yocto leaves a linux source tree with built files somewhere?), maybe using ostree commit /sysroot/ostree/deploy/torizon/deploy/[hash].0 to commit the changes and then deploy the new commit to prevent the changes from being overwritten? Or is it perhaps easier to switch to a non-ostree BSP reference image (possibly netbooted?) for such development? How do you folks at Toradex do such kernel development?

[gerko]

Hi @matthijs ,

we have found a path that we can walk.
Actually it is quite easy. However, I wanted to explore this path a bit more and that’s why I haven’t replied here yet.

@matthijs

If I understand that correctly, maybe the docs at that page could make this a bit more explicit, pointing people to the custom Yocto build if they are building on top of TC images?

I feel the same way.

@matthijs

2. Provided I manage to make a custom Yocto i …

I can’t really answer all of that. But here is our approach:
(A colleague found the solution)

• Create an unmodified Torizon OS image as described here. these instructions
• This way you also get the corresponding kernel sources that you can edit and that will be used in the next build.
  For me it looks like this after the following command:

MACHINE=apalis-imx8 source setup-environment ~/torizon-os/torizon-os-img
bitbake torizon-core-docker

• ~/torizon-os/torizon-os-img/tmp/work-shared/apalis-imx8/kernel-source // for me

• In my case we only changed 2 values in an rpmsg driver source file that I didn’t easily change through a new self-created kernel module. Whether by applying a patch or by editing the sources directly doesn’t seem to matter.

• bitbake it again

• A customized image that can then be used in the Torizon Core Builder can be found at: (in my case)

~/torizon-os/torizon-os-img/deploy/images/apalis-imx8/torizon-core-docker-apalis-imx8-Tezi_6.4.0-devel-20231024082742+build.0.tar

Then follow these instructions: OS Customization Documentation Overview | Toradex Developer Center
When making any adjustments (dto, custom kernel-modules, …), you have to be careful that here the original kernel is not used again. You also have to develop a good understanding here.

The more professional approach is certainly to work with recipes and layering. We will still have to deal with this too. But the key insight is finding the tar file and then how to use it.

Maybe we could lead you on the almost ‘right path’.

Greetings
Gerald

[matthijs]

Hey Gerald, that sounds like a reasonable approach that I would probably have gone looking for, so thanks for providing your pointers! I’m sure that will save me a ton of time. I just had a quick look - I did the unmodified Yocto build using docker, so your torizon-os/torizon-os-image directory is named yocto-workdir/torizon/build-torizon for me. Inside, everything looks the same.

When making any adjustments (dto, custom kernel-modules, …), you have to be careful that here the original kernel is not used again. You also have to develop a good understanding here.

I’m not sure I understand this, how could that happen? The original kernel is no longer part of your new torizon-core-docker-apalis-imx8-Tezi_6.4.0-devel-20231024082742+build.0.tar file, so how could torizoncore-builder introduce it again?

Looking at the kernel sources, I also see a kernel-build-artifacts directory beside it, which seems to contain some of the build directory (at least the kernel config used it seems and some generated .h files, though not any compiled .o files), so maybe that could be used to compile the kernel outside of yocto manually, and then just updating the kernel or modules on the running system manually, for even quicker turnaround time during development/debugging.

In any case, my original question doe still stand, if anyone (from Toradex maybe) has a suggestion on doing quick turnaround kernel changes (i.e. making changes without having to run the full kernel build every time), I would like to hear it!

[gerko]

@matthijs

I’m not sure I understand this, how could that happen?

I think I messed things up and didn’t really understand the unpack command and what it does. I just wanted to point out that some strange things happened in my endless experimentation.

[matthijs]

@gerko

I think I messed things up and didn’t really understand the unpack command and what it does.

Yeah, I’ve been there (trying to make changes, only to discover I was forgetting the unpack command so I had been deploying the exact same unmodified image for half a day ;-p). Thanks for the heads up, but I think I have that part covered by now

Regarding your modify-kernel-and-bitbake-again approach, I do see one little mystery: How does the build system know that it should build the kernel again (and also without wiping and replacing the source directory)? I would think it keeps track of things it already built, and just skips over the kernel task in this case? If not, I guess it must inspect each source directory for changes, but if it does, I would expect it would just wipe the source directory because it is changed relative to what it should be?

Do you have any idea? Or did you just try it and it “just works”?

[gerko]

@matthijs

Do you have any idea?

I’m not that much of an expert. Toradex support should be able to answer that. I just look at the date of the new tar file and hope everything went well.
My colleague intentionally introduced a bug into the source code. An error is immediately displayed and you know that exactly this source is being used. I know, doesn’t sound professional.

[henrique.tx]

HI @gerko and @matthijs !

Sorry for the delayed answer.

And thanks @gerko for being so helpful!

I would like to try to contribute here

@matthijs

If I understand that correctly, maybe the docs at that page could make this a bit more explicit, pointing people to the custom Yocto build if they are building on top of TC images?

That’s a good point. I brought it up internally yesterday. It is being discussed.

Thanks for pointing that out. Related to it, we have this section of this article that shows the use cases (and examples) for TorizonCore Builder: TorizonCore Builder Tool - Customizing Torizon OS Images | Toradex Developer Center. But, as you said, we are missing to explicitly show the use cases where going to Yocto is needed.

Also, related, the Torizon OS Reference Minimal was recently released, in case you would like to check:

Torizon OS Reference Minimal Image Technical Overview | Toradex Developer Center

@matthijs

Provided I manage to make a custom Yocto image and deploy it, is there any way to more quickly iterate?

There are several ways to perform the iteration process. See below some possibilities and my personal view of how relatively “quick” I think the iteration is (quick on quotes as it might depend on your own setup and also might be subjective). I purposely wrote it as briefly as possible, but let me know if you would like to better explore it.

Index	Build	Deployment	How quick?
1	Yocto	Flashing using Tezi	Slowest (always flashing the whole image)
2	Yocto	Using TorizonCore Builder	Nice and quick (tcbuild.yaml helps to automate this better)
3	Yocto	Using OStree directly	Quick, but needs OStree knowledge
4	Yocto	Using OTA	Not that quick, but not that slow (Needs internet)
5	Yocto	Using TFTP and NFS	Quick, but needs some knowledge on show to setup

For TorizonCore Builder (1), we have some thorough articles: OS Customization Documentation Overview | Toradex Developer Center
For OStree (3), we have the article OSTree | Toradex Developer Center. The section OSTree Update From a Local Torizon OS Build in there is especially useful.
For TFTP and NFS (5), we have this article Boot from a TFTP/NFS Server | Toradex Developer Center.
Also, I will bring up this kernel development on Torizon use case internally. It might become an interesting feature

@matthijs

and doing a full Yocto build everytime is probably going to be slow (unless its caching is really good?)

You would not be doing a full Yocto build everytime because indeed the caching is good
If you are interested, we have some webinars about Yocto that can help you. Not to mention that we also have some partners that offer training on Yocto. The training might interest you too, @gerko, to accelerate development

@gerko

The more professional approach is certainly to work with recipes and layering.

Agreed

@matthijs

Regarding your modify-kernel-and-bitbake-again approach, I do see one little mystery: How does the build system know that it should build the kernel again (and also without wiping and replacing the source directory)?

Here I would like to comment on another possibility: maybe the easiest way of developing kernel-related stuff is to have a simpler image installed (e.g. based on BSP or reference minimal image) to really quickly iterate with kernel and/or modules customization (getting training from Toradex’s partners on Kernel development would be great here). Then, after settling on the kernel modifications, you could go to the next step which would be integrating it on a Torizon OS Yocto build to validate the development against the other elements of the OS.

But, in the end, if you are able to maintain and keep track of your development, I would be bold enough to say that it is good enough for you

Hopefully, my personal views are helpful in this discussion.

Best regards,

[matthijs]

@matthijs

Regarding your modify-kernel-and-bitbake-again approach, I do see one little mystery: How does the build system know that it should build the kernel again (and also without wiping and replacing the source directory)? I would think it keeps track of things it already built, and just skips over the kernel task in this case?

I’ve tried @gerko’s approach, making some changes to the kernel source in build-torizon/tmp/work-shared/verdin-imx8mp/kernel-source (introduced a compilation error for testing), and reran bitbake, but as I expected, it just skips over the kernel build task since it was already completed (even more, it just skips over all tasks, saying:

NOTE: Tasks Summary: Attempted 5923 tasks of which 5923 didn't need to be rerun and all succeeded.

So I’m wondering, did you maybe make additional changes to the build tree, maybe something in the bitbake layers/recipes that would prompt bitbake to build the kernel again?

@henrique.tx, thanks for your detailed reply, that is very helpful!

Yocto Using OStree directly Quick, but needs OStree knowledge

This was what I was looking at as well, but I could not quickly find any docs on how to do this. In particular, I wonder:

• Is there a way for yocto to create an ostree commit that I can somehow push to the the board directly, rather than yocto creating a tarball that I have to unpack again and them somehow push?
• How would pushing with ostree work? I read that you can use ostree pull to pull in commits that are served over http, but that seems a bit cumbersome. Maybe GitHub - dbnicholson/ostree-push: Push ostree commits over ssh could be helpful to push over ssh?
• How is the kernel image and initrd to use during startup determined? I noticed that some bootloader configuration is (looks like) generated by ostree deploy or similar (not sure where just now and I do not have a board handy to check), but I could not quite figure out what code is responsible for generating the bootloader configuration and where it takes its data (e.g. filenames) from.
  Initially, I looked at compiling a kernel separately as documented here, so that would likely generate a kernel version different from the one already in my image, so I would need to update the bootloader config as well. Now that I have built a yocto image from scratch, I could probably also upload that and then make further kernel updates by replacing the existing kernel image file (or modules), though.

@henrique.tx

Here I would like to comment on another possibility: maybe the easiest way of developing kernel-related stuff is to have a simpler image installed

I had been thinking about this as well, but was hoping to stick with the torizoncore image to minimize the changes compared to our final environment and not have to dive into these different images at all. But I see the point - the simpler images are easier to selectively update, I guess, and you probably also need them for doing NFS/TFTP booting (or can you also do that with an ostree/torizoncore image?). The netboot docs focus only on how to set up the booting and seems to not discuss where to get the kernel and rootfs (or in what form exactly).

@henrique.tx

Hopefully, my personal views are helpful in this discussion.

Definitely, it really helps direct my own investigation and efforts, so thanks already!

[henrique.tx]

Hi @matthijs !

@matthijs

Is there a way for yocto to create an ostree commit that I can somehow push to the the board directly, rather than yocto creating a tarball that I have to unpack again and them somehow push?

Please refer to the section OSTree Update From a Local Torizon OS Build from OSTree | Toradex Developer Center. You will see that the default build of Torizon OS on Yocto already outputs a ostree deployment

@matthijs

How would pushing with ostree work? I read that you can use ostree pull to pull in commits that are served over http, but that seems a bit cumbersome. Maybe GitHub - dbnicholson/ostree-push: Push ostree commits over ssh could be helpful to push over ssh?

Is the same place referenced in the article above, you will see that you can use TorizonCore Builder to deploy
If you need, here is TorizonCore Builder’s documentation as well: TorizonCore Builder Tool - Customizing Torizon OS Images | Toradex Developer Center. More specifically, the ostree serve command from TorizonCore Builder.

@matthijs

How is the kernel image and initrd to use during startup determined? I noticed that some bootloader configuration is (looks like) generated by ostree deploy or similar (not sure where just now and I do not have a board handy to check), but I could not quite figure out what code is responsible for generating the bootloader configuration and where it takes its data (e.g. filenames) from.

Taking a look at the OSTree section of this article might help you: Torizon Updates Technical Overview | Toradex Developer Center. To be honest, I do not know the details of how OSTree-related stuff goes down. If you really want to go deeper, you can go to U-Boot’s source code and search to environment variables that are related to OSTree (e.g. mentioned in OSTree | Toradex Developer Center).

@matthijs

I had been thinking about this as well, but was hoping to stick with the torizoncore image to minimize the changes compared to our final environment and not have to dive into these different images at all.

That’s true! With OSTree deployment, maybe developing kernel-related stuff is even easier, since you can maybe keep the “non-crashing one” in the rollback

@matthijs

Definitely, it really helps direct my own investigation and efforts, so thanks already!

Thanks for the feedback
You are most welcome. It is nice to think about this kind of stuff from time to time

Best regards,

[matthijs]

@henrique.tx

Please refer to the section OSTree Update From a Local Torizon OS Build from OSTree | Toradex Developer Center. You will see that the default build of Torizon OS on Yocto already outputs a ostree deployment

Thanks, that is exactly the docs I needed!

I’ve also further iterated on the approach suggested by @gerko of making changes directly in the yocto tmp kernel-source dir, and figured out how to get bitbake to pick up on the changes (with some help from this stackoverflow post). Here’s what I’m doing now:

• Make changes in (for the docker build) to yocto-workdir/torizon/build-torizon/tmp/work-shared/verdin-imx8mp/kernel-source (no need to commit them, uncommitted changes are picked up too)
• Force a kernel compile with bitbake -C compile virtual/kernel
• This already produces new modules-verdin-imx8mp.tgz and Image.gz in yocto-workdir/torizon/build-torizon/deploy/images (that maybe you could manually copy to the target system directly?)
• Build a full image with bitbake torizon-core-docker
• This produces a full TorizonCore image (e.g. torizon-core-docker-verdin-imx8mp-Tezi_6.5.0-devel-20231127103659+build.0.tar) in yocto-workdir/torizon/build-torizon/deploy/images, which can be (customized and) deployed through torizoncore-builder or other methods normally.

Note that both bitbake runs will show:

WARNING: /workdir/torizon/build-torizon/conf/../../layers/meta-toradex-nxp/recipes-kernel/linux/linux-toradex_5.15-2.2.x.bb:do_compile is tainted from a forced run

Which seems to be an indication that it is indeed using the manually compiled version.

[henrique.tx]

Hi @matthijs !

Thanks for sharing!

I see you marked you marked this topic as solved. Thanks!

Have a nice day!