TorizonCore-Builder workflow and automation

Technical Support
1

Hi all,
I’m working with Torizon for some weeks for a new project. Recently, the work gets more and more difficult with the Torizon environment. I’d like to figure out if I am doing something wrong or if we may have reached a level where Torizon is the wrong system (compared to Yocto).

Currently I’m mostly struggling with the build system, TorizonCore-Builder. I have two external kernel modules. They are needed for asterisk, one is the general driver for analog phones (dahdi) and a custom driver for a specific analog phone adapter. As I understand, the workflow is (please let me know if I miss something/do something wrong):

  1. Download the module sources manually
  2. Patch the module sources manually
  3. Build the modules with TorizonCore-Builder automatically
  4. Deploy the modules manually (should be done by TorizonCore-Builder, some not-yet investigated problem)

What I am missing is support to do everything automatically, i.e. downloading the module sources, patching them, and maybe adding a custom post-compilation step. In the end, I’d like to have one repository that defines all versions in my Linux image and one script/command to build the image.
My question is:

Do I use TorizonCore-Builder as intended?

Any help or input is appreciated.

2

Hi, I’m the Toradex AI Assistant. I’m in early beta, and I would appreciate your feedback.
The Toradex Support team will follow up with you shortly!

Did you like my answer?

:+1: :-1:

:handshake:

It sounds like you’re looking for a more automated and streamlined workflow for handling external kernel modules with TorizonCore-Builder. While TorizonCore-Builder is designed to simplify certain tasks, it may not cover all the automation you are seeking out-of-the-box. Here are some suggestions to help you achieve a more automated process:

  1. Automate Source Downloading and Patching:

    • You can create a script that automates the downloading and patching of your module sources. This script can be integrated into your build process.
    • Use tools like wget or curl for downloading and patch for applying patches.

  2. Integrate with TorizonCore-Builder:

    • Once you have your sources and patches ready, you can use TorizonCore-Builder to build the modules. You can create a custom script that first runs your download and patch script, and then invokes TorizonCore-Builder commands to build and deploy the modules.

  3. Custom Post-Compilation Steps:

    • If you need custom post-compilation steps, you can add these to your script as well. This could include additional configuration, packaging, or deployment steps.

  4. Single Repository and Build Script:

    • Maintain a single repository that includes your scripts, patches, and any other necessary files.
    • Create a master build script that orchestrates the entire process: downloading, patching, building, and deploying.

Here is a rough outline of what your master build script might look like:

#!/bin/bash

# Step 1: Download module sources
./download_sources.sh

# Step 2: Apply patches
./apply_patches.sh

# Step 3: Build modules with TorizonCore-Builder
torizoncore-builder build

# Step 4: Deploy modules
torizoncore-builder deploy

By structuring your workflow in this way, you can achieve a higher level of automation and consistency in your build process.

Best regards, Toradex AI Assistant

3

Hello @Urs,

Thank you for reaching out to us with your concerns.
May I ask which hardware and which version of Torizon OS you are using?

Anyway, I would like to point out to you that TorizonCore Builder is a tool that covers exactly this use case (along with many other use-cases) of building and adding an external kernel module.

With TorizonCore Builder you don’t have to patch the module sources manually since the tool will do this for you during its build process. The concept behind this tool is that you can take a vanilla Torizon OS image provided by us, and add your customizations on top, build the modified Torizon OS image using the builder tool. What you get as an output is your modified/custom Torizon OS image. You can use the tool in two different ways: either by using one configuration file (tcbuild.yaml file where you will list down your input image and all your additional modifications) and then build the output image. The other method is to use standalone commands of TorizonCore builder to add each of your modification and to build the final image eventually. I recommend the first approach as it is scalable and more convenient for you.

That being said and hoping that you understand the intention behind the TorizonCore Builder, I can point you to some articles on how to build your custom Torizon image with your external kernel modules added. Here is a section that shows a simple example of how your tcbuild.yaml file should look like: TorizonCore Builder Tool “build” command | Toradex Developer Center
Here you will find step by step instructions on how build the custom image: Building External Kernel Modules With Torizon | Toradex Developer Center

Please let me know if this helps.

4

@rudhi.tx thanks for the answer.

I am using Torizon OS 6 (torizon-core-docker-verdin-imx8mp-Tezi_6.7.0+build.18.tar) as a base. This is the latest production image I found.

I create the customized image using tcbuild.yaml and then torizoncore-builder build. The tcbuild.yaml is:

input:
  easy-installer:
    local: "images/torizon-core-docker-verdin-imx8mp-Tezi_6.7.0+build.18.tar"

customization:
  splash-screen: resources/my-boot-logo.png
  kernel:
    modules:
      - source-dir: dahdi-linux-current/dahdi-linux-3.4.0/
        autoload: no
      - source-dir: amfeltec_usb_4.0.5/
        autoload: yes

output:
  easy-installer:
    local: my-customized-image
    bundle:
      compose-file: docker-compose.yml

It seems that I got the idea how it is intended to be used, according to the examples/tutorials. It also works fine for the Hello World Kernel Module example.

There are separate hurdles I have right now:

Downloading the resources
As @ToradexAI suggested, I also came to the conclusion that I need to write my own script or look for other tools to download external resources like the 3rd party kernel modules.

Patching 3rd party kernel modules
The patching is needed for 3rd party kernel modules. To make both modules work, I have to patch the Makefile to adjust variables like KVERS, KSRC and KDIR. For dahdi-linux I have a bugfix- that is not yet in the release. The need to adjust the variables is probably a problem in the 3rd party modules, but that’s how it is. Adding custom patches for bugfixes or individual adjustments is a common use case.

Manual deployment of kernel modules
The kernel modules are compiled but not included in the image (checked with find / -name "*.ko" on the target). I haven’t investigated this yet but assume it is because there are multiple kernel modules or because they are in a further down subdirectory. This seems like more of a bug or not yet considered use case.

I am familiar with Yocto where all this issues can be handled. I am happy to learn if I do something wrong or have false assumptions because of my prior knowledge :nerd_face:.

My overall question is how do you recommend to work with this scenario:

  • Do I miss something, or use something in the wrong way?
  • How do you or other Torizon users get the resources needed to customize the image? Own script, a specific tool, manually, …?
  • Is this still in the use case of tcb?
  • What are the plans for Torizon/CoreBuilder? Will such scenarios be handled in tcb in the future (when)?
5

Hello @Urs,

It seems like you understood the workflow of tcb. That is very good.

  • How do you or other Torizon users get the resources needed to customize the image? Own script, a specific tool, manually, …?

Unfortunately, Torizoncore builder does not support an automated way to get third party resources currently. The only way would be to write your own script or to do it manually. We also don’t have a plan in the current roadmap to add this feature.

The kernel modules are compiled but not included in the image (checked with find / -name "*.ko" on the target).

Could you please try find /lib/modules/ -name "*.ko" | grep <name_of_module>

6

Thanks for the information about the tcb roadmap. That helps for further decisions.

On the build system, the modules are at:

  • amfeltec_usb_4.0.5/driver/amf_usb.ko
  • dahdi-linux-current/dahdi-linux-3.4.0/drivers/dahdi/dahdi.ko
    • There are 33 modules, but we only need the above mentioned

On the target, I don’t get any results with find /lib/modules/ -name "*.ko" | grep amf_usb.ko, nor for dahdi.ko.

Interestingly, tcb finds amf_usb.ko and marks it for autoload as seen at the end of the tcb log excerpt:

=>> Handling customization section
2024-08-27 08:26:32,945 - torizon.tcbuilder.cli.build - INFO - 
=> Setting splash screen
2024-08-27 08:26:32,986 - torizon.tcbuilder.cli.splash - INFO - splash screen merged to initramfs
2024-08-27 08:26:32,986 - torizon.tcbuilder.cli.build - INFO - 
=> Building module located at 'dahdi-linux-current/dahdi-linux-3.4.0/'
make: Entering directory '/workdir/dahdi-linux-current/dahdi-linux-3.4.0'
make -C drivers/dahdi/firmware firmware-loaders
make[1]: Entering directory '/workdir/dahdi-linux-current/dahdi-linux-3.4.0/drivers/dahdi/firmware'
make[1]: Leaving directory '/workdir/dahdi-linux-current/dahdi-linux-3.4.0/drivers/dahdi/firmware'
make -C /storage/linux/ M=/workdir/dahdi-linux-current/dahdi-linux-3.4.0/drivers/dahdi DAHDI_INCLUDE=/workdir/dahdi-linux-current/dahdi-linux-3.4.0/include DAHDI_MODULES_EXTRA=" " HOTPLUG_FIRMWARE=yes modules DAHDI_BUILD_ALL=m
make[1]: Entering directory '/storage/linux'
/workdir/dahdi-linux-current/dahdi-linux-3.4.0/drivers/dahdi/Kbuild:126: CPU Architecture 'arm64' does not support VPMADT032 or HPEC. Skipping.
  CC [M]  /workdir/dahdi-linux-current/dahdi-linux-3.4.0/drivers/dahdi/dahdi-base.o
...
  MODPOST /workdir/dahdi-linux-current/dahdi-linux-3.4.0/drivers/dahdi/Module.symvers
  CC [M]  /workdir/dahdi-linux-current/dahdi-linux-3.4.0/drivers/dahdi/dahdi.mod.o
  LD [M]  /workdir/dahdi-linux-current/dahdi-linux-3.4.0/drivers/dahdi/dahdi.ko
...
make[1]: Leaving directory '/storage/linux'
make: Leaving directory '/workdir/dahdi-linux-current/dahdi-linux-3.4.0'

2024-08-27 08:27:58,563 - torizon.tcbuilder.cli.kernel - INFO - Kernel module(s) successfully built and ready to deploy.
2024-08-27 08:27:58,564 - torizon.tcbuilder.cli.kernel - INFO - All kernel module(s) have been built and prepared.
2024-08-27 08:27:58,564 - torizon.tcbuilder.cli.build - INFO - 
=> Building module located at 'amfeltec_usb_4.0.5/'
make: Entering directory '/workdir/amfeltec_usb_4.0.5'

 +----------------- Checking Dahdi Sources -----------------+

   Compiling with NEW DAHDI Support!
   Dahdi Dir: /workdir/dahdi-linux-current/dahdi-linux-3.4.0/


 +----------------------------------------------------------+

make KBUILD_VERBOSE=0 -C /storage/linux/ M=/workdir/amfeltec_usb_4.0.5/driver EXTRA_FLAGS="-D__DAHDI__ -I/workdir/dahdi-linux-current/dahdi-linux-3.4.0//drivers/dahdi -I/workdir/dahdi-linux-current/dahdi-linux-3.4.0//include/dahdi -DDAHDI_VERSION=31 -I/storage/linux//include/linux -I/workdir/amfeltec_usb_4.0.5/include  -I/workdir/amfeltec_usb_4.0.5/firmware -I/workdir/amfeltec_usb_4.0.5/include/os -I/workdir/dahdi-linux-current/dahdi-linux-3.4.0//include/ -O -w -O" KBUILD_EXTRA_SYMBOLS="/workdir/dahdi-linux-current/dahdi-linux-3.4.0//drivers/dahdi/Module.symvers" modules
make[1]: Entering directory '/storage/linux'
  CC [M]  /workdir/amfeltec_usb_4.0.5/driver/amf_main.o
  CC [M]  /workdir/amfeltec_usb_4.0.5/driver/amf_usb_driver.o
  CC [M]  /workdir/amfeltec_usb_4.0.5/driver/amf_usb_core.o
  CC [M]  /workdir/amfeltec_usb_4.0.5/driver/amf_usb_cpu.o
  CC [M]  /workdir/amfeltec_usb_4.0.5/driver/amf_usb_dahdi.o
  CC [M]  /workdir/amfeltec_usb_4.0.5/driver/amf_usb_cp210x.o
  CC [M]  /workdir/amfeltec_usb_4.0.5/driver/amf_usb_daa.o
  CC [M]  /workdir/amfeltec_usb_4.0.5/driver/amf_usb_proslic.o
  CC [M]  /workdir/amfeltec_usb_4.0.5/driver/amf_usb_cpu_regs.o
  LD [M]  /workdir/amfeltec_usb_4.0.5/driver/amf_usb.o
  MODPOST /workdir/amfeltec_usb_4.0.5/driver/Module.symvers
  CC [M]  /workdir/amfeltec_usb_4.0.5/driver/amf_usb.mod.o
  LD [M]  /workdir/amfeltec_usb_4.0.5/driver/amf_usb.ko
make[1]: Leaving directory '/storage/linux'
make -C /workdir/amfeltec_usb_4.0.5/firmware EXTRA_FLAGS="EXTRA_UTIL_FLAGS"  
make[1]: Entering directory '/workdir/amfeltec_usb_4.0.5/firmware'
make[1]: 'usb_fw_up' is up to date.
make[1]: Leaving directory '/workdir/amfeltec_usb_4.0.5/firmware'
make: Leaving directory '/workdir/amfeltec_usb_4.0.5'

2024-08-27 08:28:25,273 - torizon.tcbuilder.cli.kernel - INFO - Kernel module(s) successfully built and ready to deploy.
2024-08-27 08:28:25,279 - torizon.tcbuilder.cli.kernel - INFO - amfeltec_usb_4.0.5/driver/amf_usb.ko is set to be autoloaded on boot.
2024-08-27 08:28:25,279 - torizon.tcbuilder.cli.kernel - INFO - All kernel module(s) have been built and prepared.
2024-08-27 08:28:25,279 - torizon.tcbuilder.cli.build - INFO -
7

Hi @Urs

Sorry for the late reply,
but were you able to solve your problem?

If you are still having any problem, could you please share the exactly steps you are taking?