Boot errors after using externally built kernel on Verdin iMX8MP + Mallow Carrier Board

Technical Support
,
1

Hello Toradex Support Team,

I’m working with the Verdin iMX8M Plus Quad 8GB Wi-Fi / Bluetooth IT on the Mallow Carrier Board V1.1, and I’m evaluating a custom driver development environment using a kernel built outside of Yocto. The official Yocto-based image boots and operates correctly, including the cinema application.

This is basically what I have done to build a new Yocto image as below:
$ repo init -u git://git.toradex.com/toradex-manifest.git -b scarthgap-7.x.y -m tdxref/default.xml
$ repo sync
$. export
$vi build/conf/local.conf
MACHINE ?= “verdin-imx8mp”
ACCEPT_FSL_EULA = “1”
$bitbake tdx-reference-multimedia-image
$~/oe-core/build/deploy/images/verdin-imx8mp$ tar xf Verdin-iMX8MP_Reference-Multimedia-Image-Tezi_7.3.0-devel-20250717025907+build.0.tar -C

However, I encountered boot-related issues when switching to a kernel built manually from the Toradex Git repository. I have two key questions:

Question 1: Boot errors with externally built kernel

I cloned the kernel source from:
$ git clone -b toradex_6.6-2.2.x-imx git://git.toradex.com/linux-toradex.git

Then I used the official kernel-config from:
“JFrog”

The steps I followed are:
$ cp kernel-config .config
$ make olddefconfig
$ make -j$(nproc) Image.gz
$ make -j$(nproc) modules
$ make modules_install INSTALL_MOD_PATH=modules
$ tar -czf modules.tar.gz modules

Then I copied Image.gz to /boot and extracted modules.tar.gz to / respectively on the target.

After rebooting, the following systemd errors occurred:
[FAILED] Failed to start BlueALSA service.
[FAILED] Failed to start Weston, a Wayland compositor, as a system service.
[DEPEND] Dependency failed for Start a wayland application.

systemctl --failed

  • bluealsa.service
  • bluetooth.service
  • weston.service
  • systemd-logind.service
  • systemd-timesyncd.service

Despite using the same kernel version and identical .config, these errors persist.
From what I can tell, the kernel boots, but critical userspace services fail to start.

Is there anything I might be missing when compiling or installing the kernel/modules externally?

Question 2: Boot error when restoring default Image.gz

To revert back, I had previously backed up the working Image.gz from Yocto:
'# cp Image.gz Image.gz.bk

However, when I restore it with:
'# cp Image.gz.bk Image.gz && sync
'# reboot

and then reboot (even with power-cycle), I encounter the same boot errors mentioned in Question 1.

Does this imply that the restored Image.gz is not being used properly? Or is something cached or overwritten during the module install?
Is there a better way to revert completely to the original kernel state without re-flashing the entire image via Easy Installer?

System Details

root@verdin-imx8mp-15722079:~# tdx-info

Software summary

Bootloader: U-Boot
Kernel version: 6.6.94-7.3.0-devel #1 SMP PREEMPT Wed Jun 25 09:14:55 UTC 2025
Kernel command line: root=PARTUUID=9376571f-02 ro rootwait console=tty1 console=ttymxc2,115200 quiet loglevel=3
Distro name: NAME=“TDX Wayland with XWayland”
Distro version: VERSION_ID=7.3.0-devel-20250717025907-build.0
Distro variant: -
Hostname: verdin-imx8mp-15722079

Hardware info

HW model: Toradex Verdin iMX8M Plus WB on Verdin Development Board
Toradex version: 0070 V1.1A
Serial number: 15722079
Processor arch: aarch64

Any advice or guidance would be appreciated to understand what’s causing the system service failures and how to properly manage kernel switching (manual vs. Yocto).

Thank you for your support.

2

Hi, I’m the Toradex AI Assistant.

I might make mistakes. Our support team will step in if needed. If I answered your question, please tick the “Solution” checkbox. For follow-up questions to me, please use @ToradexAI in your post.

Did you like my answer? :+1: :-1:

Hello! Thank you for your detailed description and clear questions regarding your kernel development workflow on the Verdin iMX8M Plus with the Mallow Carrier Board.

Let’s address your questions one by one:

1. Systemd/User service failures after manual kernel+module install

Even when using the “same” kernel config, critical userspace services like bluealsa, bluetooth, weston, and systemd-logind are failing after you manually build and deploy your kernel and modules. Here are key considerations:

  • Kernel–Userspace Symbol/API Mismatch:
    Even very minor build/config changes or missing modules can impact userspace functionality. Packages like BlueZ, Weston, and systemd-logind depend on kernel features, some requiring them built as modules (not built-in).

  • Module Directory Layout:
    Yocto deploys modules to /lib/modules/<kernel-version>/. When you run make modules_install, ensure that the <kernel-version> subdirectory exactly matches the output of uname -r on your running system.
    If your manually-built kernel’s EXTRAVERSION or LOCALVERSION does not match, userspace cannot find the right modules, resulting in service failures.

    • Check your kernel’s version string with uname -r.
    • Verify the modules are in /lib/modules/$(uname -r)/.
    • Running depmod -a after installing modules updates dependencies.

  • Module Mismatch/Caching:
    Disk contents in /lib/modules/ may contain a mix of modules from your prior and current kernels, especially after repeated installs. Userspace services may load unexpected or incompatible modules.

  • Kernel Firmware Dependencies:
    Some drivers (Wi-Fi, Bluetooth, GPU, etc.) require firmware in /lib/firmware/. If your new kernel expects different firmware, this can cause failures—review dmesg/kernel logs for missing firmware errors.

  • Additional Tips:

    • After manually installing modules, always run depmod -a.
    • Reboot cleanly after kernel+module replacement.
    • Check for errors in dmesg, journalctl -xe for hints on what is missing.

For more on software/hardware specifics:

2. Restoring Official Image.gz but Persistent Failures

If restoring the original Image.gz and rebooting still triggers the same failures, the likely cause is leftover modules from your previous install. Here’s why:

  • Modules and Kernel Image Must Be in Sync:
    It’s not enough to restore just the kernel image. You must ensure that /lib/modules/<kernel-version>/ contains only the original modules that correspond to the restored kernel image.

  • Yocto’s Module Packaging:
    Yocto places your original modules in /lib/modules/$(uname -r)/, likely as part of the Tezi package. If you extracted custom modules over this, some original modules may be overwritten or deleted.

  • Clean Revert Procedure:

    1. Restore both Image.gz and the /lib/modules/<kernel-version>/ directory from a backup (ideally the Yocto-generated one).
    2. Run depmod -a after restoring.
    3. Reboot.

    If you do not have a backup of /lib/modules/, re-extract the kernel modules from the original Yocto Tezi tarball to the root (/), as this is how the reference images are deployed.

3. Best-Practices for Manual Kernel Development Workflow

  • Always treat the kernel image and its modules as a matched set; swap both, never just one.
  • Avoid mixing manual and Yocto workflows on the same rootfs without a clear backup/restore plan.
  • For experimentation, consider copying the new kernel and modules into a new /lib/modules/ version (changing the kernel LOCALVERSION is helpful). This keeps previous versions intact and simplifies rollback.
  • Review Toradex’s kernel deployment documentation for the Verdin iMX8M Plus.

Summary Checklist

  • Ensure /boot/Image.gz and /lib/modules/<kernel-version>/ always correspond.
  • Always run depmod -a after changing modules.
  • Check for missing modules with lsmod, dmesg, and journalctl.
  • Restore both kernel and modules when reverting, not just the kernel image.
  • For quick revert, unpack the official module tarball from Tezi back to your root filesystem.

If you follow these steps and see continued failures, please provide logs from dmesg and journalctl -xe to further isolate the failing modules or services.

Best regards,
Toradex AI Assistant.

3

Hi @sangyoon!

If I’m not mistaken, you are using our BSP image. Did you install it through the Toradex Easy Installer feeds, or did you build it manually?

Besides, we’ll need some logs to debug this issue. Could you share the systemctl status command output of those services that failed? In addition, could you see if there is any related log in the dmesg log?

Best regards.

4

Hi Lucas,

After completing the Yocto build, I installed the resulting image (Verdin-iMX8MP_Reference-Multimedia-Image-Tezi_7.3.0-devel-20250717025907+build.0.tar) using the Toradex Easy Installer feeds.
Everything works correctly, and the system boots successfully. After boot, the log is saved as 6.6.94-7.3.0-devel.log.

Following the Toradex kernel documentation, I cloned the Linux kernel source using:
git clone -b toradex_6.6-2.2.x-imx git://git.toradex.com/linux-toradex.git

I verified that the kernel version from the Yocto build and the git repository are the same. Here’s the git log output:
commit 5a62539de8f9c743168639237414c69e2b316914 (HEAD)
Author: Francesco Dolcini francesco.dolcini@toradex.com
Date: Wed Jun 25 11:00:55 2025 +0200

I then built and installed the new Image.gz and kernel modules, and rebooted the system.

The log from this boot attempt is saved as 6.6.94+.log.
6.6.94+.log (42.8 KB)
6.6.94-7.3.0-devel.log (42.5 KB)

Let me know if you’d like to add more detail (e.g., attach the .log files or describe the boot errors). I can help prepare that as well.

5

Hi @sangyoon!

May I kindly ask you to send the status of each server that failed? You can do that by using systemctl status <service-name>, for example systemctl status bluetooth.service.

Best regards.

6

Hi Lucas.

I have responded to your request.
When I run systemctl --failed, there are a total of 6 failed services.
I am attaching the details of each service failure in the file systemctl_failed.log.

Best regards.
systemctl_failed.log (6.4 KB)

7

Hi @sangyoon!

I think the log file you sent is partially missing (possibly because you copied it directly from the terminal, which truncated the full message).

I tested your use case, and it worked fine. Below you can find the commands that I used after building the kernel:

scp arch/arm64/boot/Image.gz root@192.168.11.1:/boot/

mkdir _install

make modules_install INSTALL_MOD_PATH=_install >/dev/null

rm -f modules.tgz

tar czf modules.tgz --owner=0 --group=0 -C _install .

scp modules.tgz root@10.22.1.94:/var/tmp

Then, in the module’s terminal:

tar xf /var/tmp/modules.tgz -C /

sync

reboot

After that, the image booted, and there were no systemctl failures.

I used a pre-built reference multimedia image from Toradex Easy Installer instead of building my own with Yocto, and, as far as I understand, it should be the same if you didn’t make any modifications to it.

Please, try this method and see if it works, and if it doesn’t, please try with a pre-built image too.

Feel free to ask if you face any doubts or issues.

Best regards.

8

Hi @lucas_az.tx

I have reviewed the information you provided.

My mistake was that, when creating the tar archive, I did not include the --owner=0 --group=0 options.
After adding these options to module.tgz and installing it on the iMX8, the system operates normally.

When building linux-toradex, the generated Image.gz has the following version:

Linux verdin-imx8mp-15722079 6.6.94+ #7 SMP PREEMPT Fri Aug 8 11:54:07 KST 2025 aarch64 GNU/Linux

After installing module.tgz, the directory /lib/modules/6.6.94+ is created.

However, with the content you provided, the following errors occur during system boot, and the cinema application does not start.
I have attached the boot error log below.
systemctl_status_weston.log (4.0 KB)

When I run journalctl -p err, I see the error “Failed to find module ‘galcore’”.
I am currently checking whether this is the cause of the Weston startup failure.

I would like to try building galcore.ko for the 6.6.94+ kernel version, but I encounter the following issues:

  1. Build error from git source
    $ git clone -b ti-5.0.11.p7-k6.12 https://git.ti.com/git/graphics/ti-gc320-driver.git
    $ make -f Kbuild AQROOT=$(pwd) KERNEL_DIR=$KERNEL_DIR CROSS_COMPILE=aarch64-tdx-linux- ARCH_TYPE=arm64
    A build error occurs, but I’m not sure if it’s due to a configuration issue or incompatibility.

  2. Build error in Yocto
    When I run:
    bitbake ti-gc320-driver
    in Yocto, I also get a build error.

Could you please provide guidance on how to build the ti-gc320-driver (galcore.ko) correctly for kernel version 6.6.94+?

Additionally, I would like your advice on how to identify the exact cause of the Weston service startup failure. Are there specific logs, debug commands, or configuration checks you recommend for pinpointing this issue?

Best regards.

9

Hi @sangyoon!

The ti-gc320-driver is not responsible for galcore. Instead, galcore is a proprietary driver of NXP, and it loads closed code firmwares.

May I kindly ask you why you are loading the kernel and the modules manually?

In my humble opinion, it is easier to use the image built by Yocto since it can be quite difficult to build and load proprietary binaries, as galcore, and the simplest mistake can cause several errors and failures.
Here is the Yocto recipe that builds it.

Another way to use the Verdin GPU is by using the upstream image, which uses etnaviv driver. The performance could be a little worse, but it also works fine.

Best regards.

10

Hi @lucas_az.tx

I fully agree with your opinion.
I raised my questions in order to review the development environment.

The current progress is as follows:

  • The weston.service failure seems to have been caused by the failure to load galcore.ko on kernel version 6.6.94+.
  • I rebuilt galcore.ko for kernel 6.6.94+, and confirmed that weston.service now works correctly.
  • The cinema application also runs without issues.

The review of building the kernel from an external source environment has now been completed.

Thank you for your response to my questions.

Best regards.

11

Hi @sangyoon!

I’m glad you could solve this issue. I’ll mark your answer as the solution.

Best regards.

12

Hi @lucas_az.tx

During my review of the Toradex Linux kernel, I referred to the following documentation:
Build the Linux Kernel from Source Code.

In the section Linux Kernel Module Compilation and Deployment, the instructions currently use:

$ cd modules
$ tar -czf ../modules.tar.gz .

I recommend changing this to:

$ tar czf modules.tgz --owner=0 --group=0 -C modules .

because the --owner=0 --group=0 options are required to ensure correct ownership when extracting the modules on the target device.

Best Regards.

1 Like
13

Hi @sangyoon!

Thanks for the heads-up, I’ll talk to the responsible team.

Best regards.