Debug drivers on Torizon OS

Hi everyone,

I’m currently working on adding support for an IC whose driver is already built into the kernel. This built-in driver interacts with several loaded modules. So far, I’ve been able to use TCB to apply my custom overlay and modify the device tree as needed.

Since I’m new to kernel debugging, I’d like to set up an environment where I can debug the drivers line by line. My goal is to monitor the values being parsed from the device tree and track how those values are being set and used by the drivers.

I understand that I’ll need to build the kernel from source. However, I’m looking for guidance on how to properly set up a debugging environment where I can:

  • Debug Driver A and seamlessly jump to Driver B and back, following the execution flow.
  • Ensure my custom overlays and device tree are applied before the debugging begins.
  • Watch variables and inspect device tree values during runtime.

Any suggestions, best practices, or references to documentation would be greatly appreciated!

Best regards,
Samarth

Hi, I’m the Toradex AI Assistant.

I might make mistakes. Our support team will step in if needed. If I answered your question, please tick the “Solution” checkbox. For follow-up questions to me, please use @ToradexAI in your post.

Did you like my answer? :+1: :-1:


Hello Samarth,

Thank you for your detailed question! Setting up a kernel debugging environment on Toradex hardware, including the application of custom device tree overlays, is a common requirement when getting deeply involved with driver development and integration. Let’s break down the process and give you some clear steps and references.


1. Build the Kernel from Source

To enable meaningful debugging, you’ll want to build your kernel (and any modules you’ll debug) with debugging symbols (CONFIG_DEBUG_INFO=y). Here’s how to get started:

  • Set up cross-compilation and fetch the source
    Follow the official guide for kernel compilation prerequisites.
  • Clone the kernel and device tree overlays
    The overlays should match the kernel branch you’re using. Example:
    cd ~/workdir
    git clone -b <branch> git://git.toradex.com/device-tree-overlays.git
    cd device-tree-overlays/
    
    Refer to: Building Device Tree Overlays from Source.

2. Build and Integrate Custom Device Tree Overlays

  • Compile your overlays
    cd overlays/
    export STAGING_KERNEL_DIR=/home/user/workdir/linux-toradex
    make
    
  • Edit the overlays Makefile to add your custom overlay:
    dtb-y += my-custom-overlay.dtbo
    
    Reference: Build Device Tree Overlays

3. Deploy Overlays for Testing

  • Copy your .dtbo files to the overlays/ directory in the boot partition.
  • Edit the overlays.txt file and add your overlay filename:
    fdt_overlays=my-custom-overlay.dtbo
    
    Save, run sync, and reboot for changes to take effect.
  • See: Deploy a Device Tree Overlay

4. Setting up Kernel Debugging

To debug kernel drivers line-by-line and watch variables, you have several options:

A. KGDB (Kernel GNU Debugger)

  • KGDB allows source-level debugging using gdb over a serial console or ethernet (netconsole).
  • You’ll need to enable kgdb options in your kernel config (CONFIG_KGDB, CONFIG_KGDB_SERIAL_CONSOLE, etc.).
  • You connect to the target board using gdb on your host.

B. Using QEMU for Emulation

  • For initial exploration, you can boot your kernel in QEMU (if your hardware/SoC is supported) and use gdb natively.

C. JTAG Debugging

  • JTAG interface allows full hardware debugging, but this requires additional hardware and configuration.

D. Dynamic Debug and printk

  • For simpler tracing, enable dynamic_debug and use printk for tracing code flow and examining variable values.

Best Practices:

  • Build with symbols: Add CONFIG_DEBUG_INFO=y in your kernel config.
  • Install Kernel Sources & Symbols: Ensure your debug symbols are on the host, and gdb knows the source path.
  • Prepare your overlays and ensure they’re loaded at boot: This sets up the device tree before any driver initializes.

Jumping between Driver A and B:
When using gdb with kgdb or (hardware/JTAG), you can set breakpoints in both driver sources and step through code as execution passes through them.


5. Watching Variables and Device Tree Properties

  • Device tree properties are available in the kernel via the device node’s platform_device or of_device_id structures.
  • With gdb, you can inspect these in memory as drivers are probed.
  • You can also inspect the parsed device tree at runtime via /proc/device-tree/ or debug code paths parsing these values in your drivers.

References


Summary:

  • Build your kernel and overlays from source with debug options.
  • Deploy overlays before boot.
  • Use kgdb or JTAG for full source-level debugging.
  • Set breakpoints, watch variables, and traverse multiple drivers.
  • Use Toradex’s documentation for overlays and device tree management.

If you need detailed steps for setting up kgdb/gdb for your specific module, feel free to provide your hardware/software versions and I’ll provide a tailored example!

Best regards,
Toradex AI Assistant.

Hi @s.gour!

The answer given by Toradex AI seems reasonable and is a good starting point. Could you please try it and see if it fulfills your expectations?

Best regards.