Build TorizonCore image with extra drivers not working

Technical Support
, , , ,
1

Hello,

I followed the steps detailed in this link (https://developer.toradex.com/knowledge-base/build-torizoncore#Building_Torizon_Tools_Container) and afterwards, I ran bitbake -c menuconfig virtual/kernel so I could add one driver to the image. After that, i ran bitbake -f -c compile virtual/kernel. Finally, again bitbake torizon-core-docker.

After running bitbake torizon-core-dockerI took the deploy directory to the EasyInstaller to install the generated image into my SoM. Flashing went ok. Booting went normally (kernel 5.3 instead of 4.20), but when I want to run any docker container, I get the following:

colibri-imx6-05097180:~$ sudo docker pull torizon/arm32v7-debian-dev-tools
Using default tag: latest
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

I thought that was weird and tried several things:

colibri-imx6-05097180:~$ systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2019-11-07 17:13:17 UTC; 15min ago
     Docs: https://docs.docker.com
  Process: 791 ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=1/FAILURE)
 Main PID: 791 (code=exited, status=1/FAILURE)

Then, I tried:

colibri-imx6-05097180:~$ sudo dockerd
Password: 
INFO[2019-11-07T21:14:29.385845606Z] Starting up                                  
INFO[2019-11-07T21:14:29.400436983Z] libcontainerd: started new containerd process  pid=1560
INFO[2019-11-07T21:14:29.400915274Z] parsed scheme: "unix"                         module=grpc
INFO[2019-11-07T21:14:29.401106924Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2019-11-07T21:14:29.401378567Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] }  module=grpc
INFO[2019-11-07T21:14:29.401524887Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2019-11-07T21:14:29.401880189Z] pickfirstBalancer: HandleSubConnStateChange: 0x59c1120, CONNECTING  module=grpc
INFO[2019-11-07T21:14:29.540352955Z] starting containerd                           revision=fd103cb716352c7e19768e4fed057f71d68902a0.m version=v1.2.7-11-gfd103cb.m
INFO[2019-11-07T21:14:29.542654752Z] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2019-11-07T21:14:29.542928394Z] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
WARN[2019-11-07T21:14:29.551769280Z] failed to load plugin io.containerd.snapshotter.v1.aufs  error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found in directory /lib/modules/5.3.6-torizon-standard\n": exit status 1"
INFO[2019-11-07T21:14:29.552491549Z] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2019-11-07T21:14:29.553142492Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2019-11-07T21:14:29.554076076Z] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
WARN[2019-11-07T21:14:29.556022238Z] failed to load plugin io.containerd.snapshotter.v1.zfs  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
INFO[2019-11-07T21:14:29.556527859Z] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
WARN[2019-11-07T21:14:29.556985819Z] could not use snapshotter aufs in metadata plugin  error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found in directory /lib/modules/5.3.6-torizon-standard\n": exit status 1"
WARN[2019-11-07T21:14:29.557441445Z] could not use snapshotter zfs in metadata plugin  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
INFO[2019-11-07T21:14:29.558464688Z] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2019-11-07T21:14:29.559005641Z] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2019-11-07T21:14:29.559573924Z] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2019-11-07T21:14:29.560339523Z] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2019-11-07T21:14:29.560867809Z] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2019-11-07T21:14:29.561811059Z] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2019-11-07T21:14:29.562112366Z] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2019-11-07T21:14:29.562373676Z] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2019-11-07T21:14:29.562673317Z] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2019-11-07T21:14:29.562919961Z] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2019-11-07T21:14:29.563733223Z] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
INFO[2019-11-07T21:14:29.564252844Z] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2019-11-07T21:14:29.568109503Z] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[2019-11-07T21:14:29.568396144Z] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2019-11-07T21:14:29.568760779Z] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2019-11-07T21:14:29.568972760Z] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2019-11-07T21:14:29.569174076Z] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2019-11-07T21:14:29.569366392Z] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2019-11-07T21:14:29.569547043Z] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2019-11-07T21:14:29.569726693Z] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2019-11-07T21:14:29.569971672Z] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2019-11-07T21:14:29.570168988Z] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2019-11-07T21:14:29.570350638Z] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
INFO[2019-11-07T21:14:29.570770268Z] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2019-11-07T21:14:29.570990248Z] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2019-11-07T21:14:29.571185564Z] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2019-11-07T21:14:29.571363215Z] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2019-11-07T21:14:29.573640348Z] serving...                                    address="/var/run/docker/containerd/containerd-debug.sock"
INFO[2019-11-07T21:14:29.574134971Z] serving...                                    address="/var/run/docker/containerd/containerd.sock"
INFO[2019-11-07T21:14:29.574338953Z] containerd successfully booted in 0.037456s  
INFO[2019-11-07T21:14:29.614512737Z] pickfirstBalancer: HandleSubConnStateChange: 0x59c1120, READY  module=grpc
INFO[2019-11-07T21:14:29.628655487Z] parsed scheme: "unix"                         module=grpc
INFO[2019-11-07T21:14:29.630758968Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2019-11-07T21:14:29.631071607Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] }  module=grpc
INFO[2019-11-07T21:14:29.631238592Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2019-11-07T21:14:29.638609608Z] pickfirstBalancer: HandleSubConnStateChange: 0x5534ea0, CONNECTING  module=grpc
INFO[2019-11-07T21:14:29.645632987Z] pickfirstBalancer: HandleSubConnStateChange: 0x5534ea0, READY  module=grpc
INFO[2019-11-07T21:14:29.648934695Z] parsed scheme: "unix"                         module=grpc
INFO[2019-11-07T21:14:29.649503645Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2019-11-07T21:14:29.649983603Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] }  module=grpc
INFO[2019-11-07T21:14:29.650354237Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2019-11-07T21:14:29.651109837Z] pickfirstBalancer: HandleSubConnStateChange: 0x55350a0, CONNECTING  module=grpc
INFO[2019-11-07T21:14:29.655354128Z] pickfirstBalancer: HandleSubConnStateChange: 0x55350a0, READY  module=grpc
INFO[2019-11-07T21:14:29.677676823Z] [graphdriver] using prior storage driver: overlay2 
WARN[2019-11-07T21:14:29.689316461Z] Your kernel does not support cgroup rt period 
WARN[2019-11-07T21:14:29.689803418Z] Your kernel does not support cgroup rt runtime 
WARN[2019-11-07T21:14:29.690147721Z] Your kernel does not support cgroup blkio weight 
WARN[2019-11-07T21:14:29.690452694Z] Your kernel does not support cgroup blkio weight_device 
INFO[2019-11-07T21:14:29.691868902Z] Loading containers: start.                   
WARN[2019-11-07T21:14:29.825139128Z] Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: could not insert 'bridge': Invalid argument
modprobe: ERROR: could not insert 'br_netfilter': Invalid argument
insmod /lib/modules/5.3.6-torizon-standard/kernel/net/bridge/bridge.ko 
insmod /lib/modules/5.3.6-torizon-standard/kernel/net/bridge/bridge.ko 
, error: exit status 1 
INFO[2019-11-07T21:14:30.343304348Z] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
INFO[2019-11-07T21:14:30.473731491Z] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
INFO[2019-11-07T21:14:30.476510246Z] stopping healthcheck following graceful shutdown  module=libcontainerd
INFO[2019-11-07T21:14:30.476779888Z] stopping event stream following graceful shutdown  error="context canceled" module=libcontainerd namespace=plugins.moby
INFO[2019-11-07T21:14:30.479892947Z] pickfirstBalancer: HandleSubConnStateChange: 0x55350a0, TRANSIENT_FAILURE  module=grpc
INFO[2019-11-07T21:14:30.480084930Z] pickfirstBalancer: HandleSubConnStateChange: 0x55350a0, CONNECTING  module=grpc
failed to start daemon: Error initializing network controller: Error creating default "bridge" network: package not installed

In several posts I found many corrected this by just rebooting, but was not my case. Any ideas?

Other thing is that I created a wired.network file in /etc/systemd/network/ to configure a static IP address over eth0. This is not working and every boot I get other address from DHCP, until I run sudo systemctl restart systemd-networkd.services when it finally sets the IP address I defined inside wired.network. This is how I have configured my network settings on the default Torizon image. I do not know why this happens.

Any and all help will be appreciated.

2

Disabling the NetworkManager service using sudo systemctl disable NetworkManager.service can solve the IP issue.

3

Thank you for your response, @airpumpkin
Unfortunately, when I disable NetworkManager.service, network won’t come back up, even if I restart systemd-networkd.

4

It seems that Docker cannot load some necessary kernel modules:

WARN[2019-11-07T21:14:29.825139128Z] Running modprobe bridge br_netfilter failed with message: 
modprobe: ERROR: could not insert 'bridge': Invalid argument
modprobe: ERROR: could not insert 'br_netfilter': Invalid argument
insmod /lib/modules/5.3.6-torizon-standard/kernel/net/bridge/bridge.ko 
insmod /lib/modules/5.3.6-torizon-standard/kernel/net/bridge/bridge.ko

Honestly we haven’t tested bitbake -c menuconfig virtual/kernel. We make use of the Yocto kernel cache to configure the kernel: GitHub - toradex/toradex-kernel-cache at toradex_5.3.y

It should be possible to just add a defconfig to SRC_URI which then should take precedence. But it should contain all the necessary kernel configuration for Docker. If you decide to go this route, a good base is the defconfig from the mainline based kernel in meta-toradex-bsp-common. It has been generated using the above kernel metadata cache some weeks ago.

5

@jaimeibk out of interest, what driver are you missing? With TorizonCore we try to build an embedded Linux distribution which should have all the important drivers already enabled/pre-installed.

6

Thank you, @stefan.tx ! I want to include a capacitive touchscreen driver (silead_gsl1680) which is not built-in.

I will dive into your suggestion to give it a try, although I am not really sure I fully understood.

7

@jaimeibk, It’s very strange. I have done the following commands on my Colibri SOM with a customized Torizon image, and it did set the static IP on it after a restart.

cp ./10-static-eth0.network /etc/systemd/network/
systemctl stop NetworkManager
systemctl disable NetworkManager
8

Hm, I see, the touchscreen driver is available in the upstream kernel but currently not enabled:

$ zcat /proc/config.gz | grep SILE
# CONFIG_TOUCHSCREEN_SILEAD is not set

Will make sure it gets enabled in future releases.

10

@stefan.tx Thanks! When would that be? Just to keep it in mind. If you are including it, can I stay en route with other parts of development, then?

Regarding the docker part, maybe the whole menuconfig virtual/kernel might have messed something up? If I run everything up until building the torizon-core-docker, everything else should work, is that correct?

One somehow off-topic question: how come the TorizonCore image included in EasyInstaller comes with kernel 4.20? It just seems a bit odd and I got curious. With the final release of TorizonCore what kernel will be included?

11

@jaimeibk

This driver has been enabled and should appear in future builds. You can access our nightly builds for Torizon using the Toradex Easy Installer and enabling our “Continuous Integration” feeds.

The standard Torizon feed will be updated with the new stable release of Torizon based off of Yocto Zeus release.

12

Thanks, @jeremias.tx ! I will try it out early tomorrow.

13

@airpumpkin I know! But other strange things were going on too. Something along the way of the kernel modifications, I guess? Anyway, I appreciate your input.

14

@jeremias.tx I have tried two images from the continuous integration feed and run: zcat /proc/config.gz | grep SIL and still found #CONFIG_TOUCHSCREEN_SILEAD is not set

Can you advice which image should I try that has the driver enabled?

15

@jaimeibk I double-checked and the respective pull requests were still undergoing initial review. They seem to have now been merged:

By the end of the week at the latest I suspect there will be builds with these new configs. Otherwise updating your local yocto build and rebuilding will also yield the new configs.

16

Yes sorry, last nights build 1.1b1.6 should have the driver enabled. This has been built using our manifest from the zeus branch. Sorry for the confusing version number, we are working on improving this.

17

Hello, all!

I see the module now when I run zcat /proc/config-gz | grep SILE, it shows CONFIG_TOUCHSCREEN_SILEAD=m.

The thing is that when I try to modprobe it modprobe -v /lib/modules/5.3.10-torizon-standard/kernel/drivers/input/touchscreen/silead.ko, I get modprobe: FATAL: Module /lib/modules/5.3.10-torizon-standard/kernel/drivers/input/touchscreen/silead.ko not found in directory /lib/modules/5.3.10-torizon-standard

So…I went to see what’s inside /lib/modules/$(uname -r)/modules.dep, so I ran cat /lib/modules/$(uname -r)/modules.dep | grep sile, which outputs: kernel/drivers/input/touchscreen/silead.ko:

How else can I activate/enable the driver? I also tried insmod /lib/modules/$(uname -r)/kernel/drivers/input/touchscreen/silead.ko, and apparently it works, because when I then run ´lsmod´, it shows the silead module, but from that point I am not sure what I should do next. Can you point to where I should look?

18

Hi @jaimeibk,

The issue is that you passed modprobe an entire file path. Modprobe is somewhat intelligent in that it always searches in /lib/modules/$(uname -r). By passing the file path what you did was to tell modprobe to search for a module called /lib/modules/5.3.10-torizon-standard/kernel/drivers/input/touchscreen/silead.ko instead of just silead.ko. Insmod is lower level utility which is why it needs a full path to find modules.

So just using modprobe silead is enough, this command seems to work on my setup as the silead module shows up via lsmod.

Best Regards,
Jeremias

19

Thank you, @jeremias.tx !!

modprobe works alright now. Yet, somehow, touchscreen is not working.

After modprobe, the touchscreen should be working, right? It is most likely that my device tree overlay may be the reason why it does not work?

-Jaime

20

Assuming the hardware is all hooked up correctly this is the most likely issue. I can only recommend to check out the device tree bindings for this touchscreen which are documented in the kernel here: https://github.com/torvalds/linux/blob/master/Documentation/devicetree/bindings/input/touchscreen/silead_gsl1680.txt

There is also several other device trees in the mainline Linux kernel that use this driver/touchscreen that you can look at as examples.

21

I’ll fo ahead and re-check it.

Thanks!