Docker: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/a...ca-init/merged: too many levels of symbolic links

Technical Support
, ,
1

Colibri iMX6DL 512 MB
Colibri Evaluation Board

When I build a release container (which is compiled in debug mode by the way) and deploy it I sometimes get this message:

docker: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/aabfa8a4e29abe383923da77ad1b0ab85e2540eaef116d28b4a71c64afedadca-init/merged: too many levels of symbolic links.

Often I can get away without this problem, but once I get it I can’t get rid of it unless I re-flash the Torizon OS to my module.

Anyone else having this problem?

2

Greetings @leighjboyd,

That’s an odd error. We’ve seen a similar error in the past very rarely, but have not seen it recently.

Just some questions to clarify:

  • This occurs when you deploy your container image from our IDE extension to your device running TorizonCore, correct? Also this is just during deployment the container is not even running yet, correct?
  • What version of TorizonCore are you running on the device?
  • What version of the Colibri i.MX6DL do you have?
  • Is this issue easily reproducible on demand? The way you worded it makes it sound like this happens somewhat randomly, but I just wanted to make sure.

Best Regards,
Jeremias

3

this occurs when i do an
f1:build release container: release
f1:deploy release container

then I run a script:
source ./EmeraAutostartup.sh

which contains:


#! /bin/bash

docker run -d --privileged --network 'host' --volume /dev:/dev --volume /home/torizon/logs:/home/torizon/logs --volume /run/udev:/run/udev --volume /tmp:/tmp --volume ContainerVolume:/NewVolume --device /dev/spidev3.0:/dev/spidev3.0 --device /dev/colibri-i2c:/dev/colibri-i2c --device /dev/gpiochip0:/dev/gpiochip0 --device /dev/gpiochip1:/dev/gpiochip1 --device /dev/gpiochip2:/dev/gpiochip2 --device /dev/gpiochip3:/dev/gpiochip3 --device /dev/spidev3.1:/dev/spidev3.1 --publish 8080:8080/tcp emerablockenergybox_arm32v7-debian-no-ssh_bullseye_release_69ac5093-51e4-4f03-8827-ec9f9390dfe3

The original exported docker run command is:
docker run --network ‘host’ --volume /dev:/dev --volume /home/torizon/logs:/home/torizon/logs --volume /run/udev:/run/udev --volume /tmp:/tmp --volume ContainerVolume:/NewVolume --device /dev/spidev3.0:/dev/spidev3.0 --device /dev/colibri-i2c:/dev/colibri-i2c --device /dev/gpiochip0:/dev/gpiochip0 --device /dev/gpiochip1:/dev/gpiochip1 --device /dev/gpiochip2:/dev/gpiochip2 --device /dev/gpiochip3:/dev/gpiochip3 --device /dev/spidev3.1:/dev/spidev3.1 --publish 8080:8080/tcp emerablockenergybox_arm32v7-debian-no-ssh_bullseye_release_69ac5093-51e4-4f03-8827-ec9f9390dfe3

(which also doesn’t seem to work)

dmesg shows an error:
colibri-imx6-10934587:~$ dmesg
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 5.4.161-5.6.0+git.ed9c66090d11 (oe-user@oe-host) (gcc version 9.3.0 (GCC)) #1-TorizonCore SMP Tue Feb 15 10:27:40 UTC 2022
[ 0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=50c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] OF: fdt: Machine model: Toradex Colibri iMX6DL/S on Colibri Evaluation Board V3
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] efi: UEFI not found.
[ 0.000000] cma: Reserved 64 MiB at 0x2c000000
[ 0.000000] On node 0 totalpages: 131072
[ 0.000000] Normal zone: 1152 pages used for memmap
[ 0.000000] Normal zone: 0 pages reserved
[ 0.000000] Normal zone: 131072 pages, LIFO batch:31
[ 0.000000] percpu: Embedded 20 pages/cpu s51212 r8192 d22516 u81920
[ 0.000000] pcpu-alloc: s51212 r8192 d22516 u81920 alloc=20*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 129920
[ 0.000000] Kernel command line: enable_wait_mode=off galcore.contiguousSize=50331648 root=LABEL=otaroot rootfstype=ext4 quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash fbcon=map:3 ostree=/ostree/boot.1/torizon/f12960b1fc481a21998f1e6dbd8722d06dffa8eda439d7a43fd9fc1785c66664/0
[ 0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] Memory: 429700K/524288K available (8192K kernel code, 900K rwdata, 4192K rodata, 1024K init, 435K bss, 29052K reserved, 65536K cma-reserved, 0K highmem)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] rcu: Hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[ 0.000000] L2C-310 erratum 769419 enabled
[ 0.000000] L2C-310 enabling early BRESP for Cortex-A9
[ 0.000000] L2C-310 full line of zeros enabled for Cortex-A9
[ 0.000000] L2C-310 ID prefetch enabled, offset 16 lines
[ 0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
[ 0.000000] L2C-310 cache controller enabled, 16 ways, 512 kB
[ 0.000000] L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x76450001
[ 0.000000] random: get_random_bytes called from start_kernel+0x287/0x40c with crng_init=0
[ 0.000000] Switching to timer-based delay loop, resolution 333ns
[ 0.000006] sched_clock: 32 bits at 3000kHz, resolution 333ns, wraps every 715827882841ns
[ 0.000020] clocksource: mxc_timer1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 637086815595 ns
[ 0.001681] Console: colour dummy device 80x30
[ 0.001693] printk: console [tty0] enabled
[ 0.001718] Calibrating delay loop (skipped), value calculated using timer frequency… 6.00 BogoMIPS (lpj=30000)
[ 0.001728] pid_max: default: 32768 minimum: 301
[ 0.001910] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[ 0.001919] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[ 0.002804] CPU: Testing write buffer coherency: ok
[ 0.002834] CPU0: Spectre v2: using BPIALL workaround
[ 0.003633] Setting up static identity map for 0x10100000 - 0x10100054
[ 0.003779] rcu: Hierarchical SRCU implementation.
[ 0.007322] EFI services will not be available.
[ 0.007665] smp: Bringing up secondary CPUs …
[ 0.008363] CPU1: Spectre v2: using BPIALL workaround
[ 0.008497] smp: Brought up 1 node, 2 CPUs
[ 0.008506] SMP: Total of 2 processors activated (12.00 BogoMIPS).
[ 0.008510] CPU: All CPU(s) started in SVC mode.
[ 0.008964] devtmpfs: initialized
[ 0.019738] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[ 0.019977] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.019992] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[ 0.023407] pinctrl core: initialized pinctrl subsystem
[ 0.024360] NET: Registered protocol family 16
[ 0.032015] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.033038] cpuidle: using governor ladder
[ 0.033060] cpuidle: using governor menu
[ 0.033443] CPU identified as i.MX6DL, silicon rev 1.4
[ 0.041403] vdd1p1: supplied by regulator-dummy
[ 0.041881] vdd3p0: supplied by regulator-dummy
[ 0.042351] vdd2p5: supplied by regulator-dummy
[ 0.042800] vddarm: supplied by regulator-dummy
[ 0.043430] vddpu: supplied by regulator-dummy
[ 0.043739] vddsoc: supplied by regulator-dummy
[ 0.053193] No ATAGs?
[ 0.053284] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[ 0.053290] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.054129] debugfs: Directory ‘dummy-iomuxc-gpr@20e0000’ with parent ‘regmap’ already present!
[ 0.054531] imx6dl-pinctrl 20e0000.iomuxc: initialized IMX pinctrl driver
[ 0.055688] Serial: AMBA PL011 UART driver
[ 0.066490] cryptd: max_cpu_qlen set to 1000
[ 0.082988] mxs-dma 110000.dma-apbh: initialized
[ 0.084483] vgaarb: loaded
[ 0.084967] SCSI subsystem initialized
[ 0.085172] libata version 3.00 loaded.
[ 0.085408] usbcore: registered new interface driver usbfs
[ 0.085454] usbcore: registered new interface driver hub
[ 0.085497] usbcore: registered new device driver usb
[ 0.085636] usb_phy_generic usbphynop1: usbphynop1 supply vcc not found, using dummy regulator
[ 0.085816] usb_phy_generic usbphynop2: usbphynop2 supply vcc not found, using dummy regulator
[ 0.090265] stmpe-i2c 0-0041: stmpe811 detected, chip id: 0x811
[ 0.189258] i2c i2c-0: IMX I2C adapter registered
[ 0.189969] i2c i2c-1: IMX I2C adapter registered
[ 0.190130] mc: Linux media interface: v0.10
[ 0.190169] videodev: Linux video capture interface: v2.00
[ 0.190285] pps_core: LinuxPPS API ver. 1 registered
[ 0.190290] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti giometti@linux.it
[ 0.190306] PTP clock support registered
[ 0.190557] Advanced Linux Sound Architecture Driver Initialized.
[ 0.191904] Bluetooth: Core ver 2.22
[ 0.191948] NET: Registered protocol family 31
[ 0.191952] Bluetooth: HCI device and connection manager initialized
[ 0.191962] Bluetooth: HCI socket layer initialized
[ 0.191970] Bluetooth: L2CAP socket layer initialized
[ 0.191987] Bluetooth: SCO socket layer initialized
[ 0.192711] clocksource: Switched to clocksource mxc_timer1
[ 0.723501] VFS: Disk quotas dquot_6.6.0
[ 0.723584] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[ 0.731890] thermal_sys: Registered thermal governor ‘step_wise’
[ 0.732277] NET: Registered protocol family 2
[ 0.732416] IP idents hash table entries: 8192 (order: 4, 65536 bytes, linear)
[ 0.733278] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear)
[ 0.733308] TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear)
[ 0.733351] TCP bind hash table entries: 4096 (order: 3, 32768 bytes, linear)
[ 0.733415] TCP: Hash tables configured (established 4096 bind 4096)
[ 0.733494] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[ 0.733523] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[ 0.733722] NET: Registered protocol family 1
[ 0.734313] RPC: Registered named UNIX socket transport module.
[ 0.734319] RPC: Registered udp transport module.
[ 0.734322] RPC: Registered tcp transport module.
[ 0.734326] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 0.734338] PCI: CLS 0 bytes, default 64
[ 0.734522] Trying to unpack rootfs image as initramfs…
[ 1.399441] Freeing initrd memory: 7864K
[ 1.400523] hw perfevents: no interrupt-affinity property for /pmu, guessing.
[ 1.400715] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[ 1.404020] Initialise system trusted keyrings
[ 1.404201] workingset: timestamp_bits=14 max_order=17 bucket_order=3
[ 1.410858] NFS: Registering the id_resolver key type
[ 1.410885] Key type id_resolver registered
[ 1.410890] Key type id_legacy registered
[ 1.453823] NET: Registered protocol family 38
[ 1.453840] Key type asymmetric registered
[ 1.453846] Asymmetric key parser ‘x509’ registered
[ 1.453896] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
[ 1.454015] io scheduler mq-deadline registered
[ 1.454021] io scheduler kyber registered
[ 1.462120] pfuze100-regulator 0-0008: Full layer: 2, Metal layer: 1
[ 1.462890] pfuze100-regulator 0-0008: FAB: 0, FIN: 0
[ 1.462900] pfuze100-regulator 0-0008: pfuze100 found.
[ 1.530964] 2020000.serial: ttymxc0 at MMIO 0x2020000 (irq = 25, base_baud = 5000000) is a IMX
[ 1.539768] printk: console [ttymxc0] enabled
[ 1.541041] 21e8000.serial: ttymxc1 at MMIO 0x21e8000 (irq = 70, base_baud = 5000000) is a IMX
[ 1.541818] 21ec000.serial: ttymxc2 at MMIO 0x21ec000 (irq = 71, base_baud = 5000000) is a IMX
[ 1.552186] etnaviv etnaviv: bound 130000.gpu (ops gpu_ops)
[ 1.552413] etnaviv etnaviv: bound 134000.gpu (ops gpu_ops)
[ 1.552425] etnaviv-gpu 130000.gpu: model: GC880, revision: 5106
[ 1.552639] etnaviv-gpu 134000.gpu: model: GC320, revision: 5007
[ 1.553204] [drm] Initialized etnaviv 1.3.0 20151214 for etnaviv on minor 0
[ 1.555138] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[ 1.555143] [drm] No driver support for vblank timestamp query.
[ 1.555310] imx-drm display-subsystem: bound imx-ipuv3-crtc.2 (ops ipu_crtc_ops)
[ 1.555418] imx-drm display-subsystem: bound imx-ipuv3-crtc.3 (ops ipu_crtc_ops)
[ 1.555671] [drm] Initialized imx-drm 1.0.0 20120507 for display-subsystem on minor 1
[ 1.555714] imx-ipuv3 2400000.ipu: IPUv3H probed
[ 1.564840] brd: module loaded
[ 1.573554] loop: module loaded
[ 1.574162] zram: Added device: zram0
[ 1.578238] spi_imx 2014000.spi: probed
[ 1.579140] libphy: Fixed MDIO Bus: probed
[ 1.581030] pps pps0: new PPS source ptp0
[ 1.582091] libphy: fec_enet_mii_bus: probed
[ 1.588798] fec 2188000.ethernet eth0: registered PHC device 0
[ 1.589060] ehci_hcd: USB 2.0 ‘Enhanced’ Host Controller (EHCI) Driver
[ 1.589088] ehci-pci: EHCI PCI platform driver
[ 1.589148] ehci-mxc: Freescale On-Chip EHCI Host driver
[ 1.590220] usbcore: registered new interface driver usb-storage
[ 1.590306] usbcore: registered new interface driver usbserial_generic
[ 1.590331] usbserial: USB Serial support registered for generic
[ 1.592314] imx_usb 2184000.usb: 2184000.usb supply vbus not found, using dummy regulator
[ 1.592584] imx_usb 2184200.usb: No over current polarity defined
[ 1.595681] snvs_rtc 20cc000.snvs:snvs-rtc-lp: registered as rtc1
[ 1.595825] i2c /dev entries driver
[ 1.598565] imx2-wdt 20bc000.wdog: timeout 60 sec (nowayout=0)
[ 1.599447] sdhci: Secure Digital Host Controller Interface driver
[ 1.599451] sdhci: Copyright(c) Pierre Ossman
[ 1.599503] Synopsys Designware Multimedia Card Interface Driver
[ 1.599974] sdhci-pltfm: SDHCI platform and OF driver helper
[ 1.602022] sdhci-esdhc-imx 2190000.usdhc: Got CD GPIO
[ 1.635817] mmc0: SDHCI controller on 2190000.usdhc [2190000.usdhc] using ADMA
[ 1.670387] mmc1: SDHCI controller on 2198000.usdhc [2198000.usdhc] using ADMA
[ 1.672463] ledtrig-cpu: registered to indicate activity on CPUs
[ 1.674649] caam 2100000.caam: Entropy delay = 3200
[ 1.735441] caam 2100000.caam: Instantiated RNG4 SH0
[ 1.796202] caam 2100000.caam: Instantiated RNG4 SH1
[ 1.796213] caam 2100000.caam: device ID = 0x0a16010000000100 (Era 4)
[ 1.796222] caam 2100000.caam: job rings = 2, qi = 0
[ 1.804759] caam algorithms registered in /proc/crypto
[ 1.808550] caam_jr 2101000.jr0: registering rng-caam
[ 1.809672] usbcore: registered new interface driver usbhid
[ 1.809677] usbhid: USB HID core driver
[ 1.813345] usbcore: registered new interface driver snd-usb-audio
[ 1.815654] sgtl5000 0-000a: sgtl5000 revision 0x11
[ 1.833985] random: fast init done
[ 1.848613] fsl-ssi-dai 2028000.ssi: No cache defaults, reading back from HW
[ 1.851770] NET: Registered protocol family 10
[ 1.852836] Segment Routing with IPv6
[ 1.852969] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[ 1.853593] NET: Registered protocol family 17
[ 1.853667] Key type dns_resolver registered
[ 1.855716] Registering SWP/SWPB emulation handler
[ 1.857686] registered taskstats version 1
[ 1.857691] Loading compiled-in X.509 certificates
[ 1.875896] imx_usb 2184000.usb: 2184000.usb supply vbus not found, using dummy regulator
[ 1.899240] mmc1: new DDR MMC card at address 0001
[ 1.900450] mmcblk1: mmc1:0001 S40004 3.64 GiB
[ 1.900963] mmcblk1boot0: mmc1:0001 S40004 partition 1 4.00 MiB
[ 1.901461] mmcblk1boot1: mmc1:0001 S40004 partition 2 4.00 MiB
[ 1.901921] mmcblk1rpmb: mmc1:0001 S40004 partition 3 4.00 MiB, chardev (245:0)
[ 1.904367] mmcblk1: p1
[ 2.044224] imx_usb 2184200.usb: No over current polarity defined
[ 2.048071] imx_thermal tempmon: Commercial CPU temperature grade - max:95C critical:90C passive:85C
[ 2.050876] fsl-ssi-dai 2028000.ssi: No cache defaults, reading back from HW
[ 2.052129] imx_usb 2184200.usb: No over current polarity defined
[ 2.055006] fsl-ssi-dai 2028000.ssi: No cache defaults, reading back from HW
[ 2.056274] imx_usb 2184200.usb: No over current polarity defined
[ 2.058405] input: gpio-keys as /devices/soc0/gpio-keys/input/input0
[ 2.059168] hctosys: unable to open rtc device (rtc0)
[ 2.060237] ALSA device list:
[ 2.060248] No soundcards found.
[ 2.063415] Freeing unused kernel memory: 1024K
[ 2.064669] fsl-ssi-dai 2028000.ssi: No cache defaults, reading back from HW
[ 2.065244] imx_usb 2184200.usb: No over current polarity defined
[ 2.103443] Run /init as init process
[ 5.903311] EXT4-fs (mmcblk1p1): recovery complete
[ 5.906215] EXT4-fs (mmcblk1p1): mounted filesystem with ordered data mode. Opts: (null)
[ 6.381307] systemd[1]: systemd 244.5+ running in system mode. (+PAM -AUDIT -SELINUX +IMA -APPARMOR -SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP -GCRYPT -GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN2 -IDN -PCRE2 default-hierarchy=hybrid)
[ 6.381887] systemd[1]: Detected architecture arm.
[ 6.419783] systemd[1]: Set hostname to .
[ 6.426686] systemd[1]: Hardware watchdog ‘imx2+ watchdog’, version 0
[ 6.426759] systemd[1]: Set hardware watchdog to 30s.
[ 8.683267] random: systemd: uninitialized urandom read (16 bytes read)
[ 8.687597] systemd[1]: Created slice system-getty.slice.
[ 8.687918] random: systemd: uninitialized urandom read (16 bytes read)
[ 8.689935] systemd[1]: Created slice system-serial\x2dgetty.slice.
[ 8.690049] random: systemd: uninitialized urandom read (16 bytes read)
[ 8.691839] systemd[1]: Created slice User and Session Slice.
[ 8.692311] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[ 8.692850] systemd[1]: Reached target Remote File Systems.
[ 8.693014] systemd[1]: Reached target Slices.
[ 8.693179] systemd[1]: Reached target Swap.
[ 8.712143] systemd[1]: Listening on RPCbind Server Activation Socket.
[ 8.712366] systemd[1]: Reached target RPC Port Mapper.
[ 8.712915] systemd[1]: Listening on initctl Compatibility Named Pipe.
[ 8.737088] systemd[1]: Condition check resulted in Journal Audit Socket being skipped.
[ 8.737994] systemd[1]: Listening on Journal Socket (/dev/log).
[ 8.738952] systemd[1]: Listening on Journal Socket.
[ 8.740039] systemd[1]: Listening on Network Service Netlink Socket.
[ 8.740788] systemd[1]: Listening on udev Control Socket.
[ 8.741509] systemd[1]: Listening on udev Kernel Socket.
[ 8.742308] systemd[1]: Condition check resulted in Huge Pages File System being skipped.
[ 8.749078] systemd[1]: Mounting POSIX Message Queue File System…
[ 8.757447] systemd[1]: Mounting Kernel Debug File System…
[ 8.765444] systemd[1]: Mounting Temporary Directory (/tmp)…
[ 8.775914] systemd[1]: Starting Create list of static device nodes for the current kernel…
[ 8.776581] systemd[1]: Condition check resulted in File System Check on Root Device being skipped.
[ 8.784994] systemd[1]: Starting Journal Service…
[ 8.794378] systemd[1]: Starting Load Kernel Modules…
[ 8.802460] systemd[1]: Starting Remount Root and Kernel File Systems…
[ 8.813109] systemd[1]: Starting udev Coldplug all Devices…
[ 8.822281] systemd[1]: Starting Setup Virtual Console…
[ 8.842146] systemd[1]: Mounted POSIX Message Queue File System.
[ 8.851731] systemd[1]: Mounted Kernel Debug File System.
[ 8.853176] systemd[1]: Mounted Temporary Directory (/tmp).
[ 8.857914] systemd[1]: Started Create list of static device nodes for the current kernel.
[ 8.870154] systemd[1]: systemd-vconsole-setup.service: Succeeded.
[ 8.883241] systemd[1]: Started Setup Virtual Console.
[ 8.909673] EXT4-fs (mmcblk1p1): re-mounted. Opts: (null)
[ 8.951391] openvswitch: Open vSwitch switching datapath
[ 8.981413] systemd[1]: Started Load Kernel Modules.
[ 8.982978] systemd[1]: Condition check resulted in FUSE Control File System being skipped.
[ 8.993398] systemd[1]: Mounting Kernel Configuration File System…
[ 9.006725] systemd[1]: Starting Apply Kernel Variables…
[ 9.058234] systemd[1]: Started Remount Root and Kernel File Systems.
[ 9.062575] systemd[1]: Mounted Kernel Configuration File System.
[ 9.065448] systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
[ 9.065758] systemd[1]: Condition check resulted in Create System Users being skipped.
[ 9.073749] systemd[1]: Starting Create Static Device Nodes in /dev…
[ 9.089528] systemd[1]: Started Apply Kernel Variables.
[ 9.155591] systemd[1]: Started Create Static Device Nodes in /dev.
[ 9.157110] systemd[1]: Reached target Local File Systems (Pre).
[ 9.157639] systemd[1]: var.mount: Directory /var to mount over is not empty, mounting anyway.
[ 9.174872] systemd[1]: Mounting /var…
[ 9.181177] systemd[1]: Starting udev Kernel Device Manager…
[ 9.218127] systemd[1]: Mounted /var.
[ 9.236776] systemd[1]: Mounting /var/volatile…
[ 9.254674] systemd[1]: Starting OSTree Remount OS/ Bind Mounts…
[ 9.264019] systemd[1]: Starting RPC Bind…
[ 9.293338] systemd[1]: Mounted /var/volatile.
[ 9.299334] systemd[1]: Started OSTree Remount OS/ Bind Mounts.
[ 9.303449] systemd[1]: Reached target Local File Systems.
[ 9.309463] systemd[1]: Starting Tell Plymouth To Write Out Runtime Data…
[ 9.309903] systemd[1]: Condition check resulted in Commit a transient machine-id on disk being skipped.
[ 9.321869] systemd[1]: Starting Load/Save Random Seed…
[ 9.328932] systemd[1]: Started Systemd Timesync Update Fixes.
[ 9.330937] systemd[1]: Started RPC Bind.
[ 9.344164] systemd[1]: Received SIGRTMIN+20 from PID 383 (plymouthd).
[ 9.355658] systemd[1]: plymouth-read-write.service: Succeeded.
[ 9.358399] systemd[1]: Started Tell Plymouth To Write Out Runtime Data.
[ 9.389995] systemd[1]: systemd-timesyncd-update.service: Succeeded.
[ 9.661683] systemd[1]: Started Journal Service.
[ 9.714719] systemd-journald[468]: Received client request to flush runtime journal.
[ 10.896641] imx_media_common: module is from the staging directory, the quality is unknown, you have been warned.
[ 10.902865] imx6_media: module is from the staging directory, the quality is unknown, you have been warned.
[ 10.906965] fsl-ssi-dai 2028000.ssi: No cache defaults, reading back from HW
[ 10.907586] imx_usb 2184200.usb: No over current polarity defined
[ 10.982144] fsl-ssi-dai 2028000.ssi: No cache defaults, reading back from HW
[ 10.982886] imx_usb 2184200.usb: No over current polarity defined
[ 10.999166] CAN device driver interface
[ 11.008722] flexcan 2090000.flexcan: 2090000.flexcan supply xceiver not found, using dummy regulator
[ 11.030363] fsl-ssi-dai 2028000.ssi: No cache defaults, reading back from HW
[ 11.030986] imx_usb 2184200.usb: No over current polarity defined
[ 11.043001] fsl-ssi-dai 2028000.ssi: No cache defaults, reading back from HW
[ 11.043757] imx_usb 2184200.usb: No over current polarity defined
[ 11.044921] coda 2040000.vpu: Direct firmware load for vpu_fw_imx6d.bin failed with error -2
[ 11.050563] coda 2040000.vpu: Using fallback firmware vpu/vpu_fw_imx6d.bin
[ 11.053230] coda 2040000.vpu: Firmware code revision: 46076
[ 11.053238] coda 2040000.vpu: Initialized CODA960.
[ 11.053243] coda 2040000.vpu: Firmware version: 3.1.1
[ 11.053496] coda 2040000.vpu: encoder registered as video0
[ 11.053638] coda 2040000.vpu: decoder registered as video1
[ 11.315030] imx-sdma 20ec000.sdma: loaded firmware 3.5
[ 11.321867] fsl-ssi-dai 2028000.ssi: No cache defaults, reading back from HW
[ 11.322576] imx_usb 2184200.usb: No over current polarity defined
[ 11.347400] fsl-asoc-card sound: sgtl5000 <-> 2028000.ssi mapping ok
[ 11.356123] imx_usb 2184200.usb: No over current polarity defined
[ 11.778234] stmpe-adc stmpe-adc: DMA mask not set
[ 11.828937] imx_usb 2184200.usb: No over current polarity defined
[ 11.842092] imx6_media_csi: module is from the staging directory, the quality is unknown, you have been warned.
[ 11.920593] imx6_media_csi: module is from the staging directory, the quality is unknown, you have been warned.
[ 11.922432] ipu1_csi0: Registered ipu1_csi0 capture as /dev/video2
[ 11.922861] ipu1_ic_prpenc: Registered ipu1_ic_prpenc capture as /dev/video3
[ 11.932889] ipu1_ic_prpvf: Registered ipu1_ic_prpvf capture as /dev/video4
[ 11.932917] imx-media: ipu1_csi0:1 → ipu1_ic_prp:0
[ 11.932923] imx-media: ipu1_csi0:1 → ipu1_vdic:0
[ 11.932928] imx-media: ipu1_vdic:2 → ipu1_ic_prp:0
[ 11.932934] imx-media: ipu1_ic_prp:1 → ipu1_ic_prpenc:0
[ 11.932939] imx-media: ipu1_ic_prp:2 → ipu1_ic_prpvf:0
[ 11.932943] imx-media: subdev ipu1_csi0 bound
[ 11.933410] imx_usb 2184200.usb: No over current polarity defined
[ 11.940571] ipu1_csi1: Registered ipu1_csi1 capture as /dev/video5
[ 11.940583] imx-media: ipu1_csi1:1 → ipu1_ic_prp:0
[ 11.940589] imx-media: ipu1_csi1:1 → ipu1_vdic:0
[ 11.940597] imx-media: subdev ipu1_csi1 bound
[ 11.940621] imx-media: ipu1_csi0_mux:5 → ipu1_csi0:0
[ 11.940658] imx-media: ipu1_csi1_mux:5 → ipu1_csi1:0
[ 11.950075] imx-media: Registered ipu_ic_pp csc/scaler as /dev/video6
[ 11.950537] imx_usb 2184200.usb: No over current polarity defined
[ 12.211801] fec 2188000.ethernet ethernet0: renamed from eth0
[ 14.202780] random: crng init done
[ 14.202788] random: 7 urandom warning(s) missed due to ratelimiting
[ 17.492906] Micrel KSZ8041 2188000.ethernet-1:00: attached PHY driver [Micrel KSZ8041] (mii_bus:phy_addr=2188000.ethernet-1:00, irq=POLL)
[ 19.593331] fec 2188000.ethernet ethernet0: Link is Up - 100Mbps/Full - flow control rx/tx
[ 19.593379] IPv6: ADDRCONF(NETDEV_CHANGE): ethernet0: link becomes ready
[ 22.463757] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[ 22.466785] Bridge firewalling registered
[ 23.083102] Initializing XFRM netlink socket
[ 26.334074] overlayfs: overlapping lowerdir path
[ 60.302597] overlayfs: overlapping lowerdir path
[ 639.315952] cgroup: cgroup: disabling cgroup2 socket matching due to net_prio or net_cls activation
[ 649.342827] IPv6: ADDRCONF(NETDEV_CHANGE): can0: link becomes ready
[ 649.370183] can: controller area network core (rev 20170425 abi 9)
[ 649.370264] NET: Registered protocol family 29
[ 649.380886] can: raw protocol (rev 20170425)
[ 7781.735083] overlayfs: overlapping lowerdir path
[ 7940.188329] overlayfs: overlapping lowerdir path

5

Just to be clear what version of TorizonCore are you seeing this issue on?

Also of the 3 steps you described:

f1:build release container: release
f1:deploy release container
source ./EmeraAutostartup.sh

Which of these 3 actually triggers the error?

Historically we’ve only ever seen this issue once internally ourselves. Furthermore we were never able to reproduce it. Usually this error suggests data related to your container got corrupted somehow.

Though it sounds like you are able to see this error with some regularity? Can you reproduce it easily or is it completely random?

Also does it only happen with your specific application container? Or have you see this happen with other containers as well?

Best Regards,
Jeremias

6

I get this error after several times (usually) of:

reload OS, make sure its “good”
develop for a while, combination of debugger, and direcly deploying the release container.
Sometimes I dont do a docker stop (container) before trying to deploy from the debugger…
Sometimes I do a sudo systemctl enable MyStartup.service which calls a batch file, which in turn calls the docker run command that was exported…

I dont do much more than that, and every so often when I reboot I get this error, that systemctl could not run because of this error.

This is now happening on multiple boards (at least 2)

7

We’ll see if we can try and reproduce this error on our side using your steps here.

Just some questions for clarity:

  • What revision of the Colibri i.MX6 are you using? (See the sticker on the module for revision information).
  • What version of TorzionCore are you using? (run cat /etc/issue on the module to get this information).
  • This issue doesn’t seem related to the code that is running in your container. But, just to be sure have you seen this issue just running simple hello-world code in a container?

Best Regards,
Jeremias

8

I am using Colibri iMX6DL 512MB (IT?) v1.1B

cat /etc/issue 
TorizonCore Upstream 5.6.0+build.13 \n \l

I have only been seeing this issue in the last couple of weeks. A few things have happened to the device tree around that time, but I have not changed the device tree custom overlay in months. The things I have changed were to get the spidev3.1 working and a lot of GPIOs.

I have also been exchanging boards back and forth between my Colibri Evaluation board and our own carrier board quite a bit, as we are starting to get our own system up and running.

I think my workflow needs to evolve into using either dockerHub push/pull or perhaps OTA update instead of physically moving the board from one computer to another.

Yesterday I observed a board which was fully working on my system, then I handed it to a colleague and it didn’t work. When he gave it back to me I did a systemctl status myStartup.service and it showed the overlay problem in the reply.

I wonder if it has to do with improper shutdown procedure? We usually just cut power… maybe we need to do a sudo shutdown now?

9

In our prior testing we were unable to reproduce this issue you are experiencing.

I wonder if it has to do with improper shutdown procedure? We usually just cut power… maybe we need to do a sudo shutdown now?

This may be a possibility, if you are shutting the system down while it’s transferring/working with container images. Then it’s possible the container image could become corrupted and cause such an issue.

I am using Colibri iMX6DL 512MB (IT?) v1.1B

According to the product page here: Colibri iMX6 | Toradex Developer Center

Starting with the 1.1B this variant of the Colibri i.MX6 features a new eMMC. There have been reports about this eMMC having some strange issues. I’m not saying that is the case here, but if the eMMC is having write issues with the container image this could cause the issue seen here.

Do you have any other versions/variants of this module? Or just this one?

Best Regards,
Jeremias

10

Okay thanks,

No other variants of the module. Just this one.

Leigh

11

When I power off on a colibri evaluation board, would it somehow shut down “nicer” than a board which did not make any considerations for shutting things down more carefully, or does Colibri Eval board just cut power as soon as the button is pressed?

Leigh

12

Not sure I quite understand the question, but in general there’s two different ways a device shuts down.

There is a hardware shutdown. For example if the power just gets disconnected or the power button is hit. This type of shutdown can happen at any time and typically takes no consideration to what the software is doing at the time.

Then there is software shutdown. This would be like running the shutdown command in Linux. This allows the operarting system to finish processes and “clean up” things before it shuts down. This makes this type of shutdown “safer” in that sense.

This is to say you should never do hardware shutdowns. But, keep in mind how it may affect your application/system if the device were to shutdown at any arbitrary time. That all said it’s still not 100% clear to me that your issues you’re observing is caused by a shutdown at the wrong moment. It would be best to test this to see if the two are even related.

Best Regards,
Jeremias