BSP 2.8b7 Console Only RT-Kernel Hangs on Boot

Hello All,

I have a custom built Linux image based on BSP 2.8b7 on an Apalis iMX6Q IT module on a custom board. This is a console only RT-Kernel image. Everything installs and runs fine, the first time. But once I reboot it, the boot process hangs at the very end. See below for console output

U-Boot 2016.11-2.8.7+g5c2d46b (Jun 29 2020 - 17:31:59 +0000)

CPU:   Freescale i.MX6Q rev1.6 at 792 MHz
Reset cause: POR
I2C:   ready
DRAM:  2 GiB
PMIC:  device id: 0x10, revision id: 0x21, programmed
MMC:   FSL_SDHC: 0, FSL_SDHC: 1, FSL_SDHC: 2
auto-detected panel vga-rgb
Display: vga-rgb (640x480)
In:    serial
Out:   serial
Err:   serial
Model: Toradex Apalis iMX6 Quad 2GB IT V1.1C, Serial# 10565579
Net:   using PHY at 7
FEC [PRIME]
Hit any key to stop autoboot:  0
Booting from internal eMMC chip...
reading imx6q-apalis-acu3mc10.dtb
53391 bytes read in 18 ms (2.8 MiB/s)
reading zImage
5550296 bytes read in 207 ms (25.6 MiB/s)
## Flattened Device Tree blob at 12100000
   Booting using the fdt blob at 0x12100000
   Using Device Tree in place at 12100000, end 1211008e

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.9.220-rt143-2.8.7+g5722926 (oe-user@oe-host) (gcc version 7.3.0 (GCC) ) #1 SMP PREEMPT RT Mon Jun 29 16:54:15 UTC 2020
[    0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt:Machine model: Toradex Apalis iMX6Q/D Module on ACU3/MC10 Carrier Board
[    0.000000] Reserved memory: created CMA memory pool at 0x7c000000, size 320 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] percpu: Embedded 13 pages/cpu s23392 r8192 d21664 u53248
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 521008
[    0.000000] Kernel command line: vmalloc=400M user_debug=30 ip=off root=/dev/mmcblk0p2 ro rootfstype=ext4 rootwait fec_mac=00:14:2d:a1:37:cb consoleblank=0 no_console_suspend=1 console=tty1 console=ttymxc0,115200n8 mxc_hdmi.only_cea=1 video=mxcfb0:dev=hdmi,1920x1080M@60,if=RGB24 video=mxcfb1:off video=mxcfb2:off video=mxcfb3:off fbmem=32M
[    0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.000000] Inode-cache hash table entries: 131072 (order: 7, 524288 bytes)
[    0.000000] Memory: 1738044K/2097152K available (8192K kernel code, 300K rwdata, 2004K rodata, 1024K init, 432K bss, 31428K reserved, 327680K cma-reserved, 90112K highmem)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
[    0.000000]     vmalloc : 0xe7000000 - 0xff800000   ( 392 MB)
[    0.000000]     lowmem  : 0x80000000 - 0xe6800000   (1640 MB)
[    0.000000]     pkmap   : 0x7fe00000 - 0x80000000   (   2 MB)
[    0.000000]     modules : 0x7f000000 - 0x7fe00000   (  14 MB)
[    0.000000]       .text : 0x80008000 - 0x80900000   (9184 kB)
[    0.000000]       .init : 0x80b00000 - 0x80c00000   (1024 kB)
[    0.000000]       .data : 0x80c00000 - 0x80c4b000   ( 300 kB)
[    0.000000]        .bss : 0x80c4b000 - 0x80cb70a4   ( 433 kB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000]  Build-time adjustment of leaf fanout to 32.
[    0.000000] NR_IRQS:16 nr_irqs:16 16
[    0.000000] L2C-310 errata 752271 769419 enabled
[    0.000000] L2C-310 enabling early BRESP for Cortex-A9
[    0.000000] L2C-310 full line of zeros enabled for Cortex-A9
[    0.000000] L2C-310 ID prefetch enabled, offset 16 lines
[    0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
[    0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
[    0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x76470001
[    0.000000] Switching to timer-based delay loop, resolution 333ns
[    0.000007] sched_clock: 32 bits at 3000kHz, resolution 333ns, wraps every 715827882841ns
[    0.000018] clocksource: mxc_timer1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 637086815595 ns
[    0.001484] Console: colour dummy device 80x30
[    0.002366] console [tty1] enabled
[    0.002389] Calibrating delay loop (skipped), value calculated using timer frequency.. 6.00 BogoMIPS (lpj=30000)
[    0.002393] pid_max: default: 32768 minimum: 301
[    0.002494] Mount-cache hash table entries: 4096 (order: 2, 16384 bytes)
[    0.002502] Mountpoint-cache hash table entries: 4096 (order: 2, 16384 bytes)
[    0.003238] CPU: Testing write buffer coherency: ok
[    0.003244] CPU0: Spectre v2: using BPIALL workaround
[    0.003641] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.003888] Setting up static identity map for 0x10100000 - 0x10100058
[    0.269874] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.269879] CPU1: Spectre v2: using BPIALL workaround
[    0.379861] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
[    0.379866] CPU2: Spectre v2: using BPIALL workaround
[    0.489859] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
[    0.489865] CPU3: Spectre v2: using BPIALL workaround
[    0.490056] Brought up 4 CPUs
[    0.490225] SMP: Total of 4 processors activated (24.00 BogoMIPS).
[    0.490255] CPU: All CPU(s) started in SVC mode.
[    0.491298] devtmpfs: initialized
[    0.508480] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[    0.508915] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.508980] futex hash table entries: 1024 (order: 4, 65536 bytes)
[    0.526801] pinctrl core: initialized pinctrl subsystem
[    0.528251] NET: Registered protocol family 16
[    0.543529] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.545303] cpuidle: using governor menu
[    0.545624] CPU identified as i.MX6Q, silicon rev 1.6
[    0.575683] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[    0.575745] hw-breakpoint: maximum watchpoint size is 4 bytes.
[    0.578003] imx6q-pinctrl 20e0000.iomuxc: initialized IMX pinctrl driver
[    0.580239] imx-gpc 20dc000.gpc: no fsl,ldo-bypass found!
[    0.580414] imx-gpc 20dc000.gpc: Registered imx-gpc
[    0.609951] usb_host_vbus: supplied by usb_host_vbus_hub
[    0.610692] vgaarb: loaded
[    0.611441] SCSI subsystem initialized
[    0.612086] usbcore: registered new interface driver usbfs
[    0.612199] usbcore: registered new interface driver hub
[    0.612367] usbcore: registered new device driver usb
[    0.613082] i2c-gpio i2c@0: using pins 80 (SDA) and 62 (SCL)
[    0.614293] i2c i2c-1: IMX I2C adapter registered
[    0.614344] i2c i2c-1: can't use DMA, using PIO instead.
[    0.614981] i2c i2c-3: IMX I2C adapter registered
[    0.615030] i2c i2c-3: can't use DMA, using PIO instead.
[    0.615247] Linux video capture interface: v2.00
[    0.615333] pps_core: LinuxPPS API ver. 1 registered
[    0.615363] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.615424] PTP clock support registered
[    0.639679] imx-ipuv3 2400000.ipu: IPU DMFC NORMAL mode: 1(0~1), 5B(4,5), 5F(6,7)
[    0.669672] imx-ipuv3 2800000.ipu: IPU DMFC NORMAL mode: 1(0~1), 5B(4,5), 5F(6,7)
[    0.670761] MIPI CSI2 driver module loaded
[    0.671025] Advanced Linux Sound Architecture Driver Initialized.
[    0.673197] clocksource: Switched to clocksource mxc_timer1
[    0.673378] VFS: Disk quotas dquot_6.6.0
[    0.673487] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    0.687074] NET: Registered protocol family 2
[    0.687872] TCP established hash table entries: 16384 (order: 4, 65536 bytes)
[    0.688086] TCP bind hash table entries: 16384 (order: 6, 458752 bytes)
[    0.689060] TCP: Hash tables configured (established 16384 bind 16384)
[    0.689220] UDP hash table entries: 1024 (order: 4, 65536 bytes)
[    0.689396] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes)
[    0.689803] NET: Registered protocol family 1
[    0.690501] RPC: Registered named UNIX socket transport module.
[    0.690544] RPC: Registered udp transport module.
[    0.690574] RPC: Registered tcp transport module.
[    0.690601] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.691626] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[    0.693756] Bus freq driver module loaded
[    0.695767] Initialise system trusted keyrings
[    0.696252] workingset: timestamp_bits=30 max_order=19 bucket_order=0
[    0.704957] NFS: Registering the id_resolver key type
[    0.705020] Key type id_resolver registered
[    0.705051] Key type id_legacy registered
[    0.705094] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.705920] ntfs: driver 2.1.32 [Flags: R/W].
[    0.706369] fuse init (API version 7.26)
[    0.714620] Key type asymmetric registered
[    0.714668] Asymmetric key parser 'x509' registered
[    0.714822] bounce: pool size: 64 pages
[    0.714853] io scheduler noop registered
[    0.714882] io scheduler deadline registered
[    0.715104] io scheduler cfq registered (default)
[    0.716432] imx-pcie 1ffc000.pcie: no reserved region node.
[    0.716549] 1ffc000.pcie supply epdev_on not found, using dummy regulator
[    0.716955] OF: PCI: host bridge /soc/pcie@0x01000000 ranges:
[    0.717021] OF: PCI:    IO 0x01f80000..0x01f8ffff -> 0x00000000
[    0.717064] OF: PCI:   MEM 0x01000000..0x01efffff -> 0x01000000
[    1.046994] imx-pcie 1ffc000.pcie: phy link never came up
[    1.047041] imx-pcie 1ffc000.pcie: Link never came up
[    1.050142] imx-pcie 1ffc000.pcie: failed to initialize host
[    1.050179] imx-pcie 1ffc000.pcie: unable to add pcie port.
[    1.050416] imx-pcie: probe of 1ffc000.pcie failed with error -110
[    1.054222] mxc_sdc_fb fb@0: NO mxc display driver found!
[    1.054341] mxc_sdc_fb fb@1: mxcfb1 is turned off!
[    1.054444] mxc_sdc_fb fb@2: mxcfb2 is turned off!
[    1.054545] mxc_sdc_fb fb@3: mxcfb3 is turned off!
[    1.056769] imx-sdma 20ec000.sdma: no iram assigned, using external mem
[    1.060829] imx-sdma 20ec000.sdma: loaded firmware 3.3
[    1.063322] pfuze100-regulator 1-0008: Full layer: 2, Metal layer: 1
[    1.064139] pfuze100-regulator 1-0008: FAB: 0, FIN: 0
[    1.064174] pfuze100-regulator 1-0008: pfuze100 found.
[    1.091299] 2020000.serial: ttymxc0 at MMIO 0x2020000 (irq = 27, base_baud = 5000000) is a IMX
[    1.936783] console [ttymxc0] enabled
[    1.941741] 21e8000.serial: ttymxc1 at MMIO 0x21e8000 (irq = 70, base_baud = 5000000) is a IMX
[    1.951280] 21f0000.serial: ttymxc3 at MMIO 0x21f0000 (irq = 71, base_baud = 5000000) is a IMX
[    1.960775] 21f4000.serial: ttymxc4 at MMIO 0x21f4000 (irq = 72, base_baud = 5000000) is a IMX
[    1.970648] [drm] Initialized
[    1.974270] [drm] Initialized vivante 1.0.0 20120216 on minor 0
[    2.045183] brd: module loaded
[    2.056978] loop: module loaded
[    2.061540] ahci-imx 2200000.sata: fsl,transmit-level-mV not specified, using 00000024
[    2.069517] ahci-imx 2200000.sata: fsl,transmit-boost-mdB not specified, using 00000480
[    2.077572] ahci-imx 2200000.sata: fsl,transmit-atten-16ths not specified, using 00002000
[    2.085798] ahci-imx 2200000.sata: fsl,receive-eq-mdB not specified, using 05000000
[    2.097468] ahci-imx 2200000.sata: SSS flag set, parallel bus scan disabled
[    2.104500] ahci-imx 2200000.sata: AHCI 0001.0300 32 slots 1 ports 3 Gbps 0x1 impl platform mode
[    2.113340] ahci-imx 2200000.sata: flags: ncq sntf stag pm led clo only pmp pio slum part ccc apst
[    2.124219] scsi host0: ahci-imx
[    2.127884] ata1: SATA max UDMA/133 mmio [mem 0x02200000-0x02203fff] port 0x100 irq 79
[    2.137854] libphy: Fixed MDIO Bus: probed
[    2.142485] vcan: Virtual CAN interface driver
[    2.146969] CAN device driver interface
[    2.151274] 2090000.flexcan supply xceiver not found, using dummy regulator
[    2.159433] flexcan 2090000.flexcan: device registered (reg_base=e7360000, irq=33)
[    2.168407] 2188000.ethernet supply phy not found, using dummy regulator
[    2.186895] pps pps0: new PPS source ptp0
[    2.192531] libphy: fec_enet_mii_bus: probed
[    2.209060] fec 2188000.ethernet eth0: registered PHC device 0
[    2.215382] usbcore: registered new interface driver asix
[    2.220885] usbcore: registered new interface driver ax88179_178a
[    2.227075] usbcore: registered new interface driver cdc_ether
[    2.232995] usbcore: registered new interface driver net1080
[    2.238761] usbcore: registered new interface driver cdc_subset
[    2.244778] usbcore: registered new interface driver zaurus
[    2.250466] usbcore: registered new interface driver cdc_ncm
[    2.256181] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    2.262740] ehci-pci: EHCI PCI platform driver
[    2.267283] ehci-mxc: Freescale On-Chip EHCI Host driver
[    2.272864] usbcore: registered new interface driver usb-storage
[    2.279033] usbcore: registered new interface driver usbserial
[    2.284973] usbcore: registered new interface driver usbserial_generic
[    2.291570] usbserial: USB Serial support registered for generic
[    2.297660] usbcore: registered new interface driver ftdi_sio
[    2.303473] usbserial: USB Serial support registered for FTDI USB Serial Device
[    2.310864] usbcore: registered new interface driver pl2303
[    2.316501] usbserial: USB Serial support registered for pl2303
[    2.324045] 2184800.usbmisc supply vbus-wakeup not found, using dummy regulator
[    2.336535] ci_hdrc ci_hdrc.0: EHCI Host Controller
[    2.341468] ci_hdrc ci_hdrc.0: new USB bus registered, assigned bus number 1
[    2.373281] ci_hdrc ci_hdrc.0: USB 2.0 started, EHCI 1.00
[    2.379845] hub 1-0:1.0: USB hub found
[    2.383674] hub 1-0:1.0: 1 port detected
[    2.394270] ci_hdrc ci_hdrc.1: EHCI Host Controller
[    2.399215] ci_hdrc ci_hdrc.1: new USB bus registered, assigned bus number 2
[    2.433280] ci_hdrc ci_hdrc.1: USB 2.0 started, EHCI 1.00
[    2.439741] hub 2-0:1.0: USB hub found
[    2.443564] hub 2-0:1.0: 1 port detected
[    2.448990] mousedev: PS/2 mouse device common for all mice
[    2.456235] snvs_rtc 20cc000.snvs:snvs-rtc-lp: rtc core: registered 20cc000.snvs:snvs-r as rtc1
[    2.465179] i2c /dev entries driver
[    2.471840] imx2-wdt 20bc000.wdog: timeout 60 sec (nowayout=0)
[    2.478586] sdhci: Secure Digital Host Controller Interface driver
[    2.484806] sdhci: Copyright(c) Pierre Ossman
[    2.489189] sdhci-pltfm: SDHCI platform and OF driver helper
[    2.495963] sdhci-esdhc-imx 2190000.usdhc: could not get ultra high speed state, work on normal mode
[    2.505207] sdhci-esdhc-imx 2190000.usdhc: Got CD GPIO
[    2.573499] mmc4: SDHCI controller on 2190000.usdhc [2190000.usdhc] using ADMA
[    2.581094] imx6q-pinctrl 20e0000.iomuxc: pin MX6Q_PAD_SD2_CMD already requested by 20e0000.iomuxc; cannot claim for 2194000.usdhc
[    2.595934] imx6q-pinctrl 20e0000.iomuxc: pin-214 (2194000.usdhc) status -22
[    2.603050] imx6q-pinctrl 20e0000.iomuxc: could not request pin 214 (MX6Q_PAD_SD2_CMD) from group usdhc2grp  on device 20e0000.iomuxc
[    2.615118] sdhci-esdhc-imx 2194000.usdhc: Error applying setting, reverse things back
[    2.623115] sdhci-esdhc-imx: probe of 2194000.usdhc failed with error -22
[    2.630640] sdhci-esdhc-imx 2198000.usdhc: could not get ultra high speed state, work on normal mode
[    2.633277] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    2.633522] ata1.00: ATA-11: SATA SSD, SBFM61.3, max UDMA/133
[    2.633531] ata1.00: 1000215216 sectors, multi 16: LBA48 NCQ (depth 31/32)
[    2.633942] ata1.00: configured for UDMA/133
[    2.634524] scsi 0:0:0:0: Direct-Access     ATA      SATA SSD         61.3 PQ: 0 ANSI: 5
[    2.635714] sd 0:0:0:0: [sda] 1000215216 512-byte logical blocks: (512 GB/477 GiB)
[    2.636177] sd 0:0:0:0: [sda] Write Protect is off
[    2.636331] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.639765] sd 0:0:0:0: [sda] Attached SCSI disk
[    2.731910] mmc4: host does not support reading read-only switch, assuming write-enable
[    2.747413] mmc4: new high speed SD card at address 0007
[    2.753488] mmcblk4: mmc4:0007 DDINC 1.84 GiB
[    2.759830]  mmcblk4: p1
[    2.763310] mmc0: SDHCI controller on 2198000.usdhc [2198000.usdhc] using ADMA
[    2.772511] mxc_vpu 2040000.vpu_fsl: VPU initialized
[    2.781404] mxc_vdoa 21e4000.vdoa: i.MX Video Data Order Adapter(VDOA) driver probed
[    2.789814] imx6q-pinctrl 20e0000.iomuxc: pin MX6Q_PAD_SD1_DAT1 already requested by 2190000.usdhc; cannot claim for leds
[    2.800850] imx6q-pinctrl 20e0000.iomuxc: pin-207 (leds) status -22
[    2.803329] usb 2-1: new high-speed USB device number 2 using ci_hdrc
[    2.813738] imx6q-pinctrl 20e0000.iomuxc: could not request pin 207 (MX6Q_PAD_SD1_DAT1) from group leds-ixora  on device 20e0000.iomuxc
[    2.829017] leds-gpio leds: Error applying setting, reverse things back
[    2.836301] leds-gpio: probe of leds failed with error -22
[    2.843077] caam 2100000.caam: ERA source: CAAMVID.
[    2.848061] caam 2100000.caam: AXI pipeline throttling enabled.
[    2.855253] caam 2100000.caam: device ID = 0x0a16010000000000 (Era 4)
[    2.855253] job rings = 2, qi = 0
[    2.867569] caam_jr 2101000.jr0: Entropy delay = 3200
[    2.905189] mmc0: new DDR MMC card at address 0001
[    2.910742] mmcblk0: mmc0:0001 Q2J54A 3.59 GiB
[    2.915676] mmcblk0boot0: mmc0:0001 Q2J54A partition 1 16.0 MiB
[    2.921972] mmcblk0boot1: mmc0:0001 Q2J54A partition 2 16.0 MiB
[    2.929218]  mmcblk0: p1 p2
[    2.933515] caam_jr 2101000.jr0: Instantiated RNG4 SH0.
[    2.994270] caam_jr 2101000.jr0: Instantiated RNG4 SH1.
[    2.994798] hub 2-1:1.0: USB hub found
[    2.994944] hub 2-1:1.0: 4 ports detected
[    3.014423] caam algorithms registered in /proc/crypto
[    3.023896] caam_jr 2101000.jr0: registering rng-caam
[    3.030333] caam-snvs 20cc000.caam-snvs: can't get snvs clock
[    3.036311] caam-snvs 20cc000.caam-snvs: violation handlers armed - non-secure state
[    3.044414] hidraw: raw HID events driver (C) Jiri Kosina
[    3.050054] usbcore: registered new interface driver usbhid
[    3.055667] usbhid: USB HID core driver
[    3.062787] sgtl5000 1-000a: sgtl5000 revision 0x11
[    3.082156] random: fast init done
[    3.104256] fsl-ssi-dai 2028000.ssi: No cache defaults, reading back from HW
[    3.272662] imx-sgtl5000 sound: sgtl5000 <-> 2028000.ssi mapping ok
[    3.282022] imx-spdif sound-spdif: snd-soc-dummy-dai <-> 2004000.spdif mapping ok
[    3.290869] imx-audio-hdmi sound-hdmi: initialize HDMI-audio failed. load HDMI-video first!
[    3.299666] ip_tables: (C) 2000-2006 Netfilter Core Team
[    3.305793] NET: Registered protocol family 10
[    3.311603] ip6_tables: (C) 2000-2006 Netfilter Core Team
[    3.317090] NET: Registered protocol family 17
[    3.321580] can: controller area network core (rev 20120528 abi 9)
[    3.327880] NET: Registered protocol family 29
[    3.332373] can: raw protocol (rev 20120528)
[    3.336675] can: broadcast manager protocol (rev 20161123 t)
[    3.342373] can: netlink gateway (rev 20130117) max_hops=1
[    3.348138] 8021q: 802.1Q VLAN Support v1.8
[    3.352589] Key type dns_resolver registered
[    3.359990] cpu cpu0: Registered imx6q-cpufreq
[    3.364518] Registering SWP/SWPB emulation handler
[    3.370159] Loading compiled-in X.509 certificates
[    3.384222] Loaded X.509 cert 'Build time autogenerated kernel key: 58d9981814122a00527530517bea50acbe97fb00'
[    3.418294] imx_thermal 2000000.aips-bus:tempmon: Industrial CPU temperature grade - max:105C critical:105C passive:95C
[    3.430493] hctosys: unable to open rtc device (rtc0)
[    3.436637] ALSA device list:
[    3.439648]   #0: imx6q-apalis-sgtl5000
[    3.443521]   #1: imx-spdif
[    3.456216] EXT4-fs (mmcblk0p2): INFO: recovery required on readonly filesystem
[    3.463585] EXT4-fs (mmcblk0p2): write access will be enabled during recovery
[    3.589816] EXT4-fs (mmcblk0p2): recovery complete
[    3.595351] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[    3.603542] VFS: Mounted root (ext4 filesystem) readonly on device 179:10.
[    3.611704] devtmpfs: mounted
[    3.616173] Freeing unused kernel memory: 1024K
[    3.784845] systemd[1]: System time before build time, advancing clock.
[    3.829893] systemd[1]: systemd 234 running in system mode. (+PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP -GCRYPT -GNUTLS -ACL +XZ +LZ4 -SECCOMP +BLKID -ELFUTILS +KMOD -IDN2 +IDN default-hierarchy=hybrid)
[    3.852121] systemd[1]: Detected architecture arm.

Welcome to The Ångström Distribution v2017.12!

[    3.883925] systemd[1]: Set hostname to <apalis-imx6>.
[    4.238943] random: systemd: uninitialized urandom read (16 bytes read)
[    4.245774] systemd[1]: Listening on udev Kernel Socket.
[  OK  ] Listening on udev Kernel Socket.
[    4.283577] random: systemd: uninitialized urandom read (16 bytes read)
[    4.290567] systemd[1]: Listening on Journal Socket.
[  OK  ] Listening on Journal Socket.
[    4.323729] random: systemd: uninitialized urandom read (16 bytes read)
[    4.330507] systemd[1]: Listening on Network Service Netlink Socket.
[  OK  ] Listening on Network Service Netlink Socket.
[    4.363501] systemd[1]: Reached target Swap.
[  OK  ] Reached target Swap.
[    4.393762] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[    4.434003] systemd[1]: Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket (/dev/log).
[    4.473750] systemd[1]: Reached target Remote File Systems.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Listening on udev Control Socket.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Created slice System Slice.
         Mounting Kernel Debug File System...
[  OK  ] Created slice system-serial\x2dgetty.slice.
         Starting Load Kernel Modules...
[  OK  ] Reached target Slices.
[    4.760902] configfs: module verification failed: signature and/or required key missing - tainting kernel
         Starting Export GPIOs to user space...
         Starting File System Check on Root Device...
[  OK  ] Created slice system-getty.slice.
         Starting Create list of required st…ce nodes for the current kernel...
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Reached target Paths.
         Starting Journal Service...
         Mounting Temporary Directory (/tmp)...
[  OK  ] Mounted Kernel Debug File System.
[  OK  ] Started Load Kernel Modules.
[  OK  ] Started Export GPIOs to user space.
[  OK  ] Started Create list of required sta…vice nodes for the current kernel.
[  OK  ] Mounted Temporary Directory (/tmp).
         Mounting Kernel Configuration File System...
         Mounting FUSE Control File System...
         Starting Apply Kernel Variables...
[  OK  ] Mounted Kernel Configuration File System.
[  OK  ] Mounted FUSE Control File System.
[  OK  ] Started File System Check on Root Device.
[  OK  ] Started Apply Kernel Variables.
[  OK  ] Started Journal Service.
         Starting Remount Root and Kernel File Systems...
[    5.140618] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[  OK  ] Started Remount Root and Kernel File Systems.
         Starting Create Static Device Nodes in /dev...
         Starting udev Coldplug all Devices...
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Create Static Device Nodes in /dev.
[    5.320620] systemd-journald[270]: Received request to flush runtime journal from PID 1
[  OK  ] Reached target Local File Systems (Pre).
[    5.355383] systemd-journald[270]: File /var/log/journal/0d19cb311e0e43bfa4018a45964a48af/system.journal corrupted or uncleanly shut down, renaming and replacing.
         Mounting /var/volatile...
         Starting udev Kernel Device Manager...
[  OK  ] Mounted /var/volatile.
[  OK  ] Started Flush Journal to Persistent Storage.
[  OK  ] Started udev Kernel Device Manager.
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
         Starting Load/Save Random Seed...
[  OK  ] Started Create Volatile Files and Directories.
[  OK  ] Started Load/Save Random Seed.
         Starting Network Time Synchronization...
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Started Network Time Synchronization.
[  OK  ] Reached target System Time Synchronized.
[  OK  ] Found device /dev/ttymxc0.
[  OK  ] Reached target Sound Card.
[  OK  ] Started udev Coldplug all Devices.
[  OK  ] Reached target System Initialization.
[  OK  ] Started dnf-automatic-download timer.
         Starting sshd.socket.
[  OK  ] Started dnf-automatic-install timer.
[  OK  ] Listening on RPCbind Server Activation Socket.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Started dnf makecache timer.
[  OK  ] Started dnf-automatic-notifyonly timer.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Listening on Avahi mDNS/DNS-SD Stack Activation Socket.
[  OK  ] Listening on sshd.socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
         Starting Login Service...
         Starting Telephony service...
         Starting Load default USB gadget schema g1.schema...
         Starting Network Time Service (one-shot ntpdate mode)...
[  OK  ] Started Hardware RNG Entropy Gatherer Daemon.
         Starting Avahi mDNS/DNS-SD Stack...
         Starting RPC Bind Service...
[    9.659938] random: crng init done
[    9.663440] random: 7 urandom warning(s) missed due to ratelimiting
[  OK  ] Started D-Bus System Message Bus.
[    9.875625] using random self ethernet address
[    9.880154] using random host ethernet address
[  OK  ] Started Avahi mDNS/DNS-SD Stack.
[  OK  ] Started Telephony service.
[   10.060794] Bluetooth: Core ver 2.22
[   10.064700] NET: Registered protocol family 31
[   10.069191] Bluetooth: HCI device and connection manager initialized
[   10.075618] Bluetooth: HCI socket layer initialized
[   10.080549] Bluetooth: L2CAP socket layer initialized
[   10.085756] Bluetooth: SCO socket layer initialized
[  OK  ] Started Timestamping service.
         Starting Save/Restore Sound Card State...
[  OK  ] Reached target Containers.
[  OK  ] Started Network Time Service (one-shot ntpdate mode).
[  OK  ] Started RPC Bind Service.
[  OK  ] Started Save/Restore Sound Card State.
[  OK  [   10.876175] usb0: HOST MAC 00:14:2d:ff:ff:fe
] Listening on Load/Save RF Kill [   10.881897] usb0: MAC 00:14:2d:ff:ff:ff
Switch Status /dev/rfkill Watch.
[   10.945151] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[  OK  ] Started Load default USB gadget schema g1.schema.
[  OK  ] Started Login Service.
[   11.005846] FAT-fs (mmcblk4p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
         Starting Network Service...
[  OK  ] Started Network Service.
[   11.229206] IPv6: ADDRCONF(NETDEV_UP): usb0: link is not ready
         Starting Wait for Network to be Configured...
[  OK  ] Reached target Network.
         Starting Simple Network Management Protocol (SNMP) Daemon....
         Starting Postfix Mail Transport Agent...
         Starting Network Name Resolution...
         Starting Install MySQL Community Server Database...
         Starting Simple Network Management Protocol (SNMP) Trap Daemon....
         Starting Xinetd A Powerful Replacement For Inetd...
         Starting Target Communication Framework agent...
         Starting Permit User Sessions...
         Starting Advanced TFTP Server...
[  OK  ] Started Install MySQL Community Server Database.
[  OK  ] Started Xinetd A Powerful Replacement For Inetd.
[  OK  ] Started Target Communication Framework agent.
[  OK  ] Started Permit User Sessions.
[  OK  ] Started Advanced TFTP Server.
[  OK  ] Started Network Name Resolution.
[  OK  ] Started Simple Network Management Protocol (SNMP) Trap Daemon..
[  OK  ] Started Simple Network Management Protocol (SNMP) Daemon..
[  OK  ] Reached target Host and Network Name Lookups.
[  OK  ] Started NFS status monitor for NFSv2/3 locking..
[  OK  ] Started Getty on tty1.
[  OK  ] Started Serial Getty on ttymxc0.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started Postfix Mail Transport Agent.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Update UTMP about System Runlevel Changes.
>>Cursor just sits here forever

I have scanned this output and there don’t seem to be any errors, so where do I begin trying to find the problem? I have googled this and it seems for desktop Linux images that this may be related to the graphics driver. I have disabled the graphics driver following the procedure in this post and everything built correctly. Is there another step that needs to be taken? Or any thoughts on where to begin looking?

Thanks!

Hi @mmorgan1361

Thanks for writing to the Toradex Support.

I tested the Preempt Patch on the demo Bsp-2.8.7-LXDE_Image and it is working fine for me.

Could you share your custom changes and the kernel config in a text file?

Thanks and best regards,
Jaski

Hello @jaski.tx

Attached are the local.conf and bblayers.conf
local/bblayers.conf

Other changes that were made are as follows

  1. Added the correct RT-Kernel patch to the /oe-core/layers/meta-toradex-nxp folder
  2. Added the following entries to DEFCONFIG files
    CONFIG_CRYPTO_DEV_FSL_CAAM_RNG_API=y
    CONFIG_TCG_TPM=y
    CONFIG_VLAN_8021Q=y
    CONFIG_BLK_DEV_SR=y
    Added to all 4 DEFCONFIG files
    /oe-core/layers/meta-toradex-nxp/recipes-kernel/linux/linux-toradex-4.4/defconfig
    /oe-core/layers/meta-toradex-nxp/recipes-kernel/linux/linux-toradex-4.9-2.3.x/apalis-imx6/defconfig
    /oe-core/layers/meta-toradex-nxp/recipes-kernel/linux/linux-toradex-rt-4.4/defconfig
    /oe-core/layers/meta-toradex-nxp/recipes-kernel/linux/linux-toradex-rt-4.9-2.3.x/apalis-imx6/defconfig
  3. Removed CONNMAN by using the following procedure from this link
    Removed CONMANPKGS from console-tdx-image.bb
    Removed CONMANPKGS from angstrom-lxde-image-bb
  4. Disabled the Vivante graphics driver
    Ensured that the following line existed in /oe-core/layers/meta-freescale/classes/fsl-vivante-kernel-driver-handler.bb
    MACHINE_HAS_VIVANTE_KERNEL_DRIVER_SUPPORT ??=“0”

The desired result is a console only OS with RT-Kernel support. Heat may be an issue, which is why we are trying to disable anything not absolutely necessary (like the graphics driver).

I have also attached the custom device tree files… I have a feeling that they may be the culprit
Device Tree Files

I appreciate any help that you can provide

Hello @jaski.tx

I am just following up to see if you have had a chance to look at any of the things I sent. Maybe have any ideas as to what the issue may be.

Best Regards

You did a lots of changes to image at once. Could you please try to do them incrementally? This way it would be easy to investigate what change is causing a problem.

HI @mmorgan1361

Sorry for the delayed answer. Perfect that your issue is solved.

Any idea why setting a static IP would cause the module to finish booting all of a sudden?

I don’t think this would be the issue.

Thank you guys for all that you do to help us out. Mike

You are welcome.

Best regards,
Jaski

Hello @jaski.tx and @alex.tx

So I think I have inadvertently fixed the problem. I made a couple changes in order to set the hostname and a static IP at OS build time and that seems to have created a stable OS. I no longer have the problems with rebooting. The changes I made are as follows

The only difference is on the last item, the systemd_%.bbappend, the file 0001-systemd-udevd.service.in-set-MountFlags-shared.patch did not exist in that folder or anywhere near it so I left that line out. Other than that, those were the changes

Any idea why setting a static IP would cause the module to finish booting all of a sudden?

Thank you guys for all that you do to help us out.
Mike