Linux Kernel stops while boot

I compiled linux kernel for T20 Board from Source provided from Toradex. (git)
I added the compiled Image to rootfs of downloaded Linux Image V2.6 as described on http://developer.toradex.com/knowledge-base/build-u-boot-and-linux-kernel-from-source-code

The Kernel starts loading, loads some drivers etc.

Then it stops. Last line is: tegra_nand: NVIDIA Tegra NAND controller @ base 0x70008000 irq=56

Sometimes this error is also shown when downloaded Image is started.

What goes wrong? What can I do?

I think it happens to me too but I don’t remember if the last line is the same.

It happens only to one T20 512MB V1.2A Linux 2.6 out of five.

I was in a hurry because I had to work on the other boards so I did not investigate the issue, thinking that I did something wrong while flashing the module.

Please share the complete boot log starting from uboot.

U-Boot SPL 2015.04 (Mar 31 2016 - 01:36:10)


U-Boot 2015.04 (Mar 31 2016 - 01:36:10)

TEGRA20
DRAM:  512 MiB
NAND:  1024 MiB
MMC:   Tegra SD/MMC: 0
*** Warning - bad CRC, using default environment

In:    serial
Out:   lcd
Err:   lcd
Model: Toradex Colibri T20 256MB V1.2A, Serial# 04975736
Net:   Net Initialization Skipped
No ethernet found.
Hit any key to stop autoboot:  1  0 
Booting from NAND...
UBI: default fastmap pool size: 200
UBI: default fastmap WL pool size: 25
UBI: attaching mtd1 to ubi0
UBI: attached by fastmap
UBI: fastmap pool size: 200
UBI: fastmap WL pool size: 25
UBI: attached mtd1 (name "mtd=3", size 1020 MiB) to ubi0
UBI: PEB size: 262144 bytes (256 KiB), LEB size: 253952 bytes
UBI: min./max. I/O unit sizes: 4096/4096, sub-page size 4096
UBI: VID header offset: 4096 (aligned 4096), data offset: 8192
UBI: good PEBs: 4080, bad PEBs: 0, corrupted PEBs: 0
UBI: user volume: 1, internal volumes: 1, max. volumes count: 128
UBI: max/mean erase counter: 3/1, WL threshold: 4096, image sequence number: 0
UBI: available PEBs: 0, total reserved PEBs: 4080, PEBs reserved for bad PEB handling: 80
** File not found /boot/tegra20-colibri-eval-v3.dtb **
Loading file '/boot/zImage' to addr 0x01000000 with size 3250072 (0x00319798)...
Done
Kernel image @ 0x1000000 [ 0x000000 - 0x319798 ]

Starting kernel ...

[    0.000000] Linux version 3.1.10-gbe10973-dirty (root@BerndLinux) (gcc version 5.2.1 20151005 (Linaro GCC 5.2-2015.11-2) ) #5 SMP PREEMPT Wed Aug 3 15:56:43 CEST 2016
[    0.000000] CPU: ARMv7 Processor [411fc090] revision 0 (ARMv7), cr=10c5387d
[    0.000000] CPU: VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] Machine: Toradex Colibri T20
[    0.000000] Found fbmem: 00c00000@17400000
[    0.000000] Found nvmem: 08000000@18000000
[    0.000000] Tegra reserved memory:
[    0.000000] LP0:                     00000000 - 00000000
[    0.000000] Bootloader framebuffer:  17400000 - 17ffffff
[    0.000000] Bootloader framebuffer2: 00000000 - 00000000
[    0.000000] Framebuffer:             16700000 - 16ffffff
[    0.000000] 2nd Framebuffer:         17000000 - 17ffffff
[    0.000000] Carveout:                18000000 - 1fffffff
[    0.000000] Vpr:                     00000000 - 00000000
[    0.000000] Memory policy: ECC disabled, Data cache writealloc
[    0.000000] Tegra SKU: 8 Rev: A04 CPU Process: 1 Core Process: 2 Speedo ID: 1
[    0.000000] Tegra Revision: A04 prime SKU: 0x8 CPU Process: 1 Core Process: 2
[    0.000000] L310 cache controller enabled
[    0.000000] l2x0: 8 ways, CACHE_ID 0x410000c4, AUX_CTRL 0x7e080001, Cache size: 1048576 B
[    0.000000] PERCPU: Embedded 8 pages/cpu @c0cf6000 s10336 r8192 d14240 u32768
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 90904
[    0.000000] Kernel command line: vmalloc=128M usb_high_speed=1 ubi.mtd=ubi root=ubi0:rootfs rootfstype=ubifs ubi.fm_autoconvert=1 mtdparts=tegra_nand:2m(u-boot)ro,1m(u-boot-env),1m(cfgblock)ro,-(ubi) asix_mac=00:14:2d:4b:ec:78 consoleblank=0 no_console_suspend=1 console=tty1 console=ttyS0,115200n8 debug_uartport=lsport,0 mem=372M@0M fbmem=12M@372M nvmem=128M@384M video=tegrafb0:640x480-16@60
[    0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Memory: 358MB = 358MB total
[    0.000000] Memory: 352840k/352840k available, 28088k reserved, 0K highmem
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
[    0.000000]     DMA     : 0xff000000 - 0xffe00000   (  14 MB)
[    0.000000]     vmalloc : 0xd7800000 - 0xf8000000   ( 520 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xd7400000   ( 372 MB)
[    0.000000]     pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
[    0.000000]     modules : 0xbf000000 - 0xbfe00000   (  14 MB)
[    0.000000]       .text : 0xc0008000 - 0xc0893a68   (8751 kB)
[    0.000000]       .init : 0xc0894000 - 0xc08e5860   ( 327 kB)
[    0.000000]       .data : 0xc08e6000 - 0xc095fcf4   ( 488 kB)
[    0.000000]        .bss : 0xc095fd18 - 0xc09d3668   ( 463 kB)
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000] NR_IRQS:480
[    0.000000] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 4294967ms
[    0.000000] Console: colour dummy device 80x30
[    0.000000] console [tty1] enabled
[    0.001218] Calibrating delay loop... 996.14 BogoMIPS (lpj=4980736)
[    0.090088] pid_max: default: 32768 minimum: 301
[    0.090296] Mount-cache hash table entries: 512
[    0.090900] Initializing cgroup subsys blkio
[    0.090991] CPU: Testing write buffer coherency: ok
[    0.091056] ftrace: allocating 23730 entries in 70 pages
[    0.110890] Brought up 2 CPUs
[    0.110931] SMP: Total of 2 processors activated (1992.29 BogoMIPS).
[    0.111522] devtmpfs: initialized
[    0.114177] print_constraints: dummy: 
[    0.114379] NET: Registered protocol family 16
[    0.115100] host1x bus init
[    0.115238] Serial: 8250/16550 driver, 9 ports, IRQ sharing disabled
[    0.381643] Selecting UARTA as the debug console
[    0.381678] The debug console clock name is uarta
[    0.381878] serial8250.0: ttyS0 at MMIO 0x70006000 (irq = 68) is a Tegra
[    0.741272] console [ttyS0] enabled
[    0.870300] tegra_init_emc: MEMPHIS MEM2G16D2DABG 333MHz memory found
[    0.879469] tegra_arb_init: initialized
[    0.883385] tegra_iovmm_register: added iovmm-gart
[    0.897236] bio: create slab  at 0
[    0.901882] SCSI subsystem initialized
[    0.906187] usbcore: registered new interface driver usbfs
[    0.911768] usbcore: registered new interface driver hub
[    0.917158] usbcore: registered new device driver usb
[    0.922315] colibri-otg colibri-otg: otg transceiver registered
[    0.929104] tps6586x 4-0034: found TPS658643, VERSIONCRC is 03
[    0.937046] print_constraints: REG-SM_0: 725 <--> 1500 mV at 1200 mV normal standby
[    0.945919] print_constraints: REG-SM_1: 725 <--> 1500 mV at 1000 mV normal standby
[    0.955057] print_constraints: REG-SM_2: 1700 <--> 1800 mV at 1800 mV normal standby
[    0.963500] print_constraints: REG-LDO_0: 1200 <--> 3300 mV at 1200 mV normal standby
[    0.972430] print_constraints: REG-LDO_1: 725 <--> 1500 mV at 1100 mV normal standby
[    0.981119] print_constraints: REG-LDO_2: 725 <--> 1500 mV at 1200 mV normal standby
[    0.989532] print_constraints: REG-LDO_3: 1250 <--> 3300 mV at 1800 mV normal standby
[    0.998222] print_constraints: REG-LDO_4: 1700 <--> 2475 mV at 1800 mV normal standby
[    1.006567] print_constraints: REG-LDO_5: 1250 <--> 3300 mV at 2850 mV normal standby
[    1.015503] print_constraints: REG-LDO_6: 2850 mV normal standby
[    1.022624] print_constraints: REG-LDO_7: 3300 mV normal standby
[    1.029740] print_constraints: REG-LDO_8: 1800 mV normal standby
[    1.036121] print_constraints: REG-LDO_9: 1250 <--> 3300 mV at 2850 mV normal standby
[    1.044375] Advanced Linux Sound Architecture Driver Version 1.0.24.
[    1.051161] Bluetooth: Core ver 2.16
[    1.054787] NET: Registered protocol family 31
[    1.059233] Bluetooth: HCI device and connection manager initialized
[    1.065602] Bluetooth: HCI socket layer initialized
[    1.070489] Bluetooth: L2CAP socket layer initialized
[    1.075555] Bluetooth: SCO socket layer initialized
[    1.080697] cfg80211: Calling CRDA to update world regulatory domain
[    1.087609] Switching to clocksource timer_us
[    1.102385] nvmap_page_pool_init: nvmap uc page pool size=11026 pages
[    1.115669] nvmap_page_pool_init: nvmap pool = uc, highmem=0, pool_size=11026,totalram=88210, freeram=76494, totalhigh=0, freehigh=0
[    1.160062] nvmap_page_pool_init: nvmap wc page pool size=11026 pages
[    1.173554] nvmap_page_pool_init: nvmap pool = wc, highmem=0, pool_size=11026,totalram=88210, freeram=65396, totalhigh=0, freehigh=0
[    1.217561] nvmap_page_pool_init: nvmap iwb page pool size=11026 pages
[    1.231122] nvmap_page_pool_init: nvmap pool = iwb, highmem=0, pool_size=11026,totalram=88210, freeram=54329, totalhigh=0, freehigh=0
[    1.275783] tegra-nvmap tegra-nvmap: created carveout iram (255KiB)
[    1.283336] tegra-nvmap tegra-nvmap: created carveout generic-0 (131072KiB)
[    1.298923] NET: Registered protocol family 2
[    1.303437] IP route cache hash table entries: 4096 (order: 2, 16384 bytes)
[    1.310698] TCP established hash table entries: 16384 (order: 5, 131072 bytes)
[    1.318150] TCP bind hash table entries: 16384 (order: 5, 196608 bytes)
[    1.325050] TCP: Hash tables configured (established 16384 bind 16384)
[    1.331579] TCP reno registered
[    1.334734] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    1.340580] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    1.347044] NET: Registered protocol family 1
[    1.351674] RPC: Registered named UNIX socket transport module.
[    1.357614] RPC: Registered udp transport module.
[    1.362328] RPC: Registered tcp transport module.
[    1.367031] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.374322] host1x host1x: initialized
[    1.378143] PMU: registered new PMU device of type 0
[    1.396070] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    1.403169] fuse init (API version 7.17)
[    1.407394] msgmni has been set to 689
[    1.412155] io scheduler noop registered (default)
[    1.418131] mpe mpe: initialized
[    1.422267] gr3d gr3d: initialized
[    1.426516] gr2d gr2d: initialized
[    1.430092] isp isp: initialized
[    1.433958] tegradc tegradc.0: probed
[    1.437772] tegradc tegradc.0: use cmd options for tegrafb0: 640x480-16@60
[    1.444692] tegradc tegradc.0: switching framebuffer to 640x480
[    1.450617] tegradc tegradc.0: Calculated sync href=0 vref=1
[    1.456291] tegradc tegradc.0: using mode 640x480 pclk=25175000 href=0 vref=1
[    1.466550] tegradc tegradc.0: switching framebuffer to 640x480
[    1.466560] tegradc tegradc.0: Calculated sync href=0 vref=1
[    1.466572] tegradc tegradc.0: using mode 640x480 pclk=25175000 href=0 vref=1
[    1.513600] Console: switching to colour frame buffer device 80x30
[    1.563912] tegradc tegradc.0: probed
[    1.574055] tegradc tegradc.1: probed
[    1.580356] tegradc tegradc.1: use default mode for tegrafb1: 640x480-16@60
[    1.589788] tegradc tegradc.1: switching framebuffer to 640x480
[    1.598212] tegradc tegradc.1: using mode 640x480 pclk=23750000 href=1 vref=1
[    1.644515] tegradc tegradc.1: hdmi: can't set audio to 44100 at 23750000 pix_clock
[    1.654815] tegradc tegradc.1: probed
[    1.672874] tegra_uart.3: ttyHS3 at MMIO 0x70006300 (irq = 122) is a TEGRA_UART
[    1.772023] Registered UART port ttyHS3
[    1.778611] tegra_uart.1: ttyHS1 at MMIO 0x70006040 (irq = 69) is a TEGRA_UART
[    1.932025] Registered UART port ttyHS1
[    1.938824] Initialized tegra uart driver
[    1.949251] loop: module loaded
[    1.956401] tegra_nand: 1 NAND chip(s) found (vend=0x2c, dev=0xa3) (Micron NAND 1GiB 1,8V 8-bit)
[    1.971237] tegra_nand: NVIDIA Tegra NAND controller @ base=0x70008000 irq=56.

Hello Toradex team,
I did some own investigation. I added some debug output to the tegra_nand.c.
It stops in function tegra_nand_probe.

And it seems it is a timing problem. When adding more or less output it stops at different parts.
When trying to change timing same…

Perhaps this helps…

But I am not able to make it run.

Bernd Hallinger

We have the same problem!
Colibri T20 510MB V1.2A but with Linux V2.3

Please test with latest BSP image.

Most possibly what you are seeing is the following issue long since fixed in our latest BSP.

As suggested, I tried the latest BSP V2.7b2 and modified it to our needs (e.g. removed different drivers, etc). But while testing, some devices still occasionally stopped while booting. After many tests and with the help of luck :wink: I found a solution which works on both of our images!

The Kernel needs to be configured to use ‘Performance’ as default CPU frequency scaling governor:

/arch/arm/configs/colibri_t20_defconfig:
- CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
+ CONFIG_CPU_FREQ_GOV_ONDEMAND=y