NFS - Kernel panic - iMx6

Dear Community,

I’m facing an issue with NFS setting, my custom board does not boot over NFS.
I’ve followed the steps described in “Developper section” to install and configure tftp-hpa and nfs-kernel-server on my host machine.

The TFTP works fine, I’m able to load uImage and device tree with tftp command in u-boot prompt.
The NFS works fine, I’m able to mount the shared rootfs on embedded target through network (read/write is also ok).

But, when I want to achieve the final step i.e boot over NFS, the boot session stucks at the same point:

Apalis iMX6 # run nfsboot
Booting via DHCP/TFTP/NFS...
Using FEC device
TFTP from server 192.168.3.4; our IP address is 192.168.3.102
Filename 'test.dtb'.
Load address: 0x12000000
Loading: ##################################################  45 KiB
         2.1 MiB/s
done
Bytes transferred = 46122 (b42a hex)
Using FEC device
TFTP from server 192.168.3.4; our IP address is 192.168.3.102
Filename 'uImage'.
Load address: 0x11000000
Loading: ##################################################  4.1 MiB
         6.5 MiB/s
done
Bytes transferred = 4327224 (420738 hex)
## Booting kernel from Legacy Image at 11000000 ...
   Image Name:   Linux-3.14.28-v2.6b1+g0bb38d1
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    4327160 Bytes = 4.1 MiB
   Load Address: 10008000
   Entry Point:  10008000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 12000000
   Booting using the fdt blob at 0x12000000
   Loading Kernel Image ... OK
   Using Device Tree in place at 12000000, end 1200e429

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.14.28-v2.6b1+g0bb38d1 (vincent@VM) (gcc version 5.2.0 (GCC) ) #2 SMP Thu Feb 13 02:35:21 CET 2020
[    0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] Machine model: Toradex Apalis iMX6Q/D Module on XX
[    0.000000] cma: CMA: reserved 320 MiB at 3c000000
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] PERCPU: Embedded 8 pages/cpu @ab713000 s8512 r8192 d16064 u32768
[    0.000000] Kernel command line: enable_wait_mode=off vmalloc=400M nfsrootdebug root=/dev/nfs rw ip=192.168.3.102:192.168.3.4:192.168.3.4:255.255.255.0::eth0 nfsroot=192.168.3.4:/srv/screeni fec_mac=00:14:2d:00:00:00 consoleblank=0 no_console_suspend=1 console=tty1 console=ttymxc0,115200n8 vt.global_cursor_default=0 video=mxcfb0:dev=ldb,if=LVDS888 video=mxcfb1:off video=mxcfb2:off video=mxcfb3:off fbmem=32M
[    0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
[    0.000000]     vmalloc : 0xc0800000 - 0xff000000   (1000 MB)
[    0.000000]     lowmem  : 0x80000000 - 0xc0000000   (1024 MB)
[    0.000000]     pkmap   : 0x7fe00000 - 0x80000000   (   2 MB)
[    0.000000]     modules : 0x7f000000 - 0x7fe00000   (  14 MB)
[    0.000000]       .text : 0x80008000 - 0x80733fcc   (7344 kB)
[    0.000000]       .init : 0x80734000 - 0x8077b140   ( 285 kB)
[    0.000000]       .data : 0x8077c000 - 0x807b62c0   ( 233 kB)
[    0.000000]        .bss : 0x807b62c8 - 0x80819f28   ( 400 kB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] NR_IRQS:16 nr_irqs:16 16
[    0.000000] L310 cache controller enabled
[    0.000000] l2x0: 16 ways, CACHE_ID 0x410000c7, AUX_CTRL 0x32470000, Cache size: 1024 kB
[    0.000000] Switching to timer-based delay loop
[    0.000007] sched_clock: 32 bits at 3000kHz, resolution 333ns, wraps every 1431655765682ns
[    0.000797] Console: colour dummy device 80x30
[    0.001153] console [tty1] enabled
[    0.001177] Calibrating delay loop (skipped), value calculated using timer frequency.. 6.00 BogoMIPS (lpj=30000)
[    0.001205] pid_max: default: 32768 minimum: 301
[    0.001377] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.001399] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.009199] Initializing cgroup subsys debug
[    0.009224] Initializing cgroup subsys freezer
[    0.009260] CPU: Testing write buffer coherency: ok
[    0.009563] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.009643] Setting up static identity map for 0x1057fac0 - 0x1057fb18
[    0.011257] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.012060] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
[    0.012848] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
[    0.012950] Brought up 4 CPUs
[    0.013011] SMP: Total of 4 processors activated.
[    0.013025] CPU: All CPU(s) started in SVC mode.
[    0.013616] devtmpfs: initialized
[    0.017320] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[    0.017663] pinctrl core: initialized pinctrl subsystem
[    0.017984] regulator-dummy: no parameters
[    0.047514] NET: Registered protocol family 16
[    0.055927] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.056516] cpuidle: using governor ladder
[    0.056535] cpuidle: using governor menu
[    0.056750] CPU identified as i.MX6Q, unknown revision
[    0.056848] Use WDOG1 as reset source
[    0.064923] syscon 20c8000.anatop: regmap [mem 0x020c8000-0x020c8fff] registered
[    0.065251] vdd1p1: 800 <--> 1375 mV at 1100 mV
[    0.065615] vdd3p0: 2800 <--> 3150 mV at 3000 mV
[    0.065963] vdd2p5: 2000 <--> 2750 mV at 2400 mV
[    0.066330] vddarm: 725 <--> 1450 mV at 1150 mV
[    0.070485] vddpu: 725 <--> 1450 mV at 1150 mV
[    0.070862] vddsoc: 725 <--> 1450 mV at 1175 mV
[    0.072786] syscon 20e0000.iomuxc-gpr: regmap [mem 0x020e0000-0x020e0037] registered
[    0.075526] syscon 21bc000.ocotp-ctrl: regmap [mem 0x021bc000-0x021bffff] registered
[    0.079833] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[    0.079860] hw-breakpoint: maximum watchpoint size is 4 bytes.
[    0.080362] imx6q-pinctrl 20e0000.iomuxc: initialized IMX pinctrl driver
[    0.087202] bio: create slab <bio-0> at 0
[    0.087981] 1P8V: 1800 mV
[    0.088194] 2P5V: 2500 mV
[    0.088411] 3P3V: 3300 mV
[    0.088718] usb_otg_vbus: 5000 mV
[    0.088989] usb_host_vbus_hub: 5000 mV
[    0.089299] usb_host_vbus: 5000 mV
[    0.089364] usb_host_vbus: supplied by usb_host_vbus_hub
[    0.089821] SCSI subsystem initialized
[    0.090092] usbcore: registered new interface driver usbfs
[    0.090172] usbcore: registered new interface driver hub
[    0.090284] usbcore: registered new device driver usb
[    0.090755] i2c-gpio i2c.16: using pins 80 (SDA) and 62 (SCL)
[    0.091293] i2c i2c-1: IMX I2C adapter registered
[    0.093376] stmpe-i2c 2-0041: stmpe811 detected, chip id: 0x811
[    0.095589] i2c i2c-2: IMX I2C adapter registered
[    0.096159] i2c i2c-3: IMX I2C adapter registered
[    0.096303] Linux video capture interface: v2.00
[    0.096366] pps_core: LinuxPPS API ver. 1 registered
[    0.096382] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.096417] PTP clock support registered
[    0.110522] imx-ipuv3 2400000.ipu: IPU DMFC NORMAL mode: 1(0~1), 5B(4,5), 5F(6,7)
[    0.130520] imx-ipuv3 2800000.ipu: IPU DMFC NORMAL mode: 1(0~1), 5B(4,5), 5F(6,7)
[    0.131896] Switched to clocksource mxc_timer1
[    0.141502] NET: Registered protocol family 2
[    0.142340] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[    0.142435] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[    0.142568] TCP: Hash tables configured (established 8192 bind 8192)
[    0.142641] TCP: reno registered
[    0.142664] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    0.142724] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    0.143003] NET: Registered protocol family 1
[    0.143280] RPC: Registered named UNIX socket transport module.
[    0.143301] RPC: Registered udp transport module.
[    0.143316] RPC: Registered tcp transport module.
[    0.143329] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.143865] hw perfevents: enabled with ARMv7 Cortex-A9 PMU driver, 7 counters available
[    0.144858] imx6_busfreq busfreq.14: DDR medium rate not supported.
[    0.145343] Bus freq driver module loaded
[    0.146233] futex hash table entries: 1024 (order: 4, 65536 bytes)
[    0.153183] VFS: Disk quotas dquot_6.5.2
[    0.155072] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.155784] NFS: Registering the id_resolver key type
[    0.155852] Key type id_resolver registered
[    0.155870] Key type id_legacy registered
[    0.156507] fuse init (API version 7.22)
[    0.157047] msgmni has been set to 2012
[    0.159981] io scheduler noop registered
[    0.160005] io scheduler deadline registered
[    0.160183] io scheduler cfq registered (default)
[    0.161390] backlight.15 supply power not found, using dummy regulator
[    0.162805] MIPI DSI driver module loaded
[    0.163297] mxc_sdc_fb fb.17: registered mxc display driver ldb
[    0.181964] mxc_sdc_fb fb.17: 1280x800 h_sync,r,l: 40,64,64  v_sync,l,u: 6,5,5 pixclock=68932000 Hz
[    0.201963] imx-ipuv3 2800000.ipu: IPU DMFC DP HIGH RESOLUTION: 1(0,1), 5B(2~5), 5F(6,7)
[    0.251966] mxc_sdc_fb fb.17: 1280x800 h_sync,r,l: 40,64,64  v_sync,l,u: 6,5,5 pixclock=68932000 Hz
[    0.281165] Console: switching to colour frame buffer device 160x50
[    0.316399] imx-sdma 20ec000.sdma: no iram assigned, using external mem
[    0.316722] imx-sdma 20ec000.sdma: no event needs to be remapped
[    0.316991] imx-sdma 20ec000.sdma: loaded firmware 3.1
[    0.319805] imx-sdma 20ec000.sdma: initialized
[    0.321408] pfuze100-regulator 2-0008: Full layer: 2, Metal layer: 1
[    0.322113] pfuze100-regulator 2-0008: FAB: 0, FIN: 0
[    0.322197] pfuze100-regulator 2-0008: pfuze100 found.
[    0.323716] SW1AB: 300 <--> 1875 mV at 1375 mV
[    0.325138] SW1C: 300 <--> 1875 mV at 1375 mV
[    0.325972] SW2: at 1000 mV
[    0.326772] SW3A: 400 <--> 1975 mV at 1500 mV
[    0.327598] SW3B: at 1500 mV
[    0.328407] SW4: at 1000 mV
[    0.329206] SWBST: 5000 <--> 5150 mV at 5000 mV
[    0.330036] VSNVS: 1000 <--> 3000 mV at 3000 mV
[    0.330291] VREFDDR: 750 mV
[    0.331080] VGEN1: 800 <--> 1550 mV at 1500 mV
[    0.331917] VGEN2: 800 <--> 1550 mV at 1500 mV
[    0.332731] VGEN3: 1800 <--> 3300 mV at 3000 mV
[    0.336474] VGEN4: 1800 <--> 3300 mV at 1800 mV
[    0.340160] VGEN5: 1800 <--> 3300 mV at 2500 mV
[    0.343788] VGEN6: 1800 <--> 3300 mV at 2800 mV
[    0.346953] Serial: IMX driver
[    0.350103] 2020000.serial: ttymxc0 at MMIO 0x2020000 (irq = 58, base_baud = 5000000) is a IMX
[    1.197174] console [ttymxc0] enabled
[    1.204583] 21f0000.serial: ttymxc3 at MMIO 0x21f0000 (irq = 61, base_baud = 5000000) is a IMX
[    1.216682] serial: Freescale lpuart driver
[    1.224311] imx sema4 driver is registered.
[    1.231503] [drm] Initialized drm 1.1.0 20060810
[    1.239439] [drm] Initialized vivante 1.0.0 20120216 on minor 0
[    1.255696] brd: module loaded
[    1.265600] loop: module loaded
[    1.272143] altera_ps altera-ps.28: ALTERA Passive Serial interface driver
[    1.283425] imx6q-pinctrl 20e0000.iomuxc: pin MX6Q_PAD_EIM_D24 already requested by 2020000.serial; cannot claim for 2008000.ecspi
[    1.298268] imx6q-pinctrl 20e0000.iomuxc: pin-45 (2008000.ecspi) status -22
[    1.308325] imx6q-pinctrl 20e0000.iomuxc: could not request pin 45 (MX6Q_PAD_EIM_D24) from group spi_csfpga  on device 20e0000.iomuxc
[    1.323548] spi_imx 2008000.ecspi: Error applying setting, reverse things back
[    1.335653] spi_imx 2008000.ecspi: probed
[    1.344330] 2188000.ethernet supply phy not found, using dummy regulator
[    1.372259] pps pps0: new PPS source ptp0
[    1.380515] libphy: fec_enet_mii_bus: probed
[    1.389565] fec 2188000.ethernet eth0: registered PHC device 0
[    1.398922] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    1.408886] usbcore: registered new interface driver usb-storage
[    1.418991] 2184800.usbmisc supply vbus-wakeup not found, using dummy regulator
[    1.433497] ci_hdrc ci_hdrc.0: doesn't support gadget
[    1.441870] ci_hdrc ci_hdrc.0: EHCI Host Controller
[    1.450019] ci_hdrc ci_hdrc.0: new USB bus registered, assigned bus number 1
[    1.471929] ci_hdrc ci_hdrc.0: USB 2.0 started, EHCI 1.00
[    1.481256] hub 1-0:1.0: USB hub found
[    1.488198] hub 1-0:1.0: 1 port detected
[    1.498976] ci_hdrc ci_hdrc.1: doesn't support gadget
[    1.509309] ci_hdrc ci_hdrc.1: EHCI Host Controller
[    1.517348] ci_hdrc ci_hdrc.1: new USB bus registered, assigned bus number 2
[    1.541925] ci_hdrc ci_hdrc.1: USB 2.0 started, EHCI 1.00
[    1.551146] hub 2-0:1.0: USB hub found
[    1.558112] hub 2-0:1.0: 1 port detected
[    1.574542] input: stmpe-ts as /devices/soc0/soc.0/2100000.aips-bus/21a4000.i2c/i2c-2/2-0041/stmpe-ts/input/input0
[    1.588717] check_alarm_past: alarm in the past
[    1.596879] snvs_rtc 20cc034.snvs-rtc-lp: rtc core: registered 20cc034.snvs-rtc-lp as rtc0
[    1.608679] i2c /dev entries driver
[    1.617082] mxc_v4l2_output v4l2_out.25: V4L2 device registered as video16
[    1.627652] mxc_v4l2_output v4l2_out.25: V4L2 device registered as video17
[    1.670854] sbs-battery 3-000b: sbs-battery: battery gas gauge device registered
[    1.682598] imx2-wdt 20bc000.wdog: IMX2+ Watchdog Timer enabled. timeout=60s (nowayout=0)
[    1.694730] sdhci: Secure Digital Host Controller Interface driver
[    1.704541] sdhci: Copyright(c) Pierre Ossman
[    1.712534] sdhci-pltfm: SDHCI platform and OF driver helper
[    1.723901] mmc1: no vqmmc regulator found
[    1.771921] mmc1: SDHCI controller on 2190000.usdhc [2190000.usdhc] using ADMA
[    1.784520] mmc2: no vqmmc regulator found
[    1.811960] usb 1-1: new full-speed USB device number 2 using ci_hdrc
[    1.831927] mmc2: SDHCI controller on 2194000.usdhc [2194000.usdhc] using ADMA
[    1.843529] mmc0: no vqmmc regulator found
[    1.891924] mmc0: SDHCI controller on 2198000.usdhc [2198000.usdhc] using ADMA
[    1.903479] mxc_vdoa 21e4000.vdoa: i.MX Video Data Order Adapter(VDOA) driver probed
[    1.916946] mxc_vpu 2040000.vpu: VPU initialized
[    1.925672] hidraw: raw HID events driver (C) Jiri Kosina
[    1.935132] usbcore: registered new interface driver usbhid
[    1.944582] usbhid: USB HID core driver
[    1.952479] TCP: cubic registered
[    1.960198] NET: Registered protocol family 10
[    1.969458] sit: IPv6 over IPv4 tunneling driver
[    1.978477] NET: Registered protocol family 17
[    1.986722] Key type dns_resolver registered
[    1.996111] Registering SWP/SWPB emulation handler
[    2.005269] SW4: disabling
[    2.011503] SW3B: disabling
[    2.019519] SW2: disabling
[    2.019829] input: eGalax Inc. eGalaxTouch EXC7200-786Bv1.000           as /devices/soc0/soc.0/2100000.aips-bus/2184000.usb/ci_hdrc.0/usb1/1-1/1-1:1.0/0003:0EEF:7200.0001/input/input1
[    2.020145] input: eGalax Inc. eGalaxTouch EXC7200-786Bv1.000           Pen as /devices/soc0/soc.0/2100000.aips-bus/2184000.usb/ci_hdrc.0/usb1/1-1/1-1:1.0/0003:0EEF:7200.0001/input/input2
[    2.020521] hid-multitouch 0003:0EEF:7200.0001: input,hiddev0,hidraw0: USB HID v2.10 Pointer [eGalax Inc. eGalaxTouch EXC7200-786Bv1.000          ] on usb-ci_hdrc.0-1/input0
[    2.093729] regulator-dummy: disabling
[    2.102917] imx mcc test is registered.
[    2.113603] input: gpio-keys.27 as /devices/soc0/gpio-keys.27/input/input4
[    2.127976] snvs_rtc 20cc034.snvs-rtc-lp: setting system clock to 2020-03-12 19:38:33 UTC (1584041913)
[    2.141966] usb 2-1: new high-speed USB device number 2 using ci_hdrc
[    2.241779] fec 2188000.ethernet eth0: Freescale FEC PHY driver [Micrel KSZ9031 Gigabit PHY] (mii_bus:phy_addr=2188000.ethernet:07, irq=190)
[    2.260130] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    2.313053] hub 2-1:1.0: USB hub found
[    2.320409] hub 2-1:1.0: 4 ports detected
[    2.327126] mmc0: BKOPS_EN bit is not set
[    2.342499] mmc0: new high speed DDR MMC card at address 0001
[    2.352172] mmcblk0: mmc0:0001 Q2J54A 3.64 GiB
[    2.360290] mmcblk0boot0: mmc0:0001 Q2J54A partition 1 2.00 MiB
[    2.369825] mmcblk0boot1: mmc0:0001 Q2J54A partition 2 2.00 MiB
[    2.380439]  mmcblk0: p1 p2
[    2.388842]  mmcblk0boot1: unknown partition table
[    2.398989]  mmcblk0boot0: unknown partition table
[   12.261972] Waiting up to 110 more seconds for network.
[   22.271972] Waiting up to 100 more seconds for network.
[   26.187331] libphy: 2188000.ethernet:07 - Link is Up - 100/Full
[   26.201996] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   26.222337] IP-Config: Complete:
[   26.233568]      device=eth0, hwaddr=00:14:2d:00:00:00, ipaddr=192.168.3.102, mask=255.255.255.0, gw=192.168.3.4
[   26.251958]      host=192.168.3.102, domain=, nis-domain=(none)
[   26.261133]      bootserver=192.168.3.4, rootserver=192.168.3.4, rootpath=

From here, if I wait a couple of seconds, the kernel panic raises.

I suppose that there is a misconfiguration somewhere cause the elements work fine independently.

BR

Vincentt

EDIT: same result with apalis-imx6_console-image-tezi_3.0b4.254-20200421 BSP, stucked at :

[    6.263080] IP-Config: Complete:
[    6.270203]      device=eth0, hwaddr=00:14:2d:00:00:00, ipaddr=192.168.3.102, mask=255.255.255.0, gw=192.168.3.4
[    6.284444]      host=192.168.3.102, domain=, nis-domain=(none)
[    6.294398]      bootserver=192.168.3.4, rootserver=192.168.3.4, rootpath=
[    6.305907] ALSA device list:
[    6.312035]   #0: imx6q-apalis-sgtl5000

Hi @Vincentt

Thanks for writing to the Toradex Support.

The TFTP works fine, I’m able to load uImage and device tree with tftp command in u-boot prompt. The NFS works fine, I’m able to mount the shared rootfs on embedded target through network (read/write is also ok).

We stopped using uImage in Bsp 2.8. Could you update to Bsp 2.8.7 and check if the issue is still present.

EDIT: same result with apalis-imx6_console-image-tezi_3.0b4.254-20200421 BSP, stucked at :

Have you done any changes to the kernel or device-tree? If yes, can you share these changes?

Best regards,
Jaski

Dear @jaski.tx ,

I’ve done no change on kernel or device tree on the apalis-imx6_console-image-tezi_3.0b4.254-20200421 BSP, the main idea was just to check with a clean image.

I have succeed to deploy through NFS server, but after a test to boot on another rootfs, it no longer works…
I will check on the switch device to be sure that there is no parameter that deny NFS protocol or else.
It works when I am connected directly on my laptop, I don’t understand why for now…

Hi @Vincentt

Thanks for the Input.
It seems to be really a switch or network configuration issue.

Best regards,
Jaski