Latest Mainline Linux Kernel 4.14.10 and 4.14.11 eMMC Issue

Dear All

I’m trying to test latest mainline Linux kernel with Colibri-VF_LXDE-Image_2.7b4-20171005 BSP rootfs on Apalis t30. What I did:

  1. Downloaded latest Linux kernel from: git clone -b linux-4.14.y git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
  2. Configured kernel with make tegra_defconfig, added a few drivers and patches as you can check here kernel diff and here link text
  3. After successfully starting the system, system experience random mmc problems as you can see in log (log contains every step from install new system, to creating a bug):

I had a quick look at your log file and do have some comments:

U-Boot 2015.04

=> That is a rather old U-Boot version especially if paired with the latest mainline Linux kernel. I would recommend using a later U-Boot e.g. from our BSP 2.7b4 (to be announced stable as well shortly).

Model: Toradex Apalis T30 2GB V1.0E, Serial# 02650923

=> That is a sample module released 2014-07-01. We since did quite some hardware improvements (incl. eMMC and gigabit Ethernet related). I would recommend using a current module version for your testing:
https://developer.toradex.com/products/apalis-t30#revision-history

reading apalis_t30/root.ext3

=> Unfortunately due to some ext4 issue in downstream we are still using ext3 in our Apalis T30 BSPs. However if using the mainline Linux kernel using ext4 would be very much recommended. We recently migrated to ext4 on our later modules (e.g. Apalis TK1) where you may find resp. changes required:
http://git.toradex.com/cgit/meta-toradex-demos.git/commit/?h=morty&id=28aba6b79a16d995659a2e46fe1a4be615de5c9b

[0.000000] Linux version 4.14.11-dirty (root@kamil-VirtualBox) (gcc version 5.2.1 20151005 (Linaro GCC 5.2-2015.11-2)) #2 SMP PREEMPT Thu Jan 4 09:24:14 CET 2018

=> GCC 5 is also rather outdated especially if paired with the latest mainline Linux kernel. I would recommend using a later stable Linaro GCC 6 or even GCC 7 tool chain:
https://developer.toradex.com/knowledge-base/build-u-boot-and-linux-kernel-from-source-code#Toolchain

[0.000000] Kernel command line: core_edp_mv=1300 usb_high_speed=1 ip=off root=/dev/mmcblk2p2 rw,noatime rootfstype=ext3 rootwait igb_mac=00:14:2d:28:73:2b consoleblank=0 no_console_suspend=1 console=tty1 console=ttyS0,115200n8 debug_uartport=lsport,0 vmalloc=256M mem=2035M@2048M fbmem=12M@4083M video=tegrafb0:1024x768R-16@60

=> The rw,noatime is not really correct and rather problematic. An updated U-Boot as recommended above will already fix this in its default environment:
http://git.toradex.com/cgit/u-boot-toradex.git/commit/?h=2016.11-toradex&id=ae968c66bbd50dd8538fb41fa68f904f901bb606
=> Of course the rootfstype=ext3 should also be changed to ext4 if migrating.

[4.876322] mcp251x spi1.1: Cannot initialize MCP2515. Wrong wiring?
[4.882728] mcp251x spi1.1: Probe failed, err=19

=> I believe I do have a fix ready for that somewhere but yet have to submit it. Not important if you are not using CAN I guess.

[6.582348] EXT4-fs (mmcblk2p2): mounting ext3 file system using the ext4 subsystem

=> That is really sub-optimal as we found out during stress testing in our temperature chambers. However I can’t remember on top of my head right now whether the errors we saw looked exactly like what you are seeing now.

[  680.880178] mmc2: Card stuck being busy! mmc_poll_for_busy
[  680.885663] mmc2: Error -110 starting bkops

=> I vaguely remember already having mentioned once to you before that bkops aka hpi is quite broken on those SKHynix resp. Hynix eMMC parts as assembled on those earlier samples. However if using later module versions as suggested above this should no longer be an issue. Otherwise you really do need disabling this by e.g. adding the following to the resp. eMMC sdhci device tree node:

mmccard@0 {
	status = "okay";
	compatible = "mmc-card";
	reg = <0>;
	broken-hpi;
};

With that I guess you do already have something to chew on for now. Let us know how it goes and we are happy to assist you further. Good luck!

So I have applied all the suggestions, but probably I have a problem with ext4. Could you point me where I have made a mistake? What I did:

  1. I have changed creation of image to ext4 as you can see in file diff file
  2. during system update still got message: reading apalis_t30/root.ext3-100 Unable to read file apalis_t30/root.ext3-100 **
    so I did a trick and changed all file names on SD card from root.ext4-
    to root.ext3-
  3. System uploaded succesfully (rootfs from Colibri-VF_LXDE-Image_2.7.4) but I getting kernel panic every boot:

resolved problem with ext4 by running a command: ./mk-u-boot-scripts.sh in bsp to rebuild images

After all the suggestion applied, still same problem with mmc, ethernet stoped working and there is a problem with all usb devices kernel diff and log here kernel diff system boot

  • about the internet, cause I have change board revision to: Model: Toradex Apalis T30 2GB V1.1A, Serial# 02715015 I had to reverse this patch:

    -#define E1000_DEV_ID_I210_COPPER_FLASHLESS 0x157B
    +#define E1000_DEV_ID_I210_COPPER_FLASHLESS 0x1531

  • about usb I have made some tests and I have noticed that, when applying new uboot from Apalis-T30_LXDE-Image_2.7b4-20171005 with mainline, usb devices like mouse etc stops working, old u-boot + mainline = usb devices works fine, new u-boot + downstream kernel usb =devices works fine

about the internet, cause I have change board revision to: Model: Toradex Apalis T30 2GB V1.1A, Serial# 02715015 I had to reverse this patch:

-#define E1000_DEV_ID_I210_COPPER_FLASHLESS 0x157B +#define E1000_DEV_ID_I210_COPPER_FLASHLESS 0x1531

Yes, volume production modules now have the i210/i211 gigabit Ethernet chips fused and do no longer require the tools only hack.

about usb I have made some tests and I have noticed that, when applying new uboot from Apalis-T30_LXDE-Image_2.7b4-20171005 with mainline, usb devices like mouse etc stops working, old u-boot + mainline = usb devices works fine, new u-boot + downstream kernel usb =devices works fine

Yes, I believe there is something askew there but I have not gotten around convincing anybody about this or fixing it myself. For the time being an easy workaround is to just do usb start from within U-Boot prior to booting the mainline Linux kernel.

after adding this command to bootcmd got usb working, but still facing 2 issues:

Net:   No ethernet found.
Hit any key to stop autoboot:  0 
starting USB...
USB0:   USB EHCI 1.00
USB1:   USB EHCI 1.00
USB2:   USB EHCI 1.00
scanning bus 0 for devices... 1 USB Device(s) found
scanning bus 1 for devices... EHCI timed out on TD - token=0x80008c80
unable to get device descriptor (error=-1)
3 USB Device(s) found
scanning bus 2 for devices... 2 USB Device(s) found

no ethernet in u-boot mode and mmc problem even after I have added broken-hpi to device tree.

[ 9301.049777] mmc2: Card stuck being busy! mmc_poll_for_busy
[ 9301.055269] mmc2: Error -110 starting bkops

for full log take a look in my 2 post above.

Hi

the Ethernet is connected to pci, so need to enable it using pci enum.
If you follow the steps above, bkops should not be loaded. Could you send a complete boot log before and after these changes. Thanks