Ethernet is down when eMMC is full

A customer uses up every bit of eMMC on Apalis iMX6Q.

root@apalis-imx6:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 3.4G      3.4G         0 100% /
devtmpfs                849.2M      4.0K    849.2M   0% /dev
tmpfs                  1009.7M         0   1009.7M   0% /dev/shm
tmpfs                  1009.7M    360.0K   1009.3M   0% /run
tmpfs                  1009.7M         0   1009.7M   0% /sys/fs/cgroup
tmpfs                  1009.7M      4.0K   1009.7M   0% /tmp
tmpfs                  1009.7M         0   1009.7M   0% /var/volatile
/dev/mmcblk0p1           15.7M      5.3M     10.4M  34% /media/mmcblk0p1
tmpfs                   201.9M         0    201.9M   0% /run/user/0

There are lots of errors during boot.

[   65.752903]       Tainted: G           O    4.9.166-2.8.6+gd899927728be #1
[   65.752906] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   65.752914] systemd-vconsol D    0   779    575 0x00000000
[   65.752918] Backtrace: 
[   65.752932] [<8083677c>] (__schedule) from [<80836cf8>] (schedule+0x44/0xa4)
[   65.752942]  r10:e2c45f00 r9:80c20c10 r8:00000002 r7:80c20c0c r6:e26ee300 r5:ffffffff
[   65.752945]  r4:ffffe000
[   65.752956] [<80836cb4>] (schedule) from [<80837088>] (schedule_preempt_disabled+0x10/0x14)
[   65.752961]  r5:ffffffff r4:80c20c08
[   65.752972] [<80837078>] (schedule_preempt_disabled) from [<80838568>] (__mutex_lock_slowpath+0xa4/0x158)
[   65.752981] [<808384c4>] (__mutex_lock_slowpath) from [<80838674>] (mutex_lock+0x58/0x5c)
[   65.752990]  r9:024000c0 r8:80c8edec r7:00400001 r6:809470d8 r5:e2c45f00 r4:80c20c08
[   65.753003] [<8083861c>] (mutex_lock) from [<804ae3b4>] (tty_open+0x250/0x6d8)
[   65.753008]  r5:e2c45f00 r4:80947150
[   65.753019] [<804ae164>] (tty_open) from [<8020eeac>] (chrdev_open+0xd4/0x194)
[   65.753028]  r10:e2c45f00 r9:00000000 r8:e2c45f00 r7:e22dc960 r6:00000000 r5:e22af980
[   65.753031]  r4:80947150
[   65.753044] [<8020edd8>] (chrdev_open) from [<80207cd4>] (do_dentry_open.constprop.4+0x124/0x30c)
[   65.753053]  r9:00000000 r8:00020102 r7:8020edd8 r6:e2c45f08 r5:e22dc960 r4:e2c45f00
[   65.753064] [<80207bb0>] (do_dentry_open.constprop.4) from [<80208cc0>] (vfs_open+0x48/0x78)
[   65.753073]  r9:00000000 r8:00020102 r7:e0023000 r6:00000000 r5:e2c45f00 r4:e2ea1ea8
[   65.753083] [<80208c78>] (vfs_open) from [<80218bd4>] (path_openat+0x680/0x1270)
[   65.753087]  r5:00000000 r4:e2ea1ea8
[   65.753096] [<80218554>] (path_openat) from [<8021a690>] (do_filp_open+0x6c/0xd0)
[   65.753105]  r10:00000142 r9:e2ea0000 r8:80107d44 r7:00000001 r6:e2ea1f5c r5:e2ea1ea8
[   65.753108]  r4:00000004
[   65.753117] [<8021a624>] (do_filp_open) from [<802090e0>] (do_sys_open+0x168/0x1e8)
[   65.753123]  r7:fffff000 r6:e2127000 r5:ffffff9c r4:00000004
[   65.753134] [<80208f78>] (do_sys_open) from [<80209198>] (SyS_openat+0x14/0x18)
[   65.753143]  r9:e2ea0000 r8:80107d44 r7:00000142 r6:00080102 r5:00000015 r4:00000000
[   65.753155] [<80209184>] (SyS_openat) from [<80107b60>] (ret_fast_syscall+0x0/0x48)

Ethernet is managed by systemd-networkd with static IP and connman is disabled.

cat /etc/systemd/network/wired.network 
[Match]
Name=eth0

[Network]
Address=192.168.0.3/24
Gateway=192.168.0.1
DNS=8.8.8.8

Eth0 is missing after boot. But eth0 does exist at /sys/class/net

# ifconfig
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1%1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:320 errors:0 dropped:0 overruns:0 frame:0
          TX packets:320 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:24892 (24.3 KiB)  TX bytes:24892 (24.3 KiB)

usb0      Link encap:Ethernet  HWaddr 00:14:2D:FF:FF:FF  
          inet addr:192.168.11.1  Bcast:192.168.11.255  Mask:255.255.255.0
          inet6 addr: fe80::214:2dff:feff:ffff%lo/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:81 errors:0 dropped:0 overruns:0 frame:0
          TX packets:68 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:9999 (9.7 KiB)  TX bytes:16518 (16.1 KiB)

root@apalis-imx6:/sys/class/net# ls
can0  can1  eth0  lo    usb0

restarting sytemd-networkd service doesn’t help.

root@apalis-imx6:/etc/systemd/system/multi-user.target.wants# systemctl restart systemd-networkd.service                                                                                      
[  246.249100] systemd-journald[229]: Failed to open system journal: No space left on device
[  246.257616] systemd-journald[229]: Failed to open system journal: No space left on device
[  246.375396] systemd-journald[229]: Failed to open system journal: No space left on device
[  246.384071] systemd-journald[229]: Failed to open system journal: No space left on device
[  246.393423] systemd-journald[229]: Failed to open system journal: No space left on device
[  246.401830] systemd-journald[229]: Failed to open system journal: No space left on device
[  246.481562] systemd-journald[229]: Failed to open system journal: No space left on device
[  246.492950] systemd-journald[229]: Failed to open system journal: No space left on device
[  246.503485] systemd-journald[229]: Failed to open system journal: No space left on device
root@apalis-imx6:/etc/systemd/sys[  246.511929] systemd-journald[229]: Failed to open system journal: No space left on device

root@apalis-imx6:/etc/systemd/system/multi-user.target.wants# systemctl status systemd-networkd.service 
��● systemd-networkd.service - Network Service
   Loaded: loaded (/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2016-01-01 15:02:28 CST; 7s ago
     Docs: man:systemd-networkd.service(8)
 Main PID: 1362 (systemd-network)
   Status: "Processing requests..."
   CGroup: /system.slice/systemd-networkd.service
           ��└��─1362 /lib/systemd/systemd-networkd

Jan 01 15:02:28 apalis-imx6 systemd[1]: Starting Network Service...
Jan 01 15:02:28 apalis-imx6 systemd-networkd[1362]: usb0: Gained IPv6LL
Jan 01 15:02:28 apalis-imx6 systemd-networkd[1362]: Enumeration completed
Jan 01 15:02:28 apalis-imx6 systemd[1]: Started Network Service.
root@apalis-imx6:/etc/systemd/system/multi-user.target.wants# 
root@apalis-imx6:/etc/systemd/system/multi-user.target.wants# 
root@apalis-imx6:/etc/systemd/system/multi-user.target.wants# systemctl restart systemd-networkd.service

Is the full eMMC related to Ethernet function? How to avoid it at the system level e.g. reserver certain space before using up all eMMC capacity? Thanks!

The errors are related to systemd-udevd. Failed tasks includ:

[FAILED] Failed to start Rebuild Dynamic Linker Cache.
[FAILED] Failed to start Network Time Synchronization.
[FAILED] Failed to start Rebuild Journal Catalog.
[FAILED] Failed to start Update UTMP about System Boot/Shutdown.


[   73.910053] CPU: 0 PID: 685 Comm: systemd-udevd Tainted: G           O L  4.9.166-2.8.6+gd899927728be #1
[   73.910056] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[   73.910060] task: e328ac00 task.stack: e2d70000
[   73.910067] PC is at 0x484ad6
[   73.910070] LR is at 0x46f913
[   73.910077] pc : [<00484ad6>]    lr : [<0046f913>]    psr: 00010030
[   73.910077] sp : 7ea33c70  ip : 004b1cbc  fp : 0000003f
[   73.910081] r10: 0000002a  r9 : 01872538  r8 : 00491ac0
[   73.910086] r7 : 00000000  r6 : 018583e8  r5 : 76d22d78  r4 : 7ea33c84
[   73.910090] r3 : 64007065  r2 : 654b7465  r1 : dbc53465  r0 : 018583e8
[   73.910096] Flags: nzcv  IRQs on  FIQs on  Mode USER_32  ISA Thumb  Segment user
[   73.910100] Control: 10c5387d  Table: 732c804a  DAC: 00000055
[   73.910107] CPU: 0 PID: 685 Comm: systemd-udevd Tainted: G           O L  4.9.166-2.8.6+gd899927728be #1
[   73.910109] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[   73.910115] Backtrace: 
[   73.910139] [<8010b334>] (dump_backtrace) from [<8010b60c>] (show_stack+0x18/0x1c)
[   73.910146]  r7:0000000d r6:60010193 r5:00000000 r4:80c1acb0
[   73.910163] [<8010b5f4>] (show_stack) from [<803fa764>] (dump_stack+0x90/0xa4)
[   73.910175] [<803fa6d4>] (dump_stack) from [<80108780>] (show_regs+0x14/0x18)
[   73.910182]  r7:0000000d r6:dbc53465 r5:0000000b r4:e328ac00
[   73.910194] [<8010876c>] (show_regs) from [<80114b58>] (__do_user_fault+0x100/0x104)
[   73.910204] [<80114a58>] (__do_user_fault) from [<80114f78>] (do_bad_area+0x78/0x94)
[   73.910212]  r8:e2d71fb0 r7:dbc53465 r6:0000000d r5:dbc53465 r4:e2d71fb0
[   73.910220] [<80114f00>] (do_bad_area) from [<80114fa4>] (do_sect_fault+0x10/0x18)
[   73.910226]  r7:dbc53465 r6:80114f94 r5:0000000d r4:80c0811c
[   73.910237] [<80114f94>] (do_sect_fault) from [<8010134c>] (do_DataAbort+0x44/0xc0)
[   73.910245] [<80101308>] (do_DataAbort) from [<8010c4e0>] (__dabt_usr+0x40/0x60)
[   73.910250] Exception stack(0xe2d71fb0 to 0xe2d71ff8)
[   73.910257] 1fa0:                                     018583e8 dbc53465 654b7465 64007065
[   73.910266] 1fc0: 7ea33c84 76d22d78 018583e8 00000000 00491ac0 01872538 0000002a 0000003f
[   73.910274] 1fe0: 004b1cbc 7ea33c70 0046f913 00484ad6 00010030 ffffffff
[   73.910282]  r8:10c5387d r7:10c5387d r6:ffffffff r5:00010030 r4:00484ad6

The entire log is here.

In the end, deleting /etc/ld.so.cache make it right.