Apalis T30 WEC7 - Network layer fails to load sometimes

kswain · May 22, 2018, 7:13pm

On first power-up after long weekend, the networking doesn’t load. The Wifi and Wired networking icons are missing from system tray. We can’t ping the wired adapter, and you can’t open networking in control panel. This seems to only happen once in a while, more often after the product has been powered off for a few days. We have hardware power sequencing to apply 5V before 3.3V, as needed.

We are using WEC7 Apalis T30IT on our own carrier board running v2.2b2. In our custom image, we add .net 3.5 framework and netcf.3.5messages.cab. Also, we add SDIO8787 u-Blox Wifi driver from Silex. Additionally, our .bat file in startup fails to finish IPEnableRouter, however, I was able to use the reg editor to do the import manually, even though the .bat file freezes. The CPU load shows near 0-1% after boot-up.

I am unsure if this is an OS bug, a registry tweak is needed or other? It could be possible that the wired ethernet receives packets as WEC7 is booting? I am unsure if that could cause this. The platform is not plugged into a LAN, but connected to a single other device, so traffic would likely not be high in that case. I have seen similar behavior if the “soft” reboot is used from updatetool, but in this case, power was simply applied after a long weekend of being unplugged.

I’d like some ideas on what to try and how to gather more debug information. Thx

kswain · May 25, 2018, 9:00pm

Thanks, any input would be helpful. Monday works for me.

raja.tx · May 28, 2018, 2:23pm

@kswain,

Could you please log serial debug message and share with us. let us go through the log message and let you know next debugging steps.

kswain · November 13, 2018, 8:25pm

This happened to us again, but we didn’t have logs turned on. Could this be related to issue 15455? I don’t see Apalis T30 as a target, but would a similar issue exist?

kswain · November 13, 2018, 9:19pm

There is also another similar issue in ticket 23745:

raja.tx · November 15, 2018, 1:15pm

Dear @kswain,

Thank you for the reply. The PHY is different for Colibri and Apalis. Those two tickets are related to Colibri PHY AX88772. This doesn’t apply to Apalis T30.

Could you please share below details, it will help us to debug the issue.

Is this issue occurring on Toradex carrier boards or eval board or your custom board?
Could you please log serial debug message and share with us. We would like to verify whether ethernet driver is loading and only issue with network adapter stack side?
Is that occurring only one module or few or all the modules?

kswain · November 15, 2018, 4:07pm

We have seen two separate issues/behavior.

On power-up, both the wireless and wired networking system tray icon are missing. Any attempt to access the networking in control panel fails. ipconfig fails and shows nothing (lockup)
On power-up, the system tray icons for both wired and wireless are present. We are unable to ping a remote device on wired ethernet. There are 3 devices on isolated LAN, and only the WEC7 platform is unresponsive. A ping just shows “network error xxxxx”. The other two devices can talk to each other fine on the same network switch.

We have seen this on multiple units, all in our own custom carrier board. To reproduce, we have a network connection to the power supply. On bootup, after 30 seconds, the platform sends a network command to cycle power the unit. If the networking fails, power cycling stops. Note that our custom carrier board has the ublox ELLA-W131 (SD8787) for wireless.

I will repeat this test in the Toradex evaluation kit today with and without the wireless. I have a debug log from yesterdays failure (type 1), that I will attach shortly.

kswain · November 15, 2018, 4:15pm

Here’s the log file. You may have to use notepad or VIM to open it. It was created with putty for windows. The problem boot would be the last one in the sequence.

kswain · November 15, 2018, 7:33pm

We are repeating the re-boot test with the Ixora 1.0 carrier board. I believe the WEC7 image is V2.1, although we’ve seen it with 2.2b2 as well. There is no ELLA-131 Wifi in this setup. It may take a while to fail today.

Is there any “additional” debugging I can turn on that would be useful? The attached debug log is what we are logging at the moment.

kswain · November 15, 2018, 9:16pm

In the logs we uploaded, scenario 1) happened, and neither the Wifi or wired ethernet functioned. We booted to WEC7 desktop, and the log shows that NETUI didn’t load. The system was not crashed, just no networking because NETUI never loaded.

kswain · November 15, 2018, 11:12pm

We have been rebooting for nearly 5 hours with no failure on the Ixora. I will add back the SD8787 card and re-run overnight.

raja.tx · November 16, 2018, 9:42am

Dear @kswain,

Thank you for the details. Please let us know Ixora and SD8787 testing result . I am analyzing the log which you shared with us, would like to get back you early next week.

kswain · November 16, 2018, 5:24pm

We tested Ixora with WEC 2.1, and successfully re-booted all night, nearly 24 hours straight at about one reboot a minute. I think this is consistent with what you have tested and seen thus far.

Today we are changing our setup:
Apalis full evaluation board v1.2 with SD8787 dev kit plugged into MMC card slot. I am also going to upgrade our OS image to V2.2b4, and run the setup over the weekend.

kswain · November 16, 2018, 8:27pm

I’ve attached a picture of our setup to run this weekend. We have two scenarios that happen. Either NETUI doesn’t load at all at startup, or the wired network is not responsive to the point it needs power cycle to correct. I believe there to actually be two separate problems with similar resulting behavior.

This could be in combination with the SD8787 either with access point configured, not configured, with, or without antenna (in range or borderline coverage). It could also possibly still be our custom motherboard. We are testing with the evaluation board V1.1A and the latest WEC7 2.2b4 and ELLA-131 evaluation kit, hoping it’s not our motherboard.

Aside from those things, our image has touchbase AR1100 touchscreen driver (connected to COM4), .NET3.5, SDIO8787 driver, and registry a few registry settings for USB, COM ports, LVDS, LAN, WiFi etc.

Our procedure pings 20 times and then issues a reboot command over the wired network. If the wired network is down, the reboot isn’t received and the test halts. I’ll have some results for Monday.

kswain · November 19, 2018, 5:05pm

The test failed after the first couple hours. The behavior was different than before. This was a full OS freeze/lockup. No mouse or keyboard response. I’ve attached the putty log file.

Toradex Windows CE 7.0 2.2b4 for Tegra Built Sep 11 2018 16:07:01
INFO:OALLogSetZones: dpCurSettings.ulZoneMask: 0xb
L2 cache enabled
MainMemoryEndAddress adjusted from 0x9F000000 to 0x9FE00000
Main Phys Mem: 0x80000000:0x9FDFFFFF
Carveout Phys: 0x9FE00000:0x9FFFFFFF
Cold boot selected
SMP: Active CPUs = 4
Extended Mem : 0xA0000000:0xBFFFFFFF
Chip Id: 0x30 (Handheld SOC) Major: 0x1 Minor: 0x3 SKU: 0xb0
ATE prog ver 4.0
Speedo: CPU: 401 (Corner: 3), Core: 219
NVRM Initialized shmoo database
PllClocks(Mhz): X=1300, M=800, C=600, P=408, A=24.576
SysClocks(Mhz): CPU=1300, AVP=240, SysBus=240, Mem=400, EMem=800
GraphicClocks(Mhz): Host=106, 3D=133, 2D=133, Epp=133, Mpe=133, Vde=408
Loading ResourceManager...
Done Loading ResourceManager (0 ms)
Loading FlashFileSystem(NAND)...
Failed(0) Loading FlashFileSystem(NAND) (0 ms)
Loading FlashFileSystem(eMMC)...
Done Loading FlashFileSystem(eMMC) (0 ms)
Loading MultimediaManager...
Done Loading MultimediaManager (0 ms)
Loading Audio...
Loading Serial3...
Loading Serial2...
Loading Serial1...
Done Loading Serial3 (1 ms)
Done Loading Serial2 (0 ms)
Failed(0) Loading Serial1 (1 ms)
Done Loading Audio (332 ms)
Loading SDIO...
Loading SDIO2...
Loading SDIOLoading PCIe...Loading SDIO(SDIO3)...(SDIO1)...NvDdkSdioSetClockFrequency: Requested: 100 KHz, Actual: 100 KHz
NvDdkSdioSetClockFrequency: Requested: 100 KHz, Actual: 100 KHz
PCIE: port 0: link down, aborting
PCIE: port 1: link up in 0 us
PCIE: port 2: link up in 0 us
Done
Bus:Device:Function: VendorID,DeviceID

0:0:0: 0x10de,0xe1c Virtual P2P to 1

0:1:0: 0x10de,0xe1d Virtual P2P to 2

0:2:0: 0x10de,0xe1d Virtual P2P to 7

2:0:0: 0x10b5,0x8605 Virtual P2P to 3

3:1:0: 0x10b5,0x8605 Virtual P2P to 4

3:2:0: 0x10b5,0x8605 Virtual P2P to 5

3:3:0: 0x10b5,0x8605 Virtual P2P to 6

7:0:0: 0x8086,0x157b

Loading I21x Ethernet...Done
Loading USBOTG(Port1)...
Loading USBHost(Port3)...
Loading USBHost(Port2)...
Done Loading USBOTG(Port1) (7 ms)
Loading DisplayDriver...
Set LCD to 1024x600 (68Hz)
Set HDMI to 640x480 (60Hz)
Set HDMI to 640x480 (60Hz)
Done Loading DisplayDriver (76 ms)
VBUS=1, ID=1
ChangeMode(0, 3)
OTG(0): Loading USBFN...
Set LCD to 1024x600 (68Hz)
Loading USBFunction(Port1)...
Done Loading USBHost(Port3) (216 ms)
Set HDMI to 1024x768 (60Hz)
Set HDMI to 1024x768 (60Hz)
Loading Keyboard/MouseDriver...
Done Loading Keyboard/MouseDriver
Done Loading USBFunction(Port1) (154 ms)
OTG(0) Done Loading USBFN
Done Loading USBHost(Port2) (391 ms)
Done(1009)
Done(1008)
Done Loading SDIO (1011 ms)
Done Loading SDIO2 (1011 ms)
NvDdkSdioSetClockFrequency: Requested: 25000 KHz, Actual: 24727 KHz
SDIO8787 driver built on Dec 20 2016 at time: 20:23:49
Remove Keys for Comm\SDIO87871\Parms
12:00:02:09-[MRVL   moal_adapter.cpp ( 545) moal_validate_reg_pa()] Roaming AbsoluteRssi 80, RoamRssi 74
TBUPDD: 1 Got port handle 0x4020067Registering SD8787 Wireless Driver... 

TBUPDD: 2 ReadFile() Port - 0x0 error=6
Invalid time on External RTC, please set a valid time
TBUPDD: 2 Got port handle 0x53400c7
SD8787: 4-Bit mode at 25000000 Hz
NvShimUSBEventHandlerThread: Semaphore Timed Out
Dumb Charger detected
USB Shim Charger Type Detected = 16 [1-SE0, 2-SJ, 4-SK, 8-SE1, 16-Dummy]
Loading NETUI...
Done NETUI
12:00:03:614-[MRVL E moal_main.cpp    (1097) get_module_handle   ()] Invalid moal id trying to get module handle.  Returning status= MLAN_STATUS_EMPTY
12:00:03:625-[MRVL E moal_tx.cpp      ( 530) send_packet         ()] Cannot get Packet module context.  Returning status= MLAN_STATUS_EMPTY
12:00:03:834-[MRVL E moal_main.cpp    (1629) moal_pnp_resume_fsm ()] Resume FSM Exit !
12:00:03:890-[MRVL E moal_oid.cpp     (3847) validate_info_buffer()] Info buffer smaller than min size [ 0 < 300 ] in valiation routine!  Returning NDIS_STATUS_BUFFER_OVERFLOW
12:00:03:904-[MRVL E moal_oid.cpp     (3188) query_gen_supported_()] Unable to validate the size of the supported OID List!.  Returning status= NDIS_STATUS_BUFFER_OVERFLOW12:00:03:919-[MRVL E moal_oid.cpp     (3847) validate_info_buffer()] Info buffer smaller than min size [ 0 < 300 ] in valiation routine!  Returning NDIS_STATUS_BUFFER_OVERFLOW
12:00:03:934-[MRVL E moal_oid.cpp     (3188) query_gen_supported_()] Unable to validate the size of the supported OID List!.  Returning status= NDIS_STATUS_BUFFER_OVERFLOW12:00:03:959-[MRVL E moal_oid5.cpp    (2926) query_802_11_capabil()] query_802_11_capability length:112
12:00:20:774-[MRVL E moal_connect.cpp (4300) moal_attempt_connect()] Adding security IE
Setting Power Save Level to None
12:00:20:801-[MRVL E moal_connect.cpp (1289) moal_set_data_rate_p()] Setting data rate user config failed, usr_datarate_index: 0xFF
12:00:20:811-[MRVL E moal_connect.cpp (2221) start_obss_scan_time()] OBSS Scan not enabled
Warning: you are requesting IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMTYPE, which has been deprecated.  Use IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMNAME instead.
Warning: you are requesting IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMTYPE, which has been deprecated.  Use IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMNAME instead.
Warning: you are requesting IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMTYPE, which has been deprecated.  Use IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMNAME instead.
Warning: you are requesting IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMTYPE, which has been deprecated.  Use IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMNAME instead.
Warning: you are requesting IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMTYPE, which has been deprecated.  Use IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMNAME instead.
Warning: you are requesting IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMTYPE, which has been deprecated.  Use IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMNAME instead.
Toradex Bootloader 2.2b4 for Tegra Built Sep 13 2018 10:46:32

Press [SPACE] to enter Bootloader Menu

Apalis T30 1GB IT 1.1B  Serial: 2936275
RAM: 1024 MB, CarveOut: 2 MB
Locating kernel image in flash...Done(603)
Decompressing IMAGE(26343026, 51236568) from FLASH(0) to RAM(80016000)...Done(468ms)
Jumping to image at 0x80016000...




Toradex Windows CE 7.0 2.2b4 for Tegra Built Sep 11 2018 16:07:01
INFO:OALLogSetZones: dpCurSettings.ulZoneMask: 0xb
L2 cache enabled
MainMemoryEndAddress adjusted from 0x9F000000 to 0x9FE00000
Main Phys Mem: 0x80000000:0x9FDFFFFF
Carveout Phys: 0x9FE00000:0x9FFFFFFF
Cold boot selected
SMP: Active CPUs = 4
Extended Mem : 0xA0000000:0xBFFFFFFF
Chip Id: 0x30 (Handheld SOC) Major: 0x1 Minor: 0x3 SKU: 0xb0
ATE prog ver 4.0
Speedo: CPU: 401 (Corner: 3), Core: 219
NVRM Initialized shmoo database
PllClocks(Mhz): X=1300, M=800, C=600, P=408, A=24.576
SysClocks(Mhz): CPU=1300, AVP=240, SysBus=240, Mem=400, EMem=800
GraphicClocks(Mhz): Host=106, 3D=133, 2D=133, Epp=133, Mpe=133, Vde=408
Loading ResourceManager...
Done Loading ResourceManager (0 ms)
Loading FlashFileSystem(NAND)...
Failed(0) Loading FlashFileSystem(NAND) (0 ms)
Loading FlashFileSystem(eMMC)...
Done Loading FlashFileSystem(eMMC) (0 ms)
Loading MultimediaManager...
Done Loading MultimediaManager (0 ms)
Loading Audio...
Loading Serial3...
Loading Serial2...
Loading Serial1...
Done Loading Serial3 (0 ms)
Done Loading Serial2 (1 ms)
Failed(0) Loading Serial1 (0 ms)
Done Loading Audio (332 ms)
Loading SDIO...
Loading SDIO2...
Loading SDIOLoading PCIe...Loading SDIO(SDIO3)...(SDIO1)...NvDdkSdioSetClockFrequency: Requested: 100 KHz, Actual: 100 KHz
NvDdkSdioSetClockFrequency: Requested: 100 KHz, Actual: 100 KHz
PCIE: port 0: link down, aborting
PCIE: port 1: link up in 0 us
PCIE: port 2: link up in 0 us
Done
Bus:Device:Function: VendorID,DeviceID

0:0:0: 0x10de,0xe1c Virtual P2P to 1

0:1:0: 0x10de,0xe1d Virtual P2P to 2

0:2:0: 0x10de,0xe1d Virtual P2P to 7

2:0:0: 0x10b5,0x8605 Virtual P2P to 3

3:1:0: 0x10b5,0x8605 Virtual P2P to 4

3:2:0: 0x10b5,0x8605 Virtual P2P to 5

3:3:0: 0x10b5,0x8605 Virtual P2P to 6

7:0:0: 0x8086,0x157b

Loading I21x Ethernet...Done
Loading USBOTG(Port1)...
Loading USBHost(Port3)...
Loading USBHost(Port2)...
Done Loading USBOTG(Port1) (6 ms)
Loading DisplayDriver...
Set LCD to 1024x600 (68Hz)
Set HDMI to 640x480 (60Hz)
VBUS=1, ID=1
Set HDMI to 640x480 (60Hz)
ChangeMode(0, 3)
Done Loading DisplayDriver (82 ms)
OTG(0): Loading USBFN...
Set LCD to 1024x600 (68Hz)
Loading USBFunction(Port1)...
Done Loading USBHost(Port3) (228 ms)
Set HDMI to 1024x768 (60Hz)
Set HDMI to 1024x768 (60Hz)
Loading Keyboard/MouseDriver...
Done Loading Keyboard/MouseDriver
Done Loading USBFunction(Port1) (151 ms)
OTG(0) Done Loading USBFN
Done Loading USBHost(Port2) (396 ms)
Done(1008)
Done(1007)
Done Loading SDIO (1010 ms)
Done Loading SDIO2 (1010 ms)
TBUPDD: 1 Got port handle 0x532005b
NvDdkSdioSetClockFrequency: Requested: 25000 KHz, Actual: 24727 KHz
SDIO8787 driver built on Dec 20 2016 at time: 20:23:49
Remove Keys for Comm\SDIO87871\Parms
12:00:02:16-[MRVL   moal_adapter.cpp ( 545) moal_validate_reg_pa()] Roaming AbsoluteRssi 80, RoamRssi 74
TBUPDD: 2 ReadFile() Port - 0x0 error=6Registering SD8787 Wireless Driver... 

Invalid time on External RTC, please set a valid time
TBUPDD: 2 Got port handle 0x531010b
Loading NETUI...
Done NETUI
SD8787: 4-Bit mode at 25000000 Hz
NvShimUSBEventHandlerThread: Semaphore Timed Out
Dumb Charger detected
USB Shim Charger Type Detected = 16 [1-SE0, 2-SJ, 4-SK, 8-SE1, 16-Dummy]
12:00:03:629-[MRVL E moal_main.cpp    (1097) get_module_handle   ()] Invalid moal id trying to get module handle.  Returning status= MLAN_STATUS_EMPTY
12:00:03:640-[MRVL E moal_tx.cpp      ( 530) send_packet         ()] Cannot get Packet module context.  Returning status= MLAN_STATUS_EMPTY
12:00:03:849-[MRVL E moal_main.cpp    (1629) moal_pnp_resume_fsm ()] Resume FSM Exit !
12:00:03:907-[MRVL E moal_oid.cpp     (3847) validate_info_buffer()] Info buffer smaller than min size [ 0 < 300 ] in valiation routine!  Returning NDIS_STATUS_BUFFER_OVERFLOW
12:00:03:920-[MRVL E moal_oid.cpp     (3188) query_gen_supported_()] Unable to validate the size of the supported OID List!.  Returning status= NDIS_STATUS_BUFFER_OVERFLOW12:00:03:935-[MRVL E moal_oid.cpp     (3847) validate_info_buffer()] Info buffer smaller than min size [ 0 < 300 ] in valiation routine!  Returning NDIS_STATUS_BUFFER_OVERFLOW
12:00:03:950-[MRVL E moal_oid.cpp     (3188) query_gen_supported_()] Unable to validate the size of the supported OID List!.  Returning status= NDIS_STATUS_BUFFER_OVERFLOW12:00:03:991-[MRVL E moal_oid5.cpp    (2926) query_802_11_capabil()] query_802_11_capability length:112
12:00:20:805-[MRVL E moal_connect.cpp (4300) moal_attempt_connect()] Adding security IE
Setting Power Save Level to None
12:00:20:833-[MRVL E moal_connect.cpp (1289) moal_set_data_rate_p()] Setting data rate user config failed, usr_datarate_index: 0xFF
12:00:20:843-[MRVL E moal_connect.cpp (2221) start_obss_scan_time()] OBSS Scan not enabled
Warning: you are requesting IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMTYPE, which has been deprecated.  Use IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMNAME instead.
Warning: you are requesting IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMTYPE, which has been deprecated.  Use IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMNAME instead.
Warning: you are requesting IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMTYPE, which has been deprecated.  Use IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMNAME instead.
Warning: you are requesting IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMTYPE, which has been deprecated.  Use IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMNAME instead.
Warning: you are requesting IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMTYPE, which has been deprecated.  Use IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMNAME instead.
Warning: you are requesting IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMTYPE, which has been deprecated.  Use IOCTL_HAL_GET_DEVICE_INFO::SPI_GETPLATFORMNAME instead.
Toradex Bootloader 2.2b4 for Tegra Built Sep 13 2018 10:46:32

Press [SPACE] to enter Bootloader Menu

Apalis T30 1GB IT 1.1B  Serial: 2936275
RAM: 1024 MB, CarveOut: 2 MB
Locating kernel image in flash...Done(603)
Decompressing IMAGE(26343026, 51236568) from FLASH(0) to RAM(80016000)...Done(467ms)
Jumping to image at 0x80016000...




Toradex Windows CE 7.0 2.2b4 for Tegra Built Sep 11 2018 16:07:01
INFO:OALLogSetZones: dpCurSettings.ulZoneMask: 0xb
L2 cache enabled
MainMemoryEndAddress adjusted from 0x9F000000 to 0x9FE00000
Main Phys Mem: 0x80000000:0x9FDFFFFF
Carveout Phys: 0x9FE00000:0x9FFFFFFF
Cold boot selected
SMP: Active CPUs = 4
Extended Mem : 0xA0000000:0xBFFFFFFF
Chip Id: 0x30 (Handheld SOC) Major: 0x1 Minor: 0x3 SKU: 0xb0
ATE prog ver 4.0
Speedo: CPU: 401 (Corner: 3), Core: 219
NVRM Initialized shmoo database
PllClocks(Mhz): X=1300, M=800, C=600, P=408, A=24.576
SysClocks(Mhz): CPU=1300, AVP=240, SysBus=240, Mem=400, EMem=800
GraphicClocks(Mhz): Host=106, 3D=133, 2D=133, Epp=133, Mpe=133, Vde=408
Loading ResourceManager...
Done Loading ResourceManager (0 ms)
Loading FlashFileSystem(NAND)...
Failed(0) Loading FlashFileSystem(NAND) (0 ms)
Loading FlashFileSystem(eMMC)...
Done Loading FlashFileSystem(eMMC) (0 ms)
Loading MultimediaManager...
Done Loading MultimediaManager (0 ms)
Loading Audio...
Loading Serial3...
Loading Serial2...
Loading Serial1...
Done Loading Serial3 (0 ms)
Done Loading Serial2 (0 ms)
Failed(0) Loading Serial1 (0 ms)
Done Loading Audio (334 ms)
Loading SDIO...
Loading SDIO2...
Loading SDIOLoading PCIe...Loading SDIO(SDIO3)...(SDIO1)...NvDdkSdioSetClockFrequency: Requested: 100 KHz, Actual: 100 KHz
NvDdkSdioSetClockFrequency: Requested: 100 KHz, Actual: 100 KHz
PCIE: port 0: link down, aborting
PCIE: port 1: link up in 0 us
PCIE: port 2: link up in 0 us
Done
Bus:Device:Function: VendorID,DeviceID

0:0:0: 0x10de,0xe1c Virtual P2P to 1

0:1:0: 0x10de,0xe1d Virtual P2P to 2

0:2:0: 0x10de,0xe1d Virtual P2P to 7

2:0:0: 0x10b5,0x8605 Virtual P2P to 3

3:1:0: 0x10b5,0x8605 Virtual P2P to 4

3:2:0: 0x10b5,0x8605 Virtual P2P to 5

3:3:0: 0x10b5,0x8605 Virtual P2P to 6

7:0:0: 0x8086,0x157b

Loading I21x Ethernet...Done
Loading USBOTG(Port1)...
Loading USBHost(Port3)...
Loading USBHost(Port2)...
Done Loading USBOTG(Port1) (6 ms)
Loading DisplayDriver...
Set LCD to 1024x600 (68Hz)
Set HDMI to 640x480 (60Hz)
Set HDMI to 640x480 (60Hz)
VBUS=1, ID=1
Done Loading DisplayDriver (79 ms)
ChangeMode(0, 3)
OTG(0): Loading USBFN...
Set LCD to 1024x600 (68Hz)
Loading USBFunction(Port1)...
Set HDMI to 1024x768 (60Hz)
Done Loading USBHost(Port3) (225 ms)
Set HDMI to 1024x768 (60Hz)
Loading Keyboard/MouseDriver...
Done Loading Keyboard/MouseDriver
Done Loading USBFunction(Port1) (154 ms)
OTG(0) Done Loading USBFN
Done Loading USBHost(Port2) (396 ms)
Done(1009)
Done(1007)
Done Loading SDIO (1013 ms)
Done Loading SDIO2 (1012 ms)
NvDdkSdioSetClockFrequency: Requested: 25000 KHz, Actual: 24727 KHz
TBUPDD: 1 Got port handle 0x577006bSDIO8787 driver built on Dec 20 2016 at time: 20:23:49

Remove Keys for Comm\SDIO87871\Parms
12:00:02:14-[MRVL   moal_adapter.cpp ( 545) moal_validate_reg_pa()] Roaming AbsoluteRssi 80, RoamRssi 74
Registering SD8787 Wireless Driver... 
TBUPDD: 2 ReadFile() Port - 0x0 error=6
Invalid time on External RTC, please set a valid time
TBUPDD: 2 Got port handle 0x4900117
SD8787: 4-Bit mode at 25000000 Hz
NvShimUSBEventHandlerThread: Semaphore Timed Out
Loading NETUI...
Done NETUI
Dumb Charger detected
USB Shim Charger Type Detected = 16 [1-SE0, 2-SJ, 4-SK, 8-SE1, 16-Dummy]
12:00:03:624-[MRVL E moal_main.cpp    (1097) get_module_handle   ()] Invalid moal id trying to get module handle.  Returning status= MLAN_STATUS_EMPTY
12:00:03:635-[MRVL E moal_tx.cpp      ( 530) send_packet         ()] Cannot get Packet module context.  Returning status= MLAN_STATUS_EMPTY
12:00:03:844-[MRVL E moal_main.cpp    (1629) moal_pnp_resume_fsm ()] Resume FSM Exit !
12:00:03:901-[MRVL E moal_oid.cpp     (3847) validate_info_buffer()] Info buffer smaller than min size [ 0 < 300 ] in valiation routine!  Returning NDIS_STATUS_BUFFER_OVERFLOW
12:00:03:915-[MRVL E moal_oid.cpp     (3188) query_gen_supported_()] Unable to validate the size of the supported OID List!.  Returning status= NDIS_STATUS_BUFFER_OVERFLOW12:00:03:929-[MRVL E moal_oid.cpp     (3847) validate_info_buffer()] Info buffer smaller than min size [ 0 < 300 ] in valiation routine!  Returning NDIS_STATUS_BUFFER_OVERFLOW
12:00:03:945-[MRVL E moal_oid.cpp     (3188) query_gen_supported_()] Unable to validate the size of the supported OID List!.  Returning status= NDIS_STATUS_BUFFER_OVERFLOW12:00:03:971-[MRVL E moal_oid5.cpp    (2926) query_802_11_capabil()] query_802_11_capability length:112

kswain · November 19, 2018, 8:07pm

We ran the test again this morning, and it lasted about an hour and crashed again. This time, the keyboard and mouse were still responsive. The pings in the start.bat just stopped responding.

kswain · November 20, 2018, 4:32pm

We re-ran the test again over night. This time, not associating to any access point, and we had another crash.

raja.tx · November 20, 2018, 5:26pm

Dear @kswain,

Thank you for the detailed test results. We would like to suggest a few test cases to narrow down the issue.

Disable SD8787 driver and remove SD8787 adapter and rerun the test
Did you modify on standard release image e.g. registry changes and running custom application? if it is so, please undo all the changes or program the standard release image and only run your ping test for ethernet and reboot. This will help us to understand, the issue occurs on standard release image
Also, Could you please share the test application source code and let me try the same testing here and try to solve the issue
Just I saw differences in the SD8787 debug log between Apalis eval board log and your carrier board log. Let me verify the debug prints with our latest SD8787 driver and let you know if there are any update

kswain · November 20, 2018, 5:48pm

1) Disable SD8787 driver and remove SD8787 adapter and rerun the test

We already ran the test with just wired Ethernet, and it ran overnight with no crash. With the SDIO8787, it crashed with or without associated access point. It seems that SDIO8787 in combination with wired Ethernet is required.

2) Did you modify on standard release image e.g. registry changes and running custom application? if it is so, please undo all the changes or program the standard release image and only run your ping test for ethernet and reboot. This will help us to understand, the issue occurs on standard release image

We have registry setting changes for LVDS, Serial, LAN, Audio, SDIO, USB, Touchscreen. We also install .net35 and SDIO8787 driver. I’ve attached the items here.

3) Also, Could you please share the test application source code and let me try the same testing here and try to solve the issue

Our test application is just a batch file that pings 20 times to a connected wired interface. It then sends that wired interface a command over UDP to reboot. You don’t have our far end hardware for this setup, but you can probably figure out a way to issue a power cycle over ethernet in a one-shot kind of setup. Just recognize that we are doing a full hardware power cycle, and not a soft reset.

4) Just I saw differences in the SD8787 debug log between Apalis eval board log and your carrier board log. Let me verify the debug prints with our latest SD8787 driver and let you know if there are any update

I’ve attached what driver we installed above. Is this the latest driver? This is the latest that we have. This special installer also includes DLL for IPNAT.

I will re-image the setup with just the base image and SDIO8787 driver and test that. That would eliminate all the other things. It seems to fail faster when associated with Wifi Access point, so I will run it that way.

kswain · November 20, 2018, 6:37pm

Ok, we have 2.2b4 running now with just SDIO8787, .NET35, and ip address set to 192.168.0.200, 255.255.255.0, no gateway.

Our reboot batch file looks like this:

ping 192.168.0.210 -n 20 -w 1000
\FlashDisk\netsend.exe

Our application “netsend.exe” just sends “.powercycle” over UDP to 192.168.0.210:8013, where the far end platform understands that command to cycle power with power off for 4 seconds. You don’t have our hardware for the far end, so you’ll have to come up with something similar.