Wifi Interface Disappears on Verdin 1.1B IMX8M-Mini Torizon Core 6.1.0

Technical Support
, , , ,
1

Hello Toradex Team,

I am experiencing an issue with the wifi interface disappearing. This appears to be a result of the firmware crashing as indicated by dmesg. I’ve attached the dmesg trace around the time of failure. A reboot does resolve the issue, but is not viable for our production usage.

Linux verdin-imx8mm-06944118 5.15.77-6.1.0-devel+git.349786b46e61 #1-TorizonCore SMP PREEMPT Wed Dec 28 09:58:45 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
Verdin IMX8M-Mini 2GB 1.1B
Verdin Carrier Board

[106454.370957] mwifiex_sdio mmc2:0001:1: mlan0: already connected
[106454.373375] mwifiex_sdio mmc2:0001:1: info: successfully disconnected from ec:a8:1f:01:16:6e: reason code 3
[106454.379331] mwifiex_sdio mmc2:0001:1: info: trying to associate to bssid ec:a8:1f:01:16:6f
[106454.392689] mwifiex_sdio mmc2:0001:1: info: associated to bssid ec:a8:1f:01:16:6f successfully
[106481.404418] mwifiex_sdio mmc2:0001:1: info: successfully disconnected from ec:a8:1f:01:16:6f: reason code 0
[106491.628226] mwifiex_sdio mmc2:0001:1: mwifiex_cmd_timeout_func: Timeout cmd id = 0x107, act = 0x0
[106491.628259] mwifiex_sdio mmc2:0001:1: num_data_h2c_failure = 0
[106491.628269] mwifiex_sdio mmc2:0001:1: num_cmd_h2c_failure = 0
[106491.628277] mwifiex_sdio mmc2:0001:1: is_cmd_timedout = 1
[106491.628289] mwifiex_sdio mmc2:0001:1: num_tx_timeout = 0
[106491.628297] mwifiex_sdio mmc2:0001:1: last_cmd_index = 0
[106491.628308] mwifiex_sdio mmc2:0001:1: last_cmd_id: 07 01 28 00 28 00 28 00 75 00
[106491.628320] mwifiex_sdio mmc2:0001:1: last_cmd_act: 00 00 13 00 13 00 13 00 03 00
[106491.628333] mwifiex_sdio mmc2:0001:1: last_cmd_resp_index = 4
[106491.628343] mwifiex_sdio mmc2:0001:1: last_cmd_resp_id: 10 80 28 80 28 80 28 80 75 80
[106491.628352] mwifiex_sdio mmc2:0001:1: last_event_index = 4
[106491.628364] mwifiex_sdio mmc2:0001:1: last_event: 0b 00 0b 00 0a 00 0b 00 03 00
[106491.628373] mwifiex_sdio mmc2:0001:1: data_sent=0 cmd_sent=0
[106491.628386] mwifiex_sdio mmc2:0001:1: ps_mode=1 ps_state=0
[106491.629108] mwifiex_sdio mmc2:0001:1: ===mwifiex driverinfo dump start===
[106491.629123] mwifiex_sdio mmc2:0001:1: info: MWIFIEX VERSION: mwifiex 1.0 (16.68.1.p197)
[106491.629152] mwifiex_sdio mmc2:0001:1: SDIO register dump start
[106491.629391] mwifiex_sdio mmc2:0001:1: SDIO Func0 (0x0-0x9): 43 03 06 06 07 00 00 02 03 00
[106491.629535] mwifiex_sdio mmc2:0001:1: SDIO Func1 (0x10-0x17): 00 00 00 00 00 00 00 ff
[106491.629762] mwifiex_sdio mmc2:0001:1: SDIO Func1: (0x8) c3 (0x58) 00 (0x5c) 88 (0x5d) 00 (0x60) 07 (0x61) 0c (0x62) 00 (0x64) 10 (0x65) 00 (0x66) 00 (0x68) 00 (0x69) 00 (0x6a) 00
[106491.629953] mwifiex_sdio mmc2:0001:1: SDIO Func1 (0xe8-0xf2): dc fe d0 5a 8a 02 07 ea e9 01 70
[106491.642170] mwifiex_sdio mmc2:0001:1: mlan0: Ignore connection.     Card removed or FW in bad state
[106491.642426] mwifiex_sdio mmc2:0001:1: Ignore scan. Card removed or firmware in bad state
[106491.642437] mwifiex_sdio mmc2:0001:1: scan failed: -14
[106491.748086] mwifiex_sdio mmc2:0001:1: SDIO Func1 (0xe8-0xf2): dc fe d0 5a 8a 02 07 ea e9 01 70
[106491.748108] mwifiex_sdio mmc2:0001:1: SDIO register dump end
[106491.748164] mwifiex_sdio mmc2:0001:1: ===mwifiex driverinfo dump end===
[106491.748189] mwifiex_sdio mmc2:0001:1: == mwifiex firmware dump start ==
[106491.771069] mwifiex_sdio mmc2:0001:1: Fail to pull ctrl_data
[106491.771088] mwifiex_sdio mmc2:0001:1: firmware dump failed
[106491.771097] mwifiex_sdio mmc2:0001:1: == mwifiex dump information to /sys/class/devcoredump start
[106491.771219] mwifiex_sdio mmc2:0001:1: == mwifiex dump information to /sys/class/devcoredump end
[106491.771228] mwifiex_sdio mmc2:0001:1: PREP_CMD: FW is in bad state
[106491.771416] mwifiex_sdio mmc2:0001:1: info: shutdown mwifiex...
[106491.777081] mwifiex_sdio mmc2:0001:1: PREP_CMD: card is removed
[106491.778001] mwifiex_sdio mmc2:0001:1: PREP_CMD: card is removed
[106491.818526] mwifiex_sdio mmc2:0001:1: PREP_CMD: card is removed
[106491.819340] mwifiex_sdio mmc2:0001:1: PREP_CMD: card is removed
[106491.879167] mmc2: card 0001 removed
[106491.974579] mmc2: new ultra high speed SDR104 SDIO card at address 0001
[106491.984025] Bluetooth: vendor=0x2df, device=0x9142, class=255, fn=2
[106491.984355] btmrvl_sdio mmc2:0001:2: sdio device tree data not available
[106491.984533] mwifiex_sdio mmc2:0001:1: WLAN FW already running! Skip FW dnld
[106491.984540] mwifiex_sdio mmc2:0001:1: WLAN FW is active
[106497.068393] Bluetooth: module_cfg_cmd(f1) failed
[106502.508517] mwifiex_sdio mmc2:0001:1: mwifiex_cmd_timeout_func: Timeout cmd id = 0xa9, act = 0x0
[106502.508551] mwifiex_sdio mmc2:0001:1: num_data_h2c_failure = 0
[106502.508560] mwifiex_sdio mmc2:0001:1: num_cmd_h2c_failure = 0
[106502.508569] mwifiex_sdio mmc2:0001:1: is_cmd_timedout = 1
[106502.508580] mwifiex_sdio mmc2:0001:1: num_tx_timeout = 0
[106502.508589] mwifiex_sdio mmc2:0001:1: last_cmd_index = 1
[106502.508601] mwifiex_sdio mmc2:0001:1: last_cmd_id: 00 00 a9 00 00 00 00 00 00 00
[106502.508613] mwifiex_sdio mmc2:0001:1: last_cmd_act: 00 00 00 00 00 00 00 00 00 00
[106502.508624] mwifiex_sdio mmc2:0001:1: last_cmd_resp_index = 0
[106502.508635] mwifiex_sdio mmc2:0001:1: last_cmd_resp_id: 00 00 00 00 00 00 00 00 00 00
[106502.508646] mwifiex_sdio mmc2:0001:1: last_event_index = 0
[106502.508654] mwifiex_sdio mmc2:0001:1: last_event: 00 00 00 00 00 00 00 00 00 00
[106502.508665] mwifiex_sdio mmc2:0001:1: data_sent=1 cmd_sent=1
[106502.508676] mwifiex_sdio mmc2:0001:1: ps_mode=0 ps_state=0
[106502.509214] mwifiex_sdio mmc2:0001:1: info: _mwifiex_fw_dpc: unregister device
[120760.096171] audit: type=1006 audit(1677125695.533:63): pid=3287 uid=0 old-auid=4294967295 auid=1000 tty=(none) old-ses=4294967295 ses=5 res=1
[120760.097877] audit: type=1300 audit(1677125695.533:63): arch=c00000b7 syscall=64 success=yes exit=4 a0=8 a1=ffffd069ee20 a2=4 a3=0 items=0 ppid=1 pid=3287 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=5 comm="(systemd)" exe="/usr/lib/systemd/systemd" key=(null)
[120760.098220] audit: type=1327 audit(1677125695.533:63): proctitle="(systemd)"
[120760.118126] audit: type=1334 audit(1677125695.563:64): prog-id=17 op=LOAD
[120760.118199] audit: type=1300 audit(1677125695.563:64): arch=c00000b7 syscall=280 success=yes exit=8 a0=5 a1=ffffe0b071d0 a2=78 a3=0 items=0 ppid=1 pid=3287 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=5 comm="systemd" exe="/usr/lib/systemd/systemd" key=(null)
[120760.118226] audit: type=1327 audit(1677125695.563:64): proctitle="(systemd)"
[120760.118281] audit: type=1334 audit(1677125695.563:65): prog-id=0 op=UNLOAD
2

Greetings @colemurray,

Could you describe the circumstances that led to this issue. For example what were you doing with the WiFi connection and how long was the system running before experiencing this, as well as any other related information.

Also is this something you can somewhat easily reproduce?

Best Regards,
Jeremias

3

Hi @jeremias.tx,

In this case, the device was sitting idle. It had been on for somewhere around 12 hours when it crashed. This has happened previously (and what led to setting up a serial connection to capture it the next time), each time with the device sitting more/less idle.

I can provide additional logs/information if there is something specific that would help.

4

Just to confirm when you setup the WiFi connection did you do any kind of special configuration or settings or did you just set it up as shown here: Networking with TorizonCore | Toradex Developer Center

Also it looks like you just have the WiFi in client mode connected to another router/access point, is that correct? Or is it just not connected to anything?

Best Regards,
Jeremias

5

No special/non-standard configuration was done. I followed the guide and used network manager to add my wifi router to the device. It is only connected to this wifi router. It connected successfully and after some amount of time experienced this error and the interface disappeared.

6

I did a test using a Verdin i.MX8M Mini running TorizonCore 6.1.0. I connected it to the router here in my home office then let it run idly. It’s been running close to a full day now straight and the WiFi connection is still up and stable as far as I can tell.

So it doesn’t seem like I’m able to reproduce this, or maybe I just need to run this for even longer.

That said, what kind of router/environment do you have the device in? For context another customer before reported WiFi instabilities/issues as well. After further investigation it turns out the issue was with the networking infrastructure they had at their facility wasn’t playing nice with the WiFi on the module.

Now I’m not saying this is the same issue here but it might be worth considering, since I’m unable to reproduce this so far.

Best Regards,
Jeremias

7

@jeremias.tx I appreciate looking into it. Hopefully the issue will repro on your end! Let me know if there is anything further I can capture on my side that would help trace it down, should it repro again.

The environment this occurred in was a home network with a small amount of other devices on a TP-Link AC1900 router.

8

It’ll be quite hard for our teams to investigate and possibly fix this issue without a reproducible method. Or at least a concrete explanation/theory on what exactly triggers this issue for your setup. Unfortunately as you can see the logs from the WiFi driver aren’t exactly the most descriptive.

I’ll continue to see if I can reproduce this, but so far it’s not looking like it.

Best Regards,
Jeremias