We have issues where the WiFi modules stop responding to the driver.
We’ve had this issue in the past but it seems to be less frequent after switching to the proprietary driver (for WPA3 support), but it seems to be back again.
I’m not in a position to try this with a reference image on a reference carrier board at the moment, but I’m hoping someone might have suggestions anyways. It might be related to background scanning, or it might be totally unrelated. We’ve seen this with both the driver included with the kernel, and the proprietary driver, with the firmware you normally get with the Yocto build, and the firmware included with the prop driver.
We have a custom carrier board, and have observed this on both Verdin iMX8MP Q 4GB WB IT V1.1A and V1.1B. We run a custom Yocto image
Linux verdin-imx8mp 5.15.129-6.4.0-devel+git.67c3153d20ff #1 SMP PREEMPT Wed Sep 27 12:30:36 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
Only the WiFi crash (we’re not running BT, so I’m not sure about the BT stack) Linux itself and other drivers (like inserting a USB-C to Ethernet) still work.
mlan0 (uap0 and wfd0 also) disappear after the driver crash (as expected). Trying to unload and reload the driver doesn’t fix the issue either. Log of that is also included below.
Rebooting fixes the issue.
We’re running the following layer versions
meta-arm b187fb9232ca0a6b5f8f90b4715958546fc41d73
meta-clang 79169d9be565b7a87310ca280d3a21aaf608ce33
meta-flutter a130deb48a7ccaa8e4a082ad14572490d69d8f41
meta-freescale 3e9ef23d98aa842cf84251a27c9b8dde8925ea61
meta-freescale-3rdparty 1a6ea560015ecda5fff8eccaf5b1327f1bb6c57f
meta-freescale-distro d5bbb487b2816dfc74984a78b67f7361ce404253
meta-openembedded a88cb922f91fda95e8a584cee3092083d5ad3e98
meta-tem 21c89d8aef896e9aae4c4edc38c19ac30ed0c1fd
meta-toradex-bsp-common e4fa05527dcecbddd6aa1b2bc7deebbf53c6bd83
meta-toradex-demos 655ddc7a45b883d705407b6a002aac8191b58c7d
meta-toradex-distro 9480d591b81316ce0e828f37bed74329de2a5d81
meta-toradex-nxp 0dd8d488b694212cf02357c4651183938994e484
meta-yocto 8b50fe692a24a80b5c3cd1f816bcdd3e0b00418a
openembedded-core 6c7fef37d4286f6bfc7b1dcb2d1e543a110a7f6f
openembedded-core/bitbake 41b6684489d0261753344956042be2cc4adb0159
Here’s the output from dmesg around the error, including the last messages before the crash.
[28028.004034] wlan: mlan0 START SCAN
[28033.064601] wlan: SCAN COMPLETED: scanned AP count=61
[28034.651319] wlan: mlan0 START SCAN
[28039.822750] wlan: SCAN COMPLETED: scanned AP count=61
[28043.003692] wlan: mlan0 START SCAN
[28048.117389] wlan: SCAN COMPLETED: scanned AP count=61
[28049.678928] wlan: mlan0 START SCAN
[28054.773530] wlan: SCAN COMPLETED: scanned AP count=61
[28058.006570] wlan: mlan0 START SCAN
[28063.067826] wlan: SCAN COMPLETED: scanned AP count=60
[28064.706885] wlan: mlan0 START SCAN
[28085.212202] Ext scan enh timeout
[28085.212226] ------------Dump info-----------
[28085.212233] EXT_SCAN_STATUS event Timeout
[28085.212239] CurCmd Empty
[28085.212245] mlan_processing =0
[28085.212253] main_lock_flag =0
[28085.212259] main_process_cnt =167954
[28085.212266] delay_task_flag =0
[28085.212271] mlan_rx_processing =0
[28085.212276] rx_pkts_queued=0
[28085.212282] more_task_flag = 0
[28085.212287] num_cmd_timeout = 0
[28085.212293] last_cmd_index = 1
[28085.212298] last_cmd_id =
[28085.212303] 0x107
[28085.212308] 0x107
[28085.212313] 0x107
[28085.212318] 0x107
[28085.212323] 0x107
[28085.212329] 0x107
[28085.212334] 0x107
[28085.212339] 0x107
[28085.212344] 0x107
[28085.212349] 0x107
[28085.212359] last_cmd_act =
[28085.212363] 0x1
[28085.212368] 0x1
[28085.212373] 0x1
[28085.212378] 0x1
[28085.212383] 0x1
[28085.212388] 0x1
[28085.212393] 0x1
[28085.212398] 0x1
[28085.212403] 0x1
[28085.212408] 0x1
[28085.212418] last_cmd_resp_index = 1
[28085.212423] last_cmd_resp_id =
[28085.212428] 0x8107
[28085.212433] 0x8107
[28085.212438] 0x8107
[28085.212443] 0x8107
[28085.212448] 0x8107
[28085.212453] 0x8107
[28085.212458] 0x8107
[28085.212463] 0x8107
[28085.212468] 0x8107
[28085.212473] 0x8107
[28085.212484] last_event_index = 3
[28085.212489] last_event =
[28085.212493] 0x58
[28085.212498] 0x58
[28085.212503] 0x58
[28085.212508] 0x58
[28085.212513] 0x58
[28085.212518] 0x58
[28085.212524] 0x58
[28085.212529] 0x58
[28085.212533] 0x58
[28085.212538] 0x58
[28085.212548] num_data_h2c_failure = 0
[28085.212553] num_cmd_h2c_failure = 0
[28085.212559] num_data_c2h_failure = 0
[28085.212565] num_cmdevt_c2h_failure = 0
[28085.212570] num_int_read_failure = 0
[28085.212575] last_int_status = 64
[28085.212580] num_alloc_buffer_failure = 0
[28085.212586] num_pkt_dropped = 0
[28085.212592] num_no_cmd_node = 0
[28085.212597] num_event_deauth = 0
[28085.212602] num_event_disassoc = 0
[28085.212608] num_event_link_lost = 0
[28085.212613] num_cmd_deauth = 1
[28085.212619] num_cmd_assoc_success = 1
[28085.212624] num_cmd_assoc_failure = 0
[28085.212629] num_cons_assoc_failure = 0
[28085.212635] cmd_resp_received=0
[28085.212640] event_received=0
[28085.212645] max_tx_buf_size=4096
[28085.212650] tx_buf_size=0
[28085.212656] curr_tx_buf_size=3072
[28085.212663] data_sent=0 cmd_sent=0
[28085.212669] ps_mode=0 ps_state=0
[28085.212675] wakeup_dev_req=0 wakeup_tries=0 wakeup_timeout=0
[28085.212683] hs_configured=0 hs_activated=0
[28085.212688] pps_uapsd_mode=0 sleep_pd=0
[28085.212694] tx_lock_flag = 0
[28085.212702] scan_processing = 1
[28085.212707] scan_state = 0x85
[28085.212713] bypass_pkt_count=0
[28085.212721] mp_rd_bitmap=0x0 curr_rd_port=0x1a
[28085.212727] mp_wr_bitmap=0xffe00000 curr_wr_port=0x15
[28085.212733] mp_data_port_mask = 0xffffffff
[28085.212739] last_recv_rd_bitmap=0x2000000 mp_invalid_update=1
[28085.212745] last_recv_wr_bitmap=0xffffffff last_mp_index=1
[28085.212752] mp_wr_bitmap: 0xffe00000 mp_wr_ports=0x10014 len=256 curr_wr_port=0x15
[28085.212760] 0xfb
[28085.212765] 0x00
[28085.212770] 0x00
[28085.212775] 0x00
[28085.212780] 0x00
[28085.212785] 0x00
[28085.212790] 0x00
[28085.212796] 0x00
[28085.212801] 0x00
[28085.212806] 0x00
[28085.212811] 0x00
[28085.212816] 0x00
[28085.212821] 0x00
[28085.212826] 0x00
[28085.212831] 0x00
[28085.212837] 0x00
[28085.212847] mp_wr_bitmap: 0xfffff000 mp_wr_ports=0x1000b len=512 curr_wr_port=0xc
[28085.212855] 0x105
[28085.212859] 0x00
[28085.212865] 0x00
[28085.212872] 0x00
[28085.212877] 0x00
[28085.212882] 0x00
[28085.212888] 0x00
[28085.212893] 0x00
[28085.212898] 0x00
[28085.212903] 0x00
[28085.212910] 0x00
[28085.212916] 0x00
[28085.212920] 0x00
[28085.212927] 0x00
[28085.212932] 0x00
[28085.212937] 0x00
[28085.212948] mp_wr_bitmap: 0xffffe000 mp_wr_ports=0x1000c len=256 curr_wr_port=0xd
[28085.212956] 0x7c
[28085.212963] 0x00
[28085.212968] 0x00
[28085.212973] 0x00
[28085.212978] 0x00
[28085.212985] 0x00
[28085.212990] 0x00
[28085.212995] 0x00
[28085.213002] 0x00
[28085.213007] 0x00
[28085.213012] 0x00
[28085.213020] 0x00
[28085.213025] 0x00
[28085.213030] 0x00
[28085.213035] 0x00
[28085.213041] 0x00
[28085.213050] mp_wr_bitmap: 0xffffc000 mp_wr_ports=0x1000d len=256 curr_wr_port=0xe
[28085.213060] 0x90
[28085.213065] 0x00
[28085.213070] 0x00
[28085.213075] 0x00
[28085.213080] 0x00
[28085.213085] 0x00
[28085.213090] 0x00
[28085.213095] 0x00
[28085.213100] 0x00
[28085.213105] 0x00
[28085.213110] 0x00
[28085.213116] 0x00
[28085.213121] 0x00
[28085.213126] 0x00
[28085.213131] 0x00
[28085.213136] 0x00
[28085.213146] mp_wr_bitmap: 0xffff8000 mp_wr_ports=0x1000e len=512 curr_wr_port=0xf
[28085.213154] 0x107
[28085.213159] 0x00
[28085.213164] 0x00
[28085.213171] 0x00
[28085.213176] 0x00
[28085.213181] 0x00
[28085.213188] 0x00
[28085.213193] 0x00
[28085.213198] 0x00
[28085.213203] 0x00
[28085.213208] 0x00
[28085.213213] 0x00
[28085.213218] 0x00
[28085.213224] 0x00
[28085.213229] 0x00
[28085.213234] 0x00
[28085.213246] mp_wr_bitmap: 0xffff0000 mp_wr_ports=0x1000f len=512 curr_wr_port=0x10
[28085.213254] 0x107
[28085.213258] 0x00
[28085.213265] 0x00
[28085.213270] 0x00
[28085.213275] 0x00
[28085.213281] 0x00
[28085.213286] 0x00
[28085.213291] 0x00
[28085.213296] 0x00
[28085.213302] 0x00
[28085.213307] 0x00
[28085.213312] 0x00
[28085.213319] 0x00
[28085.213324] 0x00
[28085.213329] 0x00
[28085.213336] 0x00
[28085.213345] mp_wr_bitmap: 0xfffe0000 mp_wr_ports=0x10010 len=512 curr_wr_port=0x11
[28085.213353] 0x107
[28085.213358] 0x00
[28085.213363] 0x00
[28085.213368] 0x00
[28085.213376] 0x00
[28085.213381] 0x00
[28085.213386] 0x00
[28085.213393] 0x00
[28085.213398] 0x00
[28085.213403] 0x00
[28085.213408] 0x00
[28085.213413] 0x00
[28085.213418] 0x00
[28085.213423] 0x00
[28085.213428] 0x00
[28085.213433] 0x00
[28085.213442] mp_wr_bitmap: 0xfffc0000 mp_wr_ports=0x10011 len=256 curr_wr_port=0x12
[28085.213450] 0xfb
[28085.213455] 0x00
[28085.213460] 0x00
[28085.213465] 0x00
[28085.213471] 0x00
[28085.213476] 0x00
[28085.213481] 0x00
[28085.213486] 0x00
[28085.213491] 0x00
[28085.213496] 0x00
[28085.213501] 0x00
[28085.213506] 0x00
[28085.213512] 0x00
[28085.213517] 0x00
[28085.213524] 0x00
[28085.213529] 0x00
[28085.213538] mp_wr_bitmap: 0xfff80000 mp_wr_ports=0x10012 len=256 curr_wr_port=0x13
[28085.213546] 0xfb
[28085.213551] 0x00
[28085.213556] 0x00
[28085.213563] 0x00
[28085.213568] 0x00
[28085.213573] 0x00
[28085.213581] 0x00
[28085.213586] 0x00
[28085.213590] 0x00
[28085.213598] 0x00
[28085.213603] 0x00
[28085.213607] 0x00
[28085.213613] 0x00
[28085.213618] 0x00
[28085.213623] 0x00
[28085.213628] 0x00
[28085.213640] mp_wr_bitmap: 0xfff00000 mp_wr_ports=0x10013 len=256 curr_wr_port=0x14
[28085.213648] 0x68
[28085.213652] 0x00
[28085.213658] 0x00
[28085.213663] 0x00
[28085.213667] 0x00
[28085.213674] 0x00
[28085.213679] 0x00
[28085.213684] 0x00
[28085.213690] 0x00
[28085.213695] 0x00
[28085.213700] 0x00
[28085.213705] 0x00
[28085.213710] 0x00
[28085.213715] 0x00
[28085.213721] 0x00
[28085.213727] 0x00
[28085.213737] bss_index = 0, tx_pkts_queued = 0 tx_pause
[28085.213747] bss_index = 1, tx_pkts_queued = 0 tx_pause
[28085.213754] bss_index = 2, tx_pkts_queued = 0 tx_pause
[28085.213762] bss_index = 3, tx_pkts_queued = 0 tx_pause
[28085.213775] Driver version = SD8997----16.92.21.p84.4-MM6X16437.p3-GPL-(FP92)
[28085.221029] main_state = 4
[28085.223739] ioctl_pending = 1
[28085.226707] tx_pending = 0
[28085.229435] wmm_tx_pending[0] = 0
[28085.232758] wmm_tx_pending[1] = 0
[28085.236084] wmm_tx_pending[2] = 0
[28085.239402] wmm_tx_pending[3] = 0
[28085.242760] rx_pending = 0
[28085.245474] lock_count = 81
[28085.248269] malloc_count = 54
[28085.251237] mbufalloc_count = 0
[28085.254386] hs_skip_count = 0
[28085.257353] hs_force_count = 0
[28085.260407] Media state = "Disconnected"
[28085.264345] carrier off
[28085.266807] tx queue 0: stopped
[28085.269954] tx queue 1: stopped
[28085.273096] tx queue 2: stopped
[28085.276248] tx queue 3: stopped
[28085.279393] mlan0: num_tx_timeout = 0
[28085.283075] uap0: num_tx_timeout = 0
[28085.286653] wfd0: num_tx_timeout = 0
[28085.290239] Start to process hanging
[28085.293830] -------- Dump info End---------
[28085.293841] FW in debug mode (0x5a)
[28085.293978] Cancel all pending cmd and txrx queue
[28085.297353] IOCTL failed: 0000000046cc56a3 id=0x10000, sub_id=0x10003 action=1, status_code=0x80000007
[28085.297512] SDIO Func0 (0x0-0x9): 43 03 02 02 03 00 00 02 03 00
[28085.312835] SDIO Func1 (0x10-0x17): 00 00 00 00 00 00 e0 ff
[28085.319281] SDIO Func1: (0x8) c3 (0x58) 00 (0x5c) 08 (0x5d) 00 (0x60) 07 (0x61) 0c (0x62) 00 (0x64) 10 (0x65) 00 (0x66) 00 (0x68) 00 (0x69) 00 (0x6a) 00
[28085.333588] SDIO Func1 (0xe8-0xff): dc fe ca 5a c7 02 00 61 24 14 70 c8 40 03 80 00 00 00 00 00 00 00 00 00
[28085.445712] SDIO Func1 (0xe8-0xff): dc fe ca 5a c7 02 00 61 24 14 70 c8 40 03 80 00 00 00 00 00 00 00 00 00
[28085.455610] WIFI auto_fw_reload: fw_reload=1
[28085.459921] ========START IN-BAND RESET===========
[28085.464794] woal_reset_intf: get bss info failed
[28085.469537] Cancel all pending cmd and txrx queue
[28085.920633] FW wake up
[28085.924117] SDIO Trigger FW In-band Reset success
[28085.924138] SDIO: max_segs=128 max_seg_size=65535
[28085.933645] rx_work=1 cpu_num=4
[28085.936822] Enable moal_recv_amsdu_packet
[28085.936855] Attach mlan adapter operations.card_type is 0x104.
[28085.937183] wlan: Enable TX SG mode
[28085.937192] wlan: Enable RX SG mode
[28085.937202] Request firmware: nxp/sdiouart8997_combo_v4.bin
[28086.245212] wlan_sdio_poll_card_status failed, tries = 10000, cs = 0xc
[28086.245230] WLAN: FW download with helper poll status timeout @ 16
[28086.245235] wlan_dnld_fw fail ret=0xffffffff
[28086.245239] wlan_dnld_fw fail ret=0xffffffff
[28086.245242] WLAN: Fail download FW with nowwait: 0
[28086.250279] SDIO Func0 (0x0-0x9): 43 03 02 02 03 00 00 02 03 00
[28086.256486] SDIO Func1 (0x10-0x17): 00 00 00 00 00 00 00 00
[28086.262376] SDIO Func1: (0x8) 00 (0x58) 00 (0x5c) 0c (0x5d) 00 (0x60) 01 (0x61) 00 (0x62) 00 (0x64) 11 (0x65) 00 (0x66) 00 (0x68) 00 (0x69) 00 (0x6a) 00
[28086.276548] SDIO Func1 (0xe8-0xff): 00 00 00 00 00 00 00 61 24 14 70 c8 40 03 80 00 10 00 00 00 00 00 00 00
[28086.388665] SDIO Func1 (0xe8-0xff): 00 00 00 00 00 00 00 61 24 14 70 c8 40 03 80 00 10 00 00 00 00 00 00 00
[28086.398561] woal_request_fw failed
[28086.401984] Firmware Init Failed
[28086.405386] SDIO Func0 (0x0-0x9): 43 03 02 02 03 00 00 02 03 00
[28086.411521] SDIO Func1 (0x10-0x17): 00 00 00 00 00 00 00 00
[28086.417362] SDIO Func1: (0x8) 00 (0x58) 00 (0x5c) 0c (0x5d) 00 (0x60) 01 (0x61) 00 (0x62) 00 (0x64) 11 (0x65) 00 (0x66) 00 (0x68) 00 (0x69) 00 (0x6a) 00
[28086.431430] SDIO Func1 (0xe8-0xff): 00 00 00 00 00 00 00 61 24 14 70 c8 40 03 80 00 10 00 00 00 00 00 00 00
[28086.543580] SDIO Func1 (0xe8-0xff): 00 00 00 00 00 00 00 61 24 14 70 c8 40 03 80 00 10 00 00 00 00 00 00 00
[28086.640283] Free module params
[28086.643357] ========END IN-BAND RESET===========
Here’s the dmesg around reloading the driver
[146877.293210] audit: type=1327 audit(1729757467.020:70): proctitle=737368643A20726F6F74205B707269765D
[154897.716563] audit: type=1006 audit(1729765487.303:71): pid=45684 uid=0 old-auid=4294967295 auid=0 tty=(none) old-ses=4294967295 ses=4 res=1
[154897.716587] audit: type=1300 audit(1729765487.303:71): arch=c00000b7 syscall=64 success=yes exit=1 a0=7 a1=ffffcb7edf60 a2=1 a3=ffffa6162b20 items=0 ppid=1 pid=45684 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4 comm="sshd" exe="/usr/sbin/sshd" key=(null)
[154897.716597] audit: type=1327 audit(1729765487.303:71): proctitle=737368643A20726F6F74205B707269765D
[154919.750547] wlan: Unloading MWLAN driver
[154919.754826] wlan: Driver unloaded
[154939.787381] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[154939.795510] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[154939.813163] wlan: Loading MWLAN driver
[154939.817416] wlan: Register to Bus Driver...
[154939.822101] vendor=0x02DF device=0x9141 class=0 function=1
[154939.827889] Attach moal handle ops, card interface type: 0x104
[154939.834009] rps set to 0 from module param
[154939.838451] SD8997: init module param from usr cfg
[154939.843487] card_type: SD8997, config block: 0
[154939.848103] cfg80211_wext=0xf
[154939.851166] max_vir_bss=1
[154939.853931] cal_data_cfg=none
[154939.857031] ps_mode = 1
[154939.859586] auto_ds = 1
[154939.862124] host_mlme=enable
[154939.865110] fw_name=nxp/sdiouart8997_combo_v4.bin
[154939.869960] SDIO: max_segs=128 max_seg_size=65535
[154939.874786] rx_work=1 cpu_num=4
[154939.878035] Enable moal_recv_amsdu_packet
[154939.878060] Attach mlan adapter operations.card_type is 0x104.
[154939.878448] wlan: Enable TX SG mode
[154939.878459] wlan: Enable RX SG mode
[154939.883513] Request firmware: nxp/sdiouart8997_combo_v4.bin
[154940.225223] wlan_sdio_poll_card_status failed, tries = 10000, cs = 0xc
[154940.225240] WLAN: FW download with helper poll status timeout @ 0
[154940.225245] wlan_dnld_fw fail ret=0xffffffff
[154940.225248] wlan_dnld_fw fail ret=0xffffffff
[154940.225252] WLAN: Fail download FW with nowwait: 0
[154940.230546] SDIO Func0 (0x0-0x9): 43 03 02 02 03 00 08 02 03 00
[154940.237436] SDIO Func1 (0x10-0x17): 00 00 00 00 00 00 00 00
[154940.243458] SDIO Func1: (0x8) 00 (0x58) 00 (0x5c) 0c (0x5d) 00 (0x60) 01 (0x61) 00 (0x62) 00 (0x64) 11 (0x65) 03 (0x66) 00 (0x68) 00 (0x69) 00 (0x6a) 00
[154940.257626] SDIO Func1 (0xe8-0xff): 00 00 00 00 00 00 00 61 24 14 70 c8 40 03 80 00 10 00 00 00 00 00 00 00
[154940.369896] SDIO Func1 (0xe8-0xff): 00 00 00 00 00 00 00 61 24 14 70 c8 40 03 80 00 10 00 00 00 00 00 00 00
[154940.379884] woal_request_fw failed
[154940.383378] Firmware Init Failed
[154940.387204] Free module params
[154940.390407] woal_add_card failed
[154940.393806] wlan_sdio: probe of mmc0:0001:1 failed with error -1
[154940.393901] wlan: Register to Bus Driver Done
[154940.398397] wlan: Driver loaded successfully
dmesg will be included in a comment (32k characted limit)