Strange Wi-Fi behavior on Colibri iMX6ULL when Wi-Fi-network is not available

Dear Toradex community,
I have a strange behavior with Wi-Fi on a Colibri iMX6ULL 512MB WB IT.

I am getting the following timeout error:

Selected interface 'mlan0'[ 6924.670946] ieee80211 phy0: sched scan stop!

'SCAN' command timed out.
root@Checkbox20-00001:~# [ 6934.890375] mwifiex_sdio mmc1:0001:1: mwifiex_cmd_timeout_func: Timeout cmd id = 0x6b, act = 0x1
[ 6934.899317] mwifiex_sdio mmc1:0001:1: num_data_h2c_failure = 0
[ 6934.905210] mwifiex_sdio mmc1:0001:1: num_cmd_h2c_failure = 0
[ 6934.911010] mwifiex_sdio mmc1:0001:1: is_cmd_timedout = 1
[ 6934.916458] mwifiex_sdio mmc1:0001:1: num_tx_timeout = 0
[ 6934.921820] mwifiex_sdio mmc1:0001:1: last_cmd_index = 1
[ 6934.927186] mwifiex_sdio mmc1:0001:1: last_cmd_id: 0c 01 6b 00 af 00 10 00 28 00
[ 6934.934640] mwifiex_sdio mmc1:0001:1: last_cmd_act: 01 00 01 00 00 00 01 00 13 00
[ 6934.942173] mwifiex_sdio mmc1:0001:1: last_cmd_resp_index = 0
[ 6934.947973] mwifiex_sdio mmc1:0001:1: last_cmd_resp_id: 0c 81 b2 80 af 80 10 80 28 80
[ 6934.955853] mwifiex_sdio mmc1:0001:1: last_event_index = 4
[ 6934.961389] mwifiex_sdio mmc1:0001:1: last_event: 58 00 76 00 0b 00 0a 00 65 00
[ 6934.968753] mwifiex_sdio mmc1:0001:1: data_sent=0 cmd_sent=0
[ 6934.974465] mwifiex_sdio mmc1:0001:1: ps_mode=1 ps_state=0
[ 6934.992590] mwifiex_sdio mmc1:0001:1: PREP_CMD: FW is in bad state
[ 6934.998865] mwifiex_sdio mmc1:0001:1: Ignore scan. Card removed or firmware in bad state
[ 6935.007244] mwifiex_sdio mmc1:0001:1: scan failed: -14
[ 6935.016291] mwifiex_sdio mmc1:0001:1: ===mwifiex driverinfo dump start===
[ 6935.023303] mwifiex_sdio mmc1:0001:1: info: MWIFIEX VERSION: mwifiex 1.0 (16.68.1.p197)
[ 6935.031536] mwifiex_sdio mmc1:0001:1: SDIO register dump start
[ 6935.052365] mwifiex_sdio mmc1:0001:1: SDIO Func0 (0x0-0x9): 43 03 06 06 07 02 00 02 03 00
[ 6935.075065] mwifiex_sdio mmc1:0001:1: SDIO Func1 (0x10-0x17): 00 00 00 00 ff ff ff ff
[ 6935.091621] mwifiex_sdio mmc1:0001:1: SDIO Func1: (0x8) c3 (0x58) 00 (0x5c) 48 (0x5d) 00 (0x60) 07 (0x61) 0c (0x62) 00 (0x64) 10 (0x65) 00 (0x66) 00 (0x68) 00 (0x69) 00 (0x6a) 00
[ 6935.131047] mwifiex_sdio mmc1:0001:1: SDIO Func1 (0xe8-0xf2): dc fe 59 00 0e 00 00 a7 a7 01 70
[ 6935.248727] mwifiex_sdio mmc1:0001:1: SDIO Func1 (0xe8-0xf2): dc fe 66 00 19 00 00 a7 a7 01 70
[ 6935.257546] mwifiex_sdio mmc1:0001:1: SDIO register dump end
[ 6935.268555] mwifiex_sdio mmc1:0001:1: ===mwifiex driverinfo dump end===
[ 6935.278268] mwifiex_sdio mmc1:0001:1: == mwifiex firmware dump start ==
[ 6935.313857] mwifiex_sdio mmc1:0001:1: Fail to pull ctrl_data
[ 6935.319552] mwifiex_sdio mmc1:0001:1: firmware dump failed
[ 6935.330795] mwifiex_sdio mmc1:0001:1: == mwifiex dump information to /sys/class/devcoredump start
[ 6935.353814] mwifiex_sdio mmc1:0001:1: == mwifiex dump information to /sys/class/devcoredump end
[ 6935.362808] mwifiex_sdio mmc1:0001:1: PREP_CMD: FW is in bad state
[ 6935.369783] mwifiex_sdio mmc1:0001:1: info: shutdown mwifiex...
[ 6935.393153] mwifiex_sdio mmc1:0001:1: PREP_CMD: card is removed
[ 6935.399116] mwifiex_sdio mmc1:0001:1: PREP_CMD: card is removed
[ 6935.451598] mwifiex_sdio mmc1:0001:1: PREP_CMD: card is removed
[ 6935.552642] mmc1: card 0001 removed
[ 6935.712069] mmc1: new high speed SDIO card at address 0001
[ 6935.756962] Bluetooth: vendor=0x2df, device=0x9142, class=255, fn=2
[ 6936.785905] mwifiex_sdio mmc1:0001:1: info: FW download over, size 623240 bytes
[ 6937.630673] mwifiex_sdio mmc1:0001:1: WLAN FW is active
[ 6937.638402] btmrvl_sdio mmc1:0001:2: fail to parse irq_bt from device tree
[ 6937.691130] mwifiex_sdio mmc1:0001:1: Unknown api_id: 3
[ 6937.696380] mwifiex_sdio mmc1:0001:1: Unknown api_id: 4
[ 6937.701741] mwifiex_sdio mmc1:0001:1: Unknown GET_HW_SPEC TLV type: 0x217
[ 6937.821807] mwifiex_sdio mmc1:0001:1: info: MWIFIEX VERSION: mwifiex 1.0 (16.68.1.p197)
[ 6937.829920] mwifiex_sdio mmc1:0001:1: driver_version = mwifiex 1.0 (16.68.1.p197)

This error happens if the configured wifi-network is unavailable, the access point mode is disabled, and another command (like a wpa_cli scan or suspend command) is issued.

To reproduce this behavior:

  • be sure that the access mode is not active , or even the hostapd-service is not running
hostapd_cli disable
Selected interface 'uap0'
OK
  • create a hotspot on your smartphone
  • connect to this hotspot
wpa_cli add_network
wpa_cli set_network 0 ssid '"Your-hotspot-ssid"'
wpa_cli set_network 0 psk '"Your-hotspot-passphrase"'
wpa_cli enable_network 0
wpa_cli save_config
  • disable the hotspot on your smartphone
  • after few seconds, if the wpa_state is SCANNING, perform another wpa_cli scan
  • → the timeout occurrs

It seems that mwifiex blocks during the SCANNING stage so that the “sched scan stop!” times out. It doesn’t matter which command is used; every command that terminates the actual sched scan results in this timeout-error. Strangely enough, this error does not occur when the access point mode (hostapd) is active

My environment:

  • Viola Plus carrier board
  • Yocto-build as follows
Build Configuration:
BB_VERSION           = "1.46.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "arm-tdx-linux-gnueabi"
MACHINE              = "colibri-imx6ull"
DISTRO               = "tdx-none"
DISTRO_VERSION       = "5.7.0-devel-20220809070425+build.0"
TUNE_FEATURES        = "arm armv7a vfp thumb neon callconvention-hard"
TARGET_FPU           = "hard"
meta-toradex-nxp     = "HEAD:ee63c90fde9fde0229bff9ac1c5cffe356fc4f41"
meta-freescale       = "HEAD:c4347b507c2ba0d7eb3917303f83631244bf5048"
meta-freescale-3rdparty = "HEAD:88773cbb6367496b1a2ddca683afcd352be3cd58"
meta-toradex-tegra   = "HEAD:f5753af4a5b9d33f0f474b320a74c2e29a66ec39"
meta-toradex-bsp-common = "HEAD:029a663150449a5e71b84dd4000476754d525c8c"
meta-oe
meta-filesystems
meta-initramfs
meta-networking
meta-multimedia
meta-python
meta-webserver       = "HEAD:52cee67833d1975a5bd52e4556c4cd312425a017"
meta-freescale-distro = "HEAD:5d882cdf079b3bde0bd9869ce3ca3db411acbf3b"
meta-checkbox        = "HEAD:770ccb49ac3d2a335a9e60da841f732c4885d566"
meta-qt5             = "HEAD:5ef3a0ffd3324937252790266e2b2e64d33ef34f"
meta-checkbox-distro = "HEAD:dddaf186eaa33ff6824018b00824f3e7724cb70c"
meta-poky            = "HEAD:57d6803aaf475552a827d322d90d1f07ba73a97d"
meta                 = "HEAD:3f40d5f095ceb099b604750db96058df00fcd49e"
meta-rauc            = "HEAD:2fe17af894b82cd2b0d0203478d4924fa916b440"

  • wpa_supplicant.conf
cat /etc/wpa_supplicant.conf
ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=0
update_config=1
config_methods=push_button
bgscan=""

network={
        ssid="EMPhone"
        psk="da4473sf339dr3"
}
  • wpa_supplicant.service
cat /lib/systemd/system/wpa_supplicant.service
[Unit]
Description=WPA supplicant
Requires=sys-subsystem-net-devices-mlan0.device
After=sys-subsystem-net-devices-mlan0.device
Before=network.target
Wants=network.target

[Service]
Type=dbus
BusName=fi.w1.wpa_supplicant1
ExecStart=/usr/sbin/wpa_supplicant -c /etc/wpa_supplicant.conf -i mlan0 -u

[Install]
WantedBy=multi-user.target
Alias=dbus-fi.w1.wpa_supplicant1.service

Does anyone have an idea how to solve this problem?

Hi @solveEM,

Thanks for reaching out. Can you please share with us some other information about this problem? We will soon reply to you on the other threads but this information may help us to solve them.

  • Which Exact Version of the Colibri iMX6ULL are you using?
  • Which Carrier Board and version are you using?
  • Are you using two wifi antennas?
  • Did you do any customization to the image? It seems that you’re using

but it seems to not be a default name.

Best regards,
Guilherme

Hi Guilherme,

Thanks for your prompt reply. Regarding your questions:

  • The Version of the Colibri iMX6ULL: V1.1A

  • The Carrier Board: Viola Plus V1.2.B

  • Yes, I use two wifi antennas

  • The image is based on tdx-reference-minimal-image but without wayland and x11, we use qt to drive the display.

  • No, I didn’t try an uncustomized image yet.

  • Well, first I’ll try the nxp-wifi-driver. Thanks for the tip

I’ll get back to you these days with the info, whether I had success or not.

Thanks and best regards,

Manfred

Hi @solveEM ,

Thanks for the update. I’d like just to suggest you test the uncustomized image first and then go to the proprietary driver.

As you created a custom image you may have arrived at an issue with the image customization and therefore changing the driver directly may not be the solution as there could be a package missing or something else. If this problem is also present on our standard images, then you should test the driver.

Please tell us about the results of your tests.

Best regards,

Hi @gclaudino.tx,

now I tried an uncustomized image (Toradex Embedded Linux Reference Minimal Image 5.7.0+build.20 (2022-07-25)
→ the same behavior, you should be able to reproduce it on your site, following the following steps.

  • install the image
  • Replace in /lib/systemd/system/wpa_supplicant.service the ExecStart…-line with
ExecStart=/usr/sbin/wpa_supplicant -c /etc/wpa_supplicant.conf -i mlan0 -u
  • execute following commands
connmanctl enable wifi
systemctl daemon-reload
systemctl restart wpa_supplicant

→ it’s already one network defined, thus the wpa_state is SCANNING

cat /etc/wpa_supplicant.conf
ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=0
update_config=1
bgscan=""

network={
        key_mgmt=NONE
}


wpa_cli status
Selected interface 'mlan0'
wpa_state=SCANNING
address=ec:2e:98:86:7b:39
uuid=3bb88559-1901-56ea-8740-64367839139a

  • Performing the following command should result in the timeout-error
wpa_cli scan

(maybe you need to try it more times until it occurs)

Now, I’ll try the nxp-driver. You’ll hear from me …

Hi @solveEM, how are you?

Have you been able to test the NXP Drivers? How is this issue going?

Best regards,

Hi @gclaudino.tx, Currently I am in the process of getting all things regarding the NXP-NDA done to be able to download the drivers . I will get back to you as soon as I have them. Should be in the next few days :grimacing:.

Hi @solveEM, how are you? Are you still facing your issues? Have you been able to solve them with the NXP Driver?

Best regards,

Hi @gclaudino.tx,
thanks for asking. Unfortunately, I did not get the driver :sob:. Initially, it took quite a long time until an NDA was agreed upon. Then I did not get the driver after all. My company is apparently too small. The driver is not provided for the “broader market” :roll_eyes:.
I have spent the last few days trying all available versions of the community driver and concluded that your provided driver, 16.68.1.p197 is the best. This driver crashes the same way but restarts and is functional.
Now I am in the process of setting up sophisticated exception management that will hide these wifi-driver-crashes to the upper application layers.
I have noticed that terminating wpa_supplicant before standby or power-down massively reduces the probability of a crash occurring. However, all those are workarounds only. Did you manage to reproduce these crashes on your site? Do you have any other suggestions for me?

Hi @solveEM,

Sorry for the delay.

Unfortunately, I’ve not been able to reproduce the exact same errors you’re facing and I’m sorry that you couldn’t receive the proprietary driver. I may have missed one or another step but I’ll try this once more this week.

Did you do any progress on the exception management?

Also, did these workarounds also improve the topics you presented here?

Best regards,

Hi @gclaudino.tx
Well, yes, I made some improvements regarding these issues. I’ll describe it as soon as my client has successfully tested it.

Hello @solveEM ,
Do you have any updates on this topic?

Best regards,
Josep

Hello,

I’m experiencing similar behaviour while scanning for networks using wpa_supplicant using same SOM, BSP 5.7.0 with mwifiex (both stock and NXP firmware).

[38387.604327] ieee80211 phy769: sched_scan start : n_ssids=1 n_match_sets=0
[38387.611344] ieee80211 phy769: n_channels=38 interval=10 ie_len=0
[38389.815003] ieee80211 phy769: sched scan stop!
[38399.833176] mwifiex_sdio mmc1:0001:1: mwifiex_cmd_timeout_func: Timeout cmd id = 0x6b, act = 0x1
[38399.842232] mwifiex_sdio mmc1:0001:1: num_data_h2c_failure = 0
[38399.848172] mwifiex_sdio mmc1:0001:1: num_cmd_h2c_failure = 0
[38399.854012] mwifiex_sdio mmc1:0001:1: is_cmd_timedout = 1
[38399.859560] mwifiex_sdio mmc1:0001:1: num_tx_timeout = 0
[38399.864972] mwifiex_sdio mmc1:0001:1: last_cmd_index = 4
[38399.870376] mwifiex_sdio mmc1:0001:1: last_cmd_id: 07 01 07 01 07 01 6b 00 6b 00
[38399.877887] mwifiex_sdio mmc1:0001:1: last_cmd_act: 00 00 00 00 00 00 01 00 01 00
[38399.885478] mwifiex_sdio mmc1:0001:1: last_cmd_resp_index = 3
[38399.891319] mwifiex_sdio mmc1:0001:1: last_cmd_resp_id: 07 81 07 81 07 81 6b 80 07 81
[38399.899256] mwifiex_sdio mmc1:0001:1: last_event_index = 2
[38399.904830] mwifiex_sdio mmc1:0001:1: last_event: 0b 00 0a 00 65 00 58 00 58 00
[38399.912247] mwifiex_sdio mmc1:0001:1: data_sent=0 cmd_sent=1
[38399.917996] mwifiex_sdio mmc1:0001:1: ps_mode=1 ps_state=0
[38399.946738] mwifiex_sdio mmc1:0001:1: Ignore scan. Card removed or firmware in bad state
[38399.955458] mwifiex_sdio mmc1:0001:1: scan failed: -14
[38399.962302] mwifiex_sdio mmc1:0001:1: ===mwifiex driverinfo dump start===
[38399.969291] mwifiex_sdio mmc1:0001:1: info: MWIFIEX VERSION: mwifiex 1.0 (16.68.10.p162)
[38399.977590] mwifiex_sdio mmc1:0001:1: SDIO register dump start
[38400.038845] mwifiex_sdio mmc1:0001:1: Ignore scan. Card removed or firmware in bad state
[38400.047140] mwifiex_sdio mmc1:0001:1: scan failed: -14
[38400.054509] mwifiex_sdio mmc1:0001:1: SDIO Func0 (0x0-0x9): 43 03 06 06 07 02 00 02 03 00
[38400.080112] mwifiex_sdio mmc1:0001:1: SDIO Func1 (0x10-0x17): 00 00 00 00 ff ff ff ff
[38400.103786] mwifiex_sdio mmc1:0001:1: SDIO Func1: (0x8) c3 (0x58) 00 (0x5c) 48 (0x5d) 00 (0x60) 07 (0x61) 0c (0x62) 00 (0x64) 10 (0x65) 00 (0x66) 00 (0x68) 00 (0x69) 00 (0x6a) 00
[38400.143690] mwifiex_sdio mmc1:0001:1: SDIO Func1 (0xe8-0xf2): dc fe 91 00 f7 00 3f 96 96 01 70
[38400.266082] mwifiex_sdio mmc1:0001:1: SDIO Func1 (0xe8-0xf2): dc fe 9d 00 03 00 3f 96 96 01 70
[38400.274933] mwifiex_sdio mmc1:0001:1: SDIO register dump end
[38400.280918] mwifiex_sdio mmc1:0001:1: ===mwifiex driverinfo dump end===
[38400.296185] mwifiex_sdio mmc1:0001:1: == mwifiex firmware dump start ==
[38400.331680] mwifiex_sdio mmc1:0001:1: Fail to pull ctrl_data
[38400.337485] mwifiex_sdio mmc1:0001:1: firmware dump failed
[38400.351903] mwifiex_sdio mmc1:0001:1: == mwifiex dump information to /sys/class/devcoredump start
[38400.369591] mwifiex_sdio mmc1:0001:1: == mwifiex dump information to /sys/class/devcoredump end
[38400.391280] mwifiex_sdio mmc1:0001:1: PREP_CMD: FW is in bad state
[38400.399338] mwifiex_sdio mmc1:0001:1: info: shutdown mwifiex...
[38400.415549] mwifiex_sdio mmc1:0001:1: PREP_CMD: card is removed
[38400.421533] mwifiex_sdio mmc1:0001:1: PREP_CMD: card is removed
[38400.474128] mwifiex_sdio mmc1:0001:1: PREP_CMD: card is removed
[38400.548181] mmc1: card 0001 removed
[38400.698705] mmc1: new high speed SDIO card at address 0001
[38400.746249] Bluetooth: vendor=0x2df, device=0x9142, class=255, fn=2
[38401.897390] mwifiex_sdio mmc1:0001:1: info: FW download over, size 630092 bytes
[38402.743580] mwifiex_sdio mmc1:0001:1: WLAN FW is active
[38402.751636] btmrvl_sdio mmc1:0001:2: sdio device tree data not available
[38402.804134] mwifiex_sdio mmc1:0001:1: Unknown api_id: 3
[38402.809401] mwifiex_sdio mmc1:0001:1: Unknown api_id: 4
[38402.814751] mwifiex_sdio mmc1:0001:1: Unknown GET_HW_SPEC TLV type: 0x217
[38402.913342] mwifiex_sdio mmc1:0001:1: info: MWIFIEX VERSION: mwifiex 1.0 (16.68.10.p162)
[38402.921568] mwifiex_sdio mmc1:0001:1: driver_version = mwifiex 1.0 (16.68.10.p162)

Did you find anything on this issue?

BR,
Diego

Hi @d.moimas !

Does the workaround referenced above help you?

Best regards,

Hi @henrique.tx ,

we solved (at least for our case) in this way:

  • we have wpa_supplicant exposing it’s control socket and we are using it to control wifi.
  • if we have a defined network in wpa_supplicant.conf (either specific essid or generic "network without key management, so with no auth), wpa supplicant will try also to scan for this network and to autoconnect.
  • if we run a network scan from our app (thru control socket) while wpa_supplicant is already scanning, we get the previous error

So, as we don’t need any default network configured in wpa_supplicant.conf (we had one during debug), we just removed any network definition from conf file and manage everything through control socket using our software.

BTW I think there’s something wrong: if the driver/firmware cannot handle two concurrent scans, it should refuse to do it (e.g.: return some kind of busy status) and not crash

BR,
Diego

Sorry for the very late reply :grimacing:

Well, since the WLAN stack restarts itself in the event of an error, I was able to make the application layer suitably robust. I created a wifi object in my application that handles wifi driver crashes.
Also, before executing any wifi method, I check whether wpa_supplicant is successfully started.
On standby resume, where the WLAN driver crashes very often, the application restarts hostapd and wpa_supplicant.
And whenever the WLAN stack crashes due to a scan or scan abort during a scan, the wifi object guarantees error-free subsequent access.
With these steps, even if only workarounds, the whole wifi works stably.

Hello @solveEM ,

Thank you very much for the detailed explanation :slight_smile:

Best regards,
Josep