iMX8 Wifi Access point unreliable

I’m having stability issues hosting a WPA2 wifi access point on an iMX8. I’ve been using a custom image based off of BSP5.1, but have moved to testing with a Reference Minimal Image to make sure I’m not introducing any problems in my image.

I can generally get the AP to come up once and function, but after some time (hours) clients can no longer connect, the AP disappears, and I get error messages from the wifi driver in dmesg. I’ve identified the following steps that more quickly reproduce what I’m seeing, following these steps from a forum post to configure the AP:

  1. Installed a nightly reference image:

    root@apalis-imx8:~# cat /etc/issue
    TDX Wayland with XWayland 5.1.0-devel-20210110+build.185 (dunfell) \n \l
    Apalis-iMX8_Reference-Minimal-Image

  2. Configure hostapd:

    root@apalis-imx8:~# cat /etc/hostapd.conf
    interface=uap0
    ssid=test-access-point
    hw_mode=g
    channel=1
    wpa=2
    wpa_passphrase=the-password
    own_ip_addr=192.168.7.1

  3. Create a systemd-networkd network definition:

    root@apalis-imx8:~# cat /etc/systemd/network/hostapd.network
    [Match]
    Name=uap0
    [Network]
    Address=192.168.7.1/24
    DHCPServer=yes

  4. Rebooted the board

  5. Run systemctl start hostapd. No errors reported; clients can connect and the AP is working properly.

  6. To accelerate the failure, stop and start the access point: systemctl stop hostapd followed by systemctl start hostapd

  7. Observe the AP is visible to clients, but any connect attempt fails.

  8. Eventually, observe driver errors in dmesg. This can be triggered immediately by running iw dev which hangs for several seconds:

    [ 198.624550] mwifiex_pcie 0000:01:00.0: mwifiex_cmd_timeout_func: Timeout cmd id = 0x1e, act = 0x0
    [ 198.633500] mwifiex_pcie 0000:01:00.0: num_data_h2c_failure = 0
    [ 198.639480] mwifiex_pcie 0000:01:00.0: num_cmd_h2c_failure = 0
    [ 198.645356] mwifiex_pcie 0000:01:00.0: is_cmd_timedout = 1
    [ 198.650879] mwifiex_pcie 0000:01:00.0: num_tx_timeout = 0
    [ 198.656323] mwifiex_pcie 0000:01:00.0: last_cmd_index = 3
    [ 198.661764] mwifiex_pcie 0000:01:00.0: last_cmd_id: b0 00 5e 00 1e 00 1e 00 28 00
    [ 198.669298] mwifiex_pcie 0000:01:00.0: last_cmd_act: 01 00 01 00 00 00 00 00 13 00
    [ 198.676906] mwifiex_pcie 0000:01:00.0: last_cmd_resp_index = 2
    [ 198.682790] mwifiex_pcie 0000:01:00.0: last_cmd_resp_id: b0 80 5e 80 1e 80 b1 80 28 80
    [ 198.690750] mwifiex_pcie 0000:01:00.0: last_event_index = 4
    [ 198.696367] mwifiex_pcie 0000:01:00.0: last_event: 58 00 58 00 0b 00 0a 00 2e 00
    [ 198.703806] mwifiex_pcie 0000:01:00.0: data_sent=0 cmd_sent=1
    [ 198.709598] mwifiex_pcie 0000:01:00.0: ps_mode=1 ps_state=0
    [ 198.715258] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
    [ 198.716607] mwifiex_pcie 0000:01:00.0: ===mwifiex driverinfo dump start===
    [ 198.728472] mwifiex_pcie 0000:01:00.0: info: MWIFIEX VERSION: mwifiex 1.0 (16.68.1.p195)
    [ 198.737174] mwifiex_pcie 0000:01:00.0: PCIE register dump start
    [ 198.744734] mwifiex_pcie 0000:01:00.0: pcie scratch register:
    [ 198.752662] mwifiex_pcie 0000:01:00.0: reg:0xcf0, value=0xfedcba00
    [ 198.773327] mwifiex_pcie 0000:01:00.0: PCIE register dump end
    [ 198.787545] mwifiex_pcie 0000:01:00.0: ===mwifiex driverinfo dump end===
    [ 198.798881] mwifiex_pcie 0000:01:00.0: == mwifiex firmware dump start ==
    [ 238.224784] mwifiex_pcie 0000:01:00.0: == mwifiex firmware dump end ==
    [ 238.233254] mwifiex_pcie 0000:01:00.0: == mwifiex dump information to /sys/class/devcoredump start
    [ 238.242537] mwifiex_pcie 0000:01:00.0: == mwifiex dump information to /sys/class/devcoredump end
    [ 238.251558] mwifiex_pcie 0000:01:00.0: info: shutdown mwifiex…
    [ 238.258769] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
    [ 238.264893] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
    [ 238.324947] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
    [ 238.331016] mwifiex_pcie 0000:01:00.0: Failed to delete mgmt IEs!
    [ 238.337199] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
    [ 238.343528] mwifiex_pcie 0000:01:00.0: Failed to stop the BSS
    [ 238.350559] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
    [ 238.356681] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
    [ 238.362776] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
    [ 238.676634] mwifiex_pcie 0000:01:00.0: info: dnld wifi firmware from 177740 bytes
    [ 240.778911] mwifiex_pcie 0000:01:00.0: info: FW download over, size 634228 bytes
    [ 241.540562] mwifiex_pcie 0000:01:00.0: WLAN FW is active
    [ 241.585525] mwifiex_pcie 0000:01:00.0: Unknown api_id: 3
    [ 241.590892] mwifiex_pcie 0000:01:00.0: Unknown api_id: 4
    [ 241.596266] mwifiex_pcie 0000:01:00.0: Unknown GET_HW_SPEC TLV type: 0x217
    [ 241.617308] mwifiex_pcie 0000:01:00.0: info: MWIFIEX VERSION: mwifiex 1.0 (16.68.1.p195)
    [ 241.625609] mwifiex_pcie 0000:01:00.0: driver_version = mwifiex 1.0 (16.68.1.p195)

  9. hostapd service log shows no errors, but clients cannot connect to this AP until the board is restarted.

I’ve tried this in several image versions (monthly and nightly); I see it in my image based off of bsp5.1, and I see it in Torizon as well.

What is the proper method for configuring an AP with security on an iMX8?

I’ve the same and even worse situation with mwifiex_sdio driver for iMX6ULL. We are using instead USB connected card and AP mode is reliable. We would use Colibries with built in WiFi, but that mwifiex driver isn’t friendly to AP mode. Perhaps time to recheck and look for better driver, hope it’s fixable.

Hi @tyler_ ,

Are you using upstream kernel, if not then give it a try? I just rechecked situation with iMX6ULL. mwifiex AP support using downstream kernel is still broken, with upstream AP seems working quite promising. Finally, thanks Toradex!

@Edward,

Unfortunately the iMX8 SoCs are still not very well supported on upstream kernels :frowning: That’s why we’re still shipping the a downstream 5.4 kernel based on NXP’s.

Hi Gustavo,

Restarting hostapd here doesn’t resolve the issue; it remains in the same state where clients can’t connect and the driver periodically dumps error messages about command timeouts and FW is in bad state.

Restarting the board is necessary to use wifi again. Unloading the mwifiex, mwifiex_pcie, and cfg80211 modules and then reloading them before starting hostapd also does not address the issue.

Hi @gustavo.tx,

mwifiex code is practically the same in up- and downstream sources. But downstream version dies within minute from ‘connmanctl tether wifi on apname appassword’ and upstream seems being fine. Firmwares are the same. Comparing sources further I see some difference in net/wireless/chan.c . Looking further WIPHY_FLAG_DFS_OFFLOAD seems being defined in downstream’s only include/net/cfg80211.h . Could it be the reason for different driver behavior?
Looking for DFS_OFFLOAD in wifi drivers, quantenna/qtnmax seems doing some manipulations around DFS_OFFLOAD to say it is not supported. Perhaps mwifiex needs something like that.

I see drivers/net/wireless/nxp/mxm_wifiex in downstream. Did you try to switch to that driver? At least it seems supporting 8997, DFS offload etc. Driver seems having no device tree settings and has to be started using some config files. I tried following this, but something is wrong.

Regards,
Edward

@gustavo.tx , @tyler_

Well, it seems NXP WiFi driver seems working! These NXP instructions seem working, but you need with few additional steps:
UM11490.pdf

  1. First of all you need to menuconfig kernel to disable old and unsupported Marvell WiFi_Ex driver (CONFIG_MWIFIEX) or perhaps blacklist it.

  2. Enable NXP devices->NXP MxM WiFi Driver (CONFIG_MXMWIFIEX)

  3. Edit drivers/net/wireless/nxp/mxm_wifiex/wlan_src/Makefile to enable support for your flavor of former Marvel card. Here’s what I saw initially:

    ‘#’ Configuration Options

    ‘#’ Multi-chipsets
    CONFIG_SD8887=n
    CONFIG_SD8897=n
    CONFIG_USB8897=n
    CONFIG_PCIE8897=n
    CONFIG_SD8977=n
    CONFIG_SD8978=n
    CONFIG_USB8978=n
    CONFIG_SD8997=n
    CONFIG_USB8997=n
    CONFIG_PCIE8997=n
    CONFIG_SD8987=y
    CONFIG_SD9097=n
    CONFIG_USB9097=n
    CONFIG_PCIE9097=n
    CONFIG_SD9098=n
    CONFIG_USB9098=n
    CONFIG_PCIE9098=n

I said yes to iMX6ULL Colibrie’s SD8997 and instructions from pdf worked to enable driver. Driver seems performing better than old one, achieves faster speedtest.net figures, etc. AP mode is working and seems being stable.

Hope this helps.

Edward

Thanks, @Edward!

We’re also investigating using the mlan/sd8xxx/bt8xxx drivers from NXP. This info is valuable for our research.

@tyler_, can you please give this driver a try?

Hi @gustavo,

Thanks for feedback. I personally say no for old driver and yes for one from NXP! Tried pushing hostapd.conf settings to the limits, ac mode, 2xmimo, 80MHz channel. A lot of learning with hostapd, but enough time to notice crashes. Yes, 877Mbps WiFi doesn’t make sense on poor iMX6ULL, but at least all worked and didn’t crash like with old driver. I saw some errors on console, but not often and not critical ones, at least they didn’t force to reboot! For tests I was passing speedtest’s traffic from rooter to PC via Colibri, from its ethernet end to WiFi end and then from WiFi end to ethernet end. In one case connman was tether-shating connection on ethernet, in another case I used hostapd + NAT. iMX6ULL topped at 50-60Mbps in both directions, WPA2-PSK in all cases. CPU usage was up to ~70%, but that was not iMX8, whcih should be much better, especially on PCIe.
Finally these WiFi modules make more sense than before.

Edward

@Edward,

Nice! Happy that this works.

Where you able to compile this with the toradex_5.4-2.3.x-imx branch? I had some issues and I’m currently investigating.

Hi @gustavo.tx,

You are right, I was building toradex_5.4-2.1.x-imx. It had initially all PCI cards initially disabled in drivers/net/wireless/nxp/mxm_wifiex/wlan_src/Makefile, see my post above. I cloned 2.3.x, which had some PCIe cards enabled and iMX6ULL modules didn’t build until I disabled all PCIE cards with CONFIG_PCIExxx=n in that Makefile. Clearly that Makefile and Kconfig should be converted to specify NXP cards along with kernel configuration settings.

Didn’t test with 2.3.x kernel yet, but I tried to compare few days ago performance of iMX6ULL + SD8997 to the same Colibri with Realtek USB cards. With USB it is able to cross 100Mbps limit. In that test I switched to pass traffic via USB RNDIS, since it is not limited by 100Mbps. I’m not sure what actually limits iMX6ULL + SD8997, must be either SDIO or driver quality. I would like to hear what speeds you get with iMX8 + PCIe and decent 802.11n/802.11ac communication partners.

Thanks,
Edward

2.3.x works as well. Patches for confortable kernel config could be like attached. As is, not tested with other NXP cards:

Kconfig.diff Makefile.diff

I’ve been trying this driver on the iMX8. I edited the kernel config to include the NXP driver as a module, and excluded the Marvell drivers. I believe the default Makefile is correct for iMX8 (PCIE8997=y).

I’m getting two kernel modules moal and mlan and can load them, but they aren’t interacting with the card. Are there other changes I need to make to get the wifi card working with this driver?

For this experiment, I’ve moved my image to be based on the 5.2 March monthly BSP.

Edit: After power cycling the board, I see the driver try to initialize the card. It looks like I’m missing the proper firmware; I’m looking into that now.

Hi @tyler_ ,

moal driver requires config file, which further has to be specified as parameter to driver. You need to look deeper to above mentioned UM11490.pdf. For example config file could look like nano /lib/firmware/nxp/wifi_mod_para_sd8987.conf

SD8987 = {
 cfg80211_wext=0xf
 wfd_name=p2p
 cal_data_cfg=none
 max_vir_bss=1
 fw_name=nxp/sdiouart8987_combo_v0.bin
 init_hostcmd_cfg=nxp/ed_mac.bin
}

Where SD8987 is your card variant. Then moal driver should be loaded with modprobe moal mod_para=nxp/wifi_mod_para_sd8987.conf

Of course FW file name should match your card as well. Firmware that came with mwifiex driver works, just check what FW is used and edit conf file accordingly.

Thanks, @Edward . Yes it was the firmware and config that was holding me up, but I was able to get things working with firmware from NXP’s Github.

In testing yesterday, this driver seems stable through performance tests and multiple stop/starts of the access point. For performance, I’m seeing about 20Mbit/s tx and rx for G mode. I can get faster receive speeds in faster wifi modes, at the expense of slower transmit speeds through. I’m still looking into this.

Hi @tyler_ ,

I’m glad you made it working. I doubt I understand this

I can get faster receive speeds in faster wifi modes, at the expense of slower transmit speeds through

On iMX8 you should clearly get better speeds than on iMX6ULL. Do you mean simultaneous tx/rx to the same client/AP? WiFi is not very full duplex.
Which modes did you try? [HT20]/[HT40]/ short gi? 802.11n or 802.11ac?
Hostapd isn’t user friendly thing, hostapd-conf script mentioned HERE could help configuring faster modes.

Edward

@Edward , I wasn’t clear there. In my initial testing, as I tried more advanced wifi modes (G, then n, then AC) that the performance improvements were not symmetric. The iMX8 was able to download faster in the latter modes, but the upload was slower. This is of secondary concern for me and I’ll look into it later.

My primary issue is that I can’t get the driver to work on BSP 5.1.0. I’ve built it off of the 5.2 BSP (March development release) and it works well. When I build and run it in 5.1 (2020 Q4 Quarterly release) the driver loads fine, but never identifies the wireless card.

Here is an excerpt of my 5.1 dmesg when I load the module:

[    6.983380] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[    7.044534] systemd-journald[375]: Received client request to flush runtime journal.
[    7.065517] wlan: Loading MWLAN driver
[    7.071145] wlan: Driver loaded successfully
[    7.088863] openvswitch: Open vSwitch switching datapath
[    7.759465] .

In 5.2, I see the driver load, identify the card, download the firmware, etc. @gustavo.tx what BSP were you testing against while trying this driver on the iMX8?

Edit: I’ve found the driver has been updated significantly between the 2.1 and 2.3 toradex-linux branches, including enabling support for the PCIE8997 card found in the imx8, as Edward pointed out a few messages ago. I’ll patch the kernel to enable the device.

Hi @tyler_

Are you using your setup with one or two antennas?

Are you also planning to use the WiFi “Client” interface or only Access Point in your use-case?

The BSP 5.2.0 and TorizonCore 5.2.0 are now in “production release” state as of their quarterly releases. So I suggest you use them.

About your performance tests, any updates? Or are you still seeing those 20 Mbit/s?

Best regards,
André Curvello

Greetings @tyler_!

From your logs it seems that the driver properly recovered. You would need to restart hostapd then.

I think it would also be helpful to disable the non-AP interface, might help with stability.

Hi @andrecurvello.tx ,

@tyler_ clarified previously that 20Mbps in 802.11g mode, which is normal for 802.11g.

I find connman not fully compatible with cards like this one. Connman doesn’t mind that one interface is only for AP, and another one for client, which leads to various problems like scanning on both interfaces or tethering on mlan interface, which sooner or later fails. The only decent option to keep using connman with this card is config connman to ignore ap interface and not use connman wifi tethering at all.