Wifi FW crash in colibri IMX8

mag · July 21, 2022, 1:20pm

Hello community,
I’m using a Colibri with the Iris carrier board and Torizon 5.6.0 .
The following log was taken after a failure while switching the wifi through nmcli.

Here is the relevant part of the dmesg:

[ 1560.072262] mwifiex_pcie 0000:01:00.0: mwifiex_cmd_timeout_func: Timeout cmd id = 0xaf, act = 0x0
[ 1560.072276] mwifiex_pcie 0000:01:00.0: num_data_h2c_failure = 0
[ 1560.072282] mwifiex_pcie 0000:01:00.0: num_cmd_h2c_failure = 0
[ 1560.072288] mwifiex_pcie 0000:01:00.0: is_cmd_timedout = 1
[ 1560.072294] mwifiex_pcie 0000:01:00.0: num_tx_timeout = 0
[ 1560.072299] mwifiex_pcie 0000:01:00.0: last_cmd_index = 0
[ 1560.072306] mwifiex_pcie 0000:01:00.0: last_cmd_id: af 00 b0 00 b2 00 d0 00 d0 00
[ 1560.072313] mwifiex_pcie 0000:01:00.0: last_cmd_act: 00 00 01 00 00 00 00 18 00 18
[ 1560.072319] mwifiex_pcie 0000:01:00.0: last_cmd_resp_index = 4
[ 1560.072325] mwifiex_pcie 0000:01:00.0: last_cmd_resp_id: b0 80 b0 80 b2 80 d0 80 d0 80
[ 1560.072331] mwifiex_pcie 0000:01:00.0: last_event_index = 1
[ 1560.072337] mwifiex_pcie 0000:01:00.0: last_event: 76 00 0a 00 33 00 33 00 33 00
[ 1560.072343] mwifiex_pcie 0000:01:00.0: data_sent=0 cmd_sent=1
[ 1560.072349] mwifiex_pcie 0000:01:00.0: ps_mode=1 ps_state=0
[ 1560.072391] mwifiex_pcie 0000:01:00.0: Failed to reset BSS
[ 1560.072668] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.072680] mwifiex_pcie 0000:01:00.0: Failed to delete mgmt IEs!
[ 1560.072686] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.072691] mwifiex_pcie 0000:01:00.0: Failed to stop the BSS
[ 1560.072703] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.072715] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.072727] mwifiex_pcie 0000:01:00.0: uap0: changing to 2 not supported
[ 1560.072746] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.072754] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1560.072823] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.072918] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.073007] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.073014] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.073070] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.073077] mwifiex_pcie 0000:01:00.0: Failed to delete mgmt IEs!
[ 1560.073083] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.073089] mwifiex_pcie 0000:01:00.0: Failed to stop the BSS
[ 1560.073250] mwifiex_pcie 0000:01:00.0: ===mwifiex driverinfo dump start===
[ 1560.073266] mwifiex_pcie 0000:01:00.0: info: MWIFIEX VERSION: mwifiex 1.0 (16.68.10.p159) 
[ 1560.073301] mwifiex_pcie 0000:01:00.0: PCIE register dump start
[ 1560.073308] mwifiex_pcie 0000:01:00.0: pcie scratch register:
[ 1560.073322] mwifiex_pcie 0000:01:00.0: reg:0xcf0, value=0xfedcba00
               reg:0xcf8, value=0xc600ce
               reg:0xcfc, value=0x1959627

[ 1560.073329] mwifiex_pcie 0000:01:00.0: PCIE register dump end
[ 1560.073454] mwifiex_pcie 0000:01:00.0: ===mwifiex driverinfo dump end===
[ 1560.073460] mwifiex_pcie 0000:01:00.0: == mwifiex firmware dump start ==
[ 1560.080261] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.080274] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1560.080740] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.080763] mwifiex_pcie 0000:01:00.0: Failed to delete mgmt IEs!
[ 1560.080769] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.080774] mwifiex_pcie 0000:01:00.0: Failed to stop the BSS
[ 1560.080821] mwifiex_pcie 0000:01:00.0: uap0: changing to 2 not supported
[ 1560.081172] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.081182] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.081489] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.081498] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.081702] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.081711] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.081735] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.081742] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.081750] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.081756] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.081763] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.081768] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.082977] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.082989] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1560.083118] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.083125] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1560.153256] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.153269] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.161620] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.161633] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.161655] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.161660] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.161678] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.161683] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.177944] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.178047] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.178054] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.178102] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.178115] mwifiex_pcie 0000:01:00.0: Failed to delete mgmt IEs!
[ 1560.178121] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.178126] mwifiex_pcie 0000:01:00.0: Failed to stop the BSS
[ 1560.184308] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.184322] mwifiex_pcie 0000:01:00.0: Failed to delete mgmt IEs!
[ 1560.184329] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.184334] mwifiex_pcie 0000:01:00.0: Failed to stop the BSS
[ 1560.184410] mwifiex_pcie 0000:01:00.0: uap0: changing to 2 not supported
[ 1560.184561] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.184569] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.187166] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.187178] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.190930] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.190942] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.190990] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.191002] mwifiex_pcie 0000:01:00.0: Failed to delete mgmt IEs!
[ 1560.191008] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.191013] mwifiex_pcie 0000:01:00.0: Failed to stop the BSS
[ 1560.197167] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.197183] mwifiex_pcie 0000:01:00.0: set mac address failed: ret=-1
[ 1560.197670] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.197681] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.199813] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.199825] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.202723] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.207485] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.207498] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.245691] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.245704] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.254553] mwifiex_pcie 0000:01:00.0: Ignore scan. Card removed or firmware in bad state
[ 1560.254569] mwifiex_pcie 0000:01:00.0: scan failed: -14
[ 1560.265169] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.265183] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.265194] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.265199] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.265206] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.265212] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.335516] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.335530] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.335557] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.335562] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.335643] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.335649] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.394119] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.394133] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.394142] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.394147] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.394154] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.394160] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.457411] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.457425] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.457445] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.457450] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.457471] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.457476] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1560.636257] kauditd_printk_skb: 41 callbacks suppressed
[ 1563.773284] mwifiex_pcie 0000:01:00.0: Fail to pull ctrl_data
[ 1563.774877] mwifiex_pcie 0000:01:00.0: == mwifiex dump information to /sys/class/devcoredump start
[ 1563.775163] mwifiex_pcie 0000:01:00.0: == mwifiex dump information to /sys/class/devcoredump end
[ 1563.775183] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1563.775213] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1563.775278] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1563.775289] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[ 1563.775297] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1563.775304] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[ 1563.775311] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1563.775318] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[ 1563.775326] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1563.775333] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[ 1563.775340] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1563.775347] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[ 1563.775354] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1563.775358] mwifiex_pcie 0000:01:00.0: info: shutdown mwifiex...
[ 1563.775364] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[ 1563.775406] mwifiex_pcie 0000:01:00.0: rx_pending=0, tx_pending=1,   cmd_pending=0
[ 1563.794988] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1563.795009] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1563.795028] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1563.952120] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1563.952136] mwifiex_pcie 0000:01:00.0: Failed to delete mgmt IEs!
[ 1563.952144] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1563.952151] mwifiex_pcie 0000:01:00.0: Failed to stop the BSS
[ 1563.956150] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1564.331882] mwifiex_pcie 0000:01:00.0: info: dnld wifi firmware from 174528 bytes
[ 1566.463215] mwifiex_pcie 0000:01:00.0: info: FW download over, size 638992 bytes
[ 1567.181451] mwifiex_pcie 0000:01:00.0: WLAN FW is active
[ 1567.210164] mwifiex_pcie 0000:01:00.0: Unknown api_id: 3
[ 1567.210181] mwifiex_pcie 0000:01:00.0: Unknown api_id: 4
[ 1567.210190] mwifiex_pcie 0000:01:00.0: Unknown GET_HW_SPEC TLV type: 0x217
[ 1567.231325] mwifiex_pcie 0000:01:00.0: info: MWIFIEX VERSION: mwifiex 1.0 (16.68.10.p159) 
[ 1567.231339] mwifiex_pcie 0000:01:00.0: driver_version = mwifiex 1.0 (16.68.10.p159) 
[ 1567.286409] mwifiex_pcie 0000:01:00.0: uap0: changing to 2 not supported

Any ideas/suggestions?
Thanks

gclaudino.tx · July 21, 2022, 2:12pm

Hi @mag, how are you?

Could you please share with us a few more details of your use case?

From the tags, I get it’s a Colibri iMX8X. Could you please share the exact version?
Also, which version of the Iris are you using?
Are you using two antennas on your module?
Did you do any customization to the TorizonCore image?
How can we reproduce the error, i.e., what are the steps and commands necessary to do so? Does this happen every time that you follow this procedure?
Was there a nmcli error that you could see on the serial while switching networks?

Thanks a lot

mag · July 21, 2022, 3:30pm

Hi @gclaudino.tx,

Colibri IMX8QXP 2GB WB IT V1.0D
Iris Rev. 2.0
Yes, two antennas like this one

image591×1280 67.3 KB
Yes for the touchscreen (colibri-imx8x_parallel-rgb-lvds_overlay.dts + custom one for display timing)
We have a testing frameworks that communicates with our software on the board, that calls “nmcli”. My colleague in charge of the testing reported this to me and I could only get the dmesg log remotely. I never received any report on this happening before. It happened while switching off and on again the WiFi board in AP/hotspot mode
I have no other logs at my disposal

Sorry for the sparse information but it didn’t happened to me directly. I’ve already asked to be notified immediately if this happen again so I can get more in detail information.
I’ll let you know if it happens again, especially since it seems you have not received similar reports on this issue in the past. It was the first time for me seeing this too.

Thank again

gclaudino.tx · July 21, 2022, 4:27pm

Hi @mag,

Thanks for updating.

Would you please be able to check with your team if they can share with us more details on how to reproduce it or even share your codes with us so that we could try to reproduce it? You can use share.toradex.com to upload the files and send them by mail to support@toradex.com if you want to keep them private. Also, as the error happened after dealing with nmcli, it may have given some additional logs that could be useful.

I suppose this happened only once, then. Is this right?

Thanks again

mag · July 22, 2022, 11:50am

Hi @gclaudino.tx

I’ve got some more info. For sure the following commands have been issued when the crash happened:

nmcli -t radio wifi on

nmcli dev status
nmcli -f SSID,signal,security -t dev wifi list ifname mlan0
// No delay between the previous two commands and the same next two commands
nmcli dev status
nmcli -f SSID,signal,security -t dev wifi list ifname mlan0

I don’t see anything strange, maybe the timing of the requests may have an impact.

Yes, at least it’s the only time I’ve been notified about this. It never happened afterwards.
I’ll keep you informed.

Thank you again

gclaudino.tx · July 22, 2022, 3:24pm

Hi @mag,

I just got more internal information and it seems that similar behaviour has been reported before. Could you please try this workaround on the machine that presented this error and check if the issue gets solved?

Disable wifi using connman

connman disable wifi

unbind the module

cd /sys/bus/platform/drivers/sdhci-esdhc-imx
echo 30b60000.mmc >unbind

bind the module

/sys/bus/platform/drivers/sdhci-esdhc-imx# echo 30b60000.mmc >bind

enable wifi again using connman

connman enable wifi

If this does not help, we could check if you could profit of the procedure to install the NXP proprietary drivers as sometimes they have better performance for Wifi usecases.

Best regards,

mag · July 25, 2022, 6:15am

Hi @gclaudino.tx ,
I’ll definitely try it.
I’ll mark as solved for now.

Thank you again for the support
Regards

gclaudino.tx · July 25, 2022, 12:26pm

Hi @mag,

Thanks for commenting. Please test it and come back again if this does not work.

Best regards,

mag · October 3, 2022, 9:10am

Hi @gclaudino.tx ,
I had other Wifi crashes today.
The Wifi was on since boot and never switched on/off.
The Wifi network I am connected to has quite low reception.

I had 2 crashes on the Wifi after connecting the USB cable in gadget mode (OTG).
Have you ever seen something related to that?
It looks that all the network interfaces go down after the Wifi FW crash.
A reboot usually solves the issue and everything works fine most of the time.

Here is the dmesg log, if it can help:

[   26.949608] IPv6: ADDRCONF(NETDEV_CHANGE): uap0: link becomes ready
[   26.972411] mwifiex_pcie 0000:01:00.0: info: trying to associate to '4CE676160420-1' bssid 14:cc:20:7d:3e:4d
[   26.986944] mwifiex_pcie 0000:01:00.0: info: associated to bssid 14:cc:20:7d:3e:4d successfully
[   26.995181] IPv6: ADDRCONF(NETDEV_CHANGE): mlan0: link becomes ready
[   28.467213] kauditd_printk_skb: 20 callbacks suppressed
[   28.467222] audit: type=1325 audit(1664785586.690:73): table=nat family=2 entries=0
[   28.480309] audit: type=1325 audit(1664785586.700:74): table=filter family=2 entries=0
[   28.492877] audit: type=1325 audit(1664785586.720:75): table=nat family=2 entries=5
[   28.497143] audit: type=1325 audit(1664785586.720:76): table=nat family=2 entries=7
[   28.505428] audit: type=1325 audit(1664785586.730:77): table=nat family=2 entries=8
[   28.510240] audit: type=1325 audit(1664785586.730:78): table=nat family=2 entries=10
[   28.518540] audit: type=1325 audit(1664785586.740:79): table=nat family=2 entries=11
[   28.526699] audit: type=1325 audit(1664785586.750:80): table=nat family=2 entries=12
[   28.533947] audit: type=1325 audit(1664785586.760:81): table=nat family=2 entries=13
[   28.540600] audit: type=1325 audit(1664785586.760:82): table=nat family=2 entries=14
[   28.646875] eth0: renamed from vethc9ee8b4
[   28.692854] IPv6: ADDRCONF(NETDEV_CHANGE): veth2a32c75: link becomes ready
[   28.693110] br-b2a177f88f99: port 1(veth2a32c75) entered blocking state
[   28.693124] br-b2a177f88f99: port 1(veth2a32c75) entered forwarding state
[   28.693410] IPv6: ADDRCONF(NETDEV_CHANGE): br-b2a177f88f99: link becomes ready
[   32.471319] 3.3V: disabling
[   41.263752] usb 1-1.2.2.2: USB disconnect, device number 6
[   43.071305] usb 1-1.2.2.2: new high-speed USB device number 8 using cdns-usb3
[   49.163720] mwifiex_pcie 0000:01:00.0: uap0: changing to 2 not supported
[   49.169365] mwifiex_pcie 0000:01:00.0: uap0: changing to 2 not supported
[   49.279189] mwifiex_pcie 0000:01:00.0: uap0: changing to 2 not supported
[   60.001254] mwifiex_pcie 0000:01:00.0: mwifiex_cmd_timeout_func: Timeout cmd id = 0x107, act = 0x0
[   60.001268] mwifiex_pcie 0000:01:00.0: num_data_h2c_failure = 0
[   60.001277] mwifiex_pcie 0000:01:00.0: num_cmd_h2c_failure = 0
[   60.001285] mwifiex_pcie 0000:01:00.0: is_cmd_timedout = 1
[   60.001293] mwifiex_pcie 0000:01:00.0: num_tx_timeout = 0
[   60.001300] mwifiex_pcie 0000:01:00.0: last_cmd_index = 4
[   60.001309] mwifiex_pcie 0000:01:00.0: last_cmd_id: 10 00 28 00 10 00 28 00 07 01
[   60.001318] mwifiex_pcie 0000:01:00.0: last_cmd_act: 01 00 13 00 01 00 13 00 00 00
[   60.001326] mwifiex_pcie 0000:01:00.0: last_cmd_resp_index = 3
[   60.001334] mwifiex_pcie 0000:01:00.0: last_cmd_resp_id: 10 80 28 80 10 80 28 80 28 80
[   60.001342] mwifiex_pcie 0000:01:00.0: last_event_index = 3
[   60.001350] mwifiex_pcie 0000:01:00.0: last_event: 42 00 76 00 42 00 58 00 33 00
[   60.001358] mwifiex_pcie 0000:01:00.0: data_sent=0 cmd_sent=1
[   60.001366] mwifiex_pcie 0000:01:00.0: ps_mode=1 ps_state=0
[   60.002443] mwifiex_pcie 0000:01:00.0: ===mwifiex driverinfo dump start===
[   60.002463] mwifiex_pcie 0000:01:00.0: info: MWIFIEX VERSION: mwifiex 1.0 (16.68.10.p159) 
[   60.002511] mwifiex_pcie 0000:01:00.0: PCIE register dump start
[   60.002522] mwifiex_pcie 0000:01:00.0: pcie scratch register:
[   60.002544] mwifiex_pcie 0000:01:00.0: reg:0xcf0, value=0xfedcba00
               reg:0xcf8, value=0xb70041
               reg:0xcfc, value=0x17f8007

[   60.002553] mwifiex_pcie 0000:01:00.0: PCIE register dump end
[   60.002719] mwifiex_pcie 0000:01:00.0: ===mwifiex driverinfo dump end===
[   60.002731] mwifiex_pcie 0000:01:00.0: == mwifiex firmware dump start ==
[   64.071266] mwifiex_pcie 0000:01:00.0: cmd_wait_q terminated: -110
[   64.071303] mwifiex_pcie 0000:01:00.0: failed to get signal information
[   64.073347] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[   64.073362] mwifiex_pcie 0000:01:00.0: failed to get signal information
[   64.074375] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[   64.074389] mwifiex_pcie 0000:01:00.0: failed to get signal information
[   64.074574] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[   64.074584] mwifiex_pcie 0000:01:00.0: failed to get signal information
[   64.177717] mwifiex_pcie 0000:01:00.0: Fail to pull ctrl_data
[   64.179041] mwifiex_pcie 0000:01:00.0: == mwifiex dump information to /sys/class/devcoredump start
[   64.180969] mwifiex_pcie 0000:01:00.0: == mwifiex dump information to /sys/class/devcoredump end
[   64.180988] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[   64.181018] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[   64.181122] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[   64.181129] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[   64.181135] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[   64.181140] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[   64.181146] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[   64.181151] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[   64.181156] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[   64.181162] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[   64.181167] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[   64.181172] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[   64.181178] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[   64.181183] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[   64.181495] mwifiex_pcie 0000:01:00.0: info: shutdown mwifiex...
[   64.181541] mwifiex_pcie 0000:01:00.0: rx_pending=0, tx_pending=1,   cmd_pending=0
[   64.190232] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[   64.190248] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[   64.190262] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[   64.305399] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[   64.751341] mwifiex_pcie 0000:01:00.0: info: dnld wifi firmware from 174528 bytes
[   66.881615] mwifiex_pcie 0000:01:00.0: info: FW download over, size 638992 bytes
[   67.621274] mwifiex_pcie 0000:01:00.0: WLAN FW is active
[   67.660124] mwifiex_pcie 0000:01:00.0: Unknown api_id: 3
[   67.660139] mwifiex_pcie 0000:01:00.0: Unknown api_id: 4
[   67.660149] mwifiex_pcie 0000:01:00.0: Unknown GET_HW_SPEC TLV type: 0x217
[   67.682343] mwifiex_pcie 0000:01:00.0: info: MWIFIEX VERSION: mwifiex 1.0 (16.68.10.p159) 
[   67.682360] mwifiex_pcie 0000:01:00.0: driver_version = mwifiex 1.0 (16.68.10.p159) 
[   69.240447] mwifiex_pcie 0000:01:00.0: uap0: changing to 2 not supported
[   69.245803] mwifiex_pcie 0000:01:00.0: uap0: changing to 2 not supported
[   69.282065] mwifiex_pcie 0000:01:00.0: uap0: changing to 2 not supported
[   77.379523] mwifiex_pcie 0000:01:00.0: info: trying to associate to '4CE676160420-1' bssid 14:cc:20:7d:3e:4d
[   77.393442] mwifiex_pcie 0000:01:00.0: info: associated to bssid 14:cc:20:7d:3e:4d successfully
[   77.407853] IPv6: ADDRCONF(NETDEV_CHANGE): mlan0: link becomes ready
[  159.798938] tun: Universal TUN/TAP device driver, 1.6
[  941.742324] ci_hdrc ci_hdrc.0: remove, state 4
[  941.742354] usb usb3: USB disconnect, device number 1
[  941.743804] ci_hdrc ci_hdrc.0: USB bus 3 deregistered
[  941.746416] ci_hdrc ci_hdrc.0: switching to gadget role
[  942.089220] ci_hdrc ci_hdrc.0: switching to gadget role
[  942.524313] configfs-gadget gadget: high-speed config #1: RNDIS
[  942.524514] IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
[ 1088.222585] ci_hdrc ci_hdrc.0: switching to host role
[ 1088.222611] ci_hdrc ci_hdrc.0: EHCI Host Controller
[ 1088.222648] ci_hdrc ci_hdrc.0: new USB bus registered, assigned bus number 3
[ 1088.259928] ci_hdrc ci_hdrc.0: USB 2.0 started, EHCI 1.00
[ 1088.260986] hub 3-0:1.0: USB hub found
[ 1088.261041] hub 3-0:1.0: 1 port detected
[ 1088.261822] ci_hdrc ci_hdrc.0: remove, state 1
[ 1088.261849] usb usb3: USB disconnect, device number 1
[ 1088.270830] ci_hdrc ci_hdrc.0: USB bus 3 deregistered
[ 1088.273483] ci_hdrc ci_hdrc.0: switching to host role
[ 1088.273506] ci_hdrc ci_hdrc.0: EHCI Host Controller
[ 1088.273540] ci_hdrc ci_hdrc.0: new USB bus registered, assigned bus number 3
[ 1088.299950] ci_hdrc ci_hdrc.0: USB 2.0 started, EHCI 1.00
[ 1088.305372] hub 3-0:1.0: USB hub found
[ 1088.305440] hub 3-0:1.0: 1 port detected
[ 1225.334370] ci_hdrc ci_hdrc.0: remove, state 4
[ 1225.334400] usb usb3: USB disconnect, device number 1
[ 1225.339262] ci_hdrc ci_hdrc.0: USB bus 3 deregistered
[ 1225.341854] ci_hdrc ci_hdrc.0: switching to gadget role
[ 1226.083826] configfs-gadget gadget: high-speed config #1: RNDIS
[ 1385.241579] mwifiex_pcie 0000:01:00.0: mwifiex_cmd_timeout_func: Timeout cmd id = 0xa4, act = 0x0
[ 1385.241619] mwifiex_pcie 0000:01:00.0: num_data_h2c_failure = 0
[ 1385.241632] mwifiex_pcie 0000:01:00.0: num_cmd_h2c_failure = 0
[ 1385.241644] mwifiex_pcie 0000:01:00.0: is_cmd_timedout = 1
[ 1385.241655] mwifiex_pcie 0000:01:00.0: num_tx_timeout = 0
[ 1385.241665] mwifiex_pcie 0000:01:00.0: last_cmd_index = 3
[ 1385.241678] mwifiex_pcie 0000:01:00.0: last_cmd_id: a4 00 7f 00 16 00 a4 00 16 00
[ 1385.241690] mwifiex_pcie 0000:01:00.0: last_cmd_act: 00 00 00 00 00 00 00 00 00 00
[ 1385.241708] mwifiex_pcie 0000:01:00.0: last_cmd_resp_index = 1
[ 1385.241720] mwifiex_pcie 0000:01:00.0: last_cmd_resp_id: 7f 80 16 80 7f 80 16 80 a4 80
[ 1385.241731] mwifiex_pcie 0000:01:00.0: last_event_index = 2
[ 1385.241742] mwifiex_pcie 0000:01:00.0: last_event: 0b 00 0b 00 0a 00 0b 00 0a 00
[ 1385.241753] mwifiex_pcie 0000:01:00.0: data_sent=1 cmd_sent=1
[ 1385.241765] mwifiex_pcie 0000:01:00.0: ps_mode=1 ps_state=0
[ 1385.241813] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1385.243255] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1385.243272] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1385.243416] mwifiex_pcie 0000:01:00.0: ===mwifiex driverinfo dump start===
[ 1385.243436] mwifiex_pcie 0000:01:00.0: info: MWIFIEX VERSION: mwifiex 1.0 (16.68.10.p159) 
[ 1385.243484] mwifiex_pcie 0000:01:00.0: PCIE register dump start
[ 1385.243494] mwifiex_pcie 0000:01:00.0: pcie scratch register:
[ 1385.243512] mwifiex_pcie 0000:01:00.0: reg:0xcf0, value=0xfedcba00
               reg:0xcf8, value=0x7d00cc
               reg:0xcfc, value=0x1191900

[ 1385.243521] mwifiex_pcie 0000:01:00.0: PCIE register dump end
[ 1385.243714] mwifiex_pcie 0000:01:00.0: ===mwifiex driverinfo dump end===
[ 1385.243732] mwifiex_pcie 0000:01:00.0: == mwifiex firmware dump start ==
[ 1385.244609] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1385.244623] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1385.244878] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1385.244891] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1391.811102] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1391.811115] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1391.811316] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1391.811324] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1397.808731] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1397.808749] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1397.808904] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1397.808915] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1403.807208] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1403.807221] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1403.807338] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1403.807345] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1409.810151] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1409.810165] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1409.810348] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1409.810356] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1415.804537] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1415.804553] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1415.804711] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1415.804720] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1421.803717] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1421.803731] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1421.804018] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1421.804029] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1423.803234] mwifiex_pcie 0000:01:00.0: Ignore scan. Card removed or firmware in bad state
[ 1423.803251] mwifiex_pcie 0000:01:00.0: scan failed: -14
[ 1426.723591] mwifiex_pcie 0000:01:00.0: == mwifiex firmware dump end ==
[ 1426.725082] mwifiex_pcie 0000:01:00.0: == mwifiex dump information to /sys/class/devcoredump start
[ 1426.725363] mwifiex_pcie 0000:01:00.0: == mwifiex dump information to /sys/class/devcoredump end
[ 1426.725379] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1426.725410] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[ 1426.725490] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1426.725545] mwifiex_pcie 0000:01:00.0: info: shutdown mwifiex...
[ 1426.725609] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[ 1426.725617] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1426.725623] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[ 1426.725629] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1426.725634] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[ 1426.725640] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1426.725645] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[ 1426.725650] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1426.725656] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[ 1426.725662] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1426.725667] mwifiex_pcie 0000:01:00.0: deleting the crypto keys
[ 1426.725777] mwifiex_pcie 0000:01:00.0: rx_pending=0, tx_pending=2,   cmd_pending=0
[ 1426.734508] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1426.734526] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1426.734543] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1426.864043] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1426.864067] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1426.864074] mwifiex_pcie 0000:01:00.0: PREP_CMD: card is removed
[ 1427.329804] mwifiex_pcie 0000:01:00.0: info: dnld wifi firmware from 174528 bytes
[ 1429.462709] mwifiex_pcie 0000:01:00.0: info: FW download over, size 638992 bytes
[ 1430.178934] mwifiex_pcie 0000:01:00.0: WLAN FW is active
[ 1430.207778] mwifiex_pcie 0000:01:00.0: Unknown api_id: 3
[ 1430.207792] mwifiex_pcie 0000:01:00.0: Unknown api_id: 4
[ 1430.207805] mwifiex_pcie 0000:01:00.0: Unknown GET_HW_SPEC TLV type: 0x217
[ 1430.237727] mwifiex_pcie 0000:01:00.0: info: MWIFIEX VERSION: mwifiex 1.0 (16.68.10.p159) 
[ 1430.237741] mwifiex_pcie 0000:01:00.0: driver_version = mwifiex 1.0 (16.68.10.p159) 
[ 1430.312910] mwifiex_pcie 0000:01:00.0: uap0: changing to 2 not supported
[ 1441.122005] mwifiex_pcie 0000:01:00.0: info: trying to associate to '4CE676160420-1' bssid 14:cc:20:7d:3e:4d
[ 1441.140288] mwifiex_pcie 0000:01:00.0: info: associated to bssid 14:cc:20:7d:3e:4d successfully
[ 1441.151675] IPv6: ADDRCONF(NETDEV_CHANGE): mlan0: link becomes ready
[ 1867.894987] audit: type=1701 audit(1664787422.377:83): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=1283 comm="AmphaSoft_PRO" exe="/AmphaSoft_PRO/bin/AmphaSoft_PRO" sig=5 res=1
[ 1868.256097] audit: type=1325 audit(1664787422.737:84): table=nat family=2 entries=23
[ 1868.269180] audit: type=1325 audit(1664787422.757:85): table=filter family=2 entries=34
[ 1868.281984] audit: type=1325 audit(1664787422.767:86): table=nat family=2 entries=22
[ 1868.298868] audit: type=1325 audit(1664787422.787:87): table=nat family=2 entries=21
[ 1868.311389] audit: type=1325 audit(1664787422.797:88): table=filter family=2 entries=33
[ 1868.324186] audit: type=1325 audit(1664787422.807:89): table=nat family=2 entries=20
[ 1868.339789] audit: type=1325 audit(1664787422.827:90): table=nat family=2 entries=19
[ 1868.352201] audit: type=1325 audit(1664787422.837:91): table=filter family=2 entries=32
[ 1868.365721] audit: type=1325 audit(1664787422.847:92): table=nat family=2 entries=18
[ 1869.138433] mwifiex_pcie 0000:01:00.0: cmd_wait_q terminated: -110
[ 1869.138457] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1869.140329] vethc9ee8b4: renamed from eth0
[ 1869.188772] br-b2a177f88f99: port 1(veth2a32c75) entered disabled state
[ 1881.298533] mwifiex_pcie 0000:01:00.0: cmd_wait_q terminated: -110
[ 1881.298555] mwifiex_pcie 0000:01:00.0: failed to get signal information
[ 1882.892834] br-b2a177f88f99: port 1(veth2a32c75) entered disabled state
[ 1882.896730] device veth2a32c75 left promiscuous mode
[ 1882.896805] br-b2a177f88f99: port 1(veth2a32c75) entered disabled state

I’m not considering this really an issue due to its very low occurrence, but I’m hoping these logs can help if it gets worse in the future.
Thank you

kevin.tx · October 3, 2022, 3:30pm

Hi @mag ,

Thanks for adding additional information.

I am trying to reproduce the error on my side. Until now I had no success.

You mentioned low reception of the Wifi signal. Is it so low that sometimes disconnects happen anyway? Or is the connection stable from what you’re seeing?

Best Regards
Kevin

mag · October 4, 2022, 11:24am

Hi @kevin.tx,
WiFi reception is quite high but it is kind of unstable.
Antennas look fine but the reception seems to fall down from time to time without any reason.
Some instruments are totally fine, others are displaying this erratic signal behavior.
To give you some number we had around 8% of the instruments affected by WiFi instability, but in most cases this simply results in lower signal strength without any visible effect for the user.
If it can help, I can take out one of Colibris with the worst behavior and send it back to you.
Just let me know and I’ll organize.

Thank you
Regards

kevin.tx · October 4, 2022, 1:24pm

Hi @mag ,

Thanks for the additional insight.

At the moment I set up a Colibri iMX8X with Wifi and two antennas. After the module is booted, I plug in the OTG cable and I get the message ci_hdrc ci_hdrc.0: switching to gadget role as you have seen as well. After that I just wait for the Wifi to crash left it running over night, could not see this behavior.

I’ll try to remove one the antennas and try again, to see if this has any effect.

After all tests fail, that might be a good way to test further.

Thanks a lot, have a good day.

Best Regards
Kevin

kevin.tx · October 24, 2022, 1:51pm

Hi @mag,

It was nice meeting you in person.

I was unsuccessful reproducing the issue.

As a next step, would it be possible that you sent me the module, so I can reproduce it here on my desk?

You can send it to our office and put my name on it.

Thank you and have a nice day.

Best Regards
Kevin

mag · October 27, 2022, 11:55am

Hi @kevin.tx,
sorry for the delay.
I’ve retrieved the instrument and extracted the colibri module from inside.
I’ll ask out shipments office when they can pack and send everything to you.

Keep in mind that the instrument itself has shown some erratic behavior in the past, I don’t know if colibri/iris side or our custom electronics.

I’ll keep you updated when it is on its way.
Thanks again

mag · November 25, 2022, 8:47am

Hi @kevin.tx,
I’ve done some more tests on the affected colibri and it got worse.
I tried to reflash the “normal” torizon with containers image from TEZI, multiple times and the board does not boot anymore. So I think that specific colibri has some bigger problems than only wifi crashes.
This is what happened a few times before giving up:

Short the pins and lunch torizon easy installer
Select “Torizon with evaluation containers”
The flash proceed as intended
At the end it asks to power off or reboot. I tried both
The VNC connection drops but the board stays switched on (I waited almost half an hour)
Power circle the board and it never comes back up (no ssh access)

I haven’t tried with the serial connection for the kernel boot-up, if any.
The board is now with our shipping management, so you should receive it soon.
I’m sorry again, we had some intensive weeks.

Thanks and good luck

kevin.tx · December 2, 2022, 8:42am

Hi @mag ,

Sorry for the delayed response. We got your hardware

Thanks for the instructions we’ll give this a try and we’ll update you here.

Best Regards
Kevin

josep.tx · December 5, 2022, 11:08am

Hello @mag ,
I have tried to reproduce your issue with the module that we have received from you:

Put the module on an Iris board
Put the module in recovery mode and flashed Torizon with evaluation containers → Success
Rebooted the module → success
Connected to the module via SSH → Success

Best regards,
Josep