iMX8 PCIE Modem not found after reboot

Andermutu · February 25, 2022, 8:56am

Hi,

I have a Huawei ME909s PCIe modem connected to the Ixora PCIe socket. We have recently used an Apalis TK1 as SOM, but we are moving to the Apalis iMX8. The problem is that after a system reboot, the PCIe modem is not recognized anymore when using an iMX8.

Steps to reproduce my problem:

1.- When is Ixora is turned off, and power disconnected, connect the Huawei PCIe Modem.

2.-Power on the Ixora, the modem is recognized:

root@apalis-imx8-07028727:~# lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 003: ID 12d1:15c1 Huawei Technologies Co., Ltd. ME906s LTE M.2 Module
Bus 004 Device 002: ID 0424:3503 Microchip Technology, Inc. (formerly SMSC)
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

root@apalis-imx8-07028727:~# usb-devices
T:  Bus=04 Lev=02 Prnt=02 Port=00 Cnt=01 Dev#=  3 Spd=480 MxCh= 0
D:  Ver= 2.00 Cls=00(>ifc ) Sub=00 Prot=ff MxPS=64 #Cfgs=  3
P:  Vendor=12d1 ProdID=15c1 Rev=01.02
S:  Manufacturer=Huawei Technologies Co., Ltd.
S:  Product=HUAWEI Mobile V7R11
S:  SerialNumber=0123456789ABCDEF
C:  #Ifs= 7 Cfg#= 2 Atr=a0 MxPwr=2mA
I:  If#=0x0 Alt= 0 #EPs= 1 Cls=02(commc) Sub=06 Prot=00 Driver=cdc_ether
I:  If#=0x1 Alt= 0 #EPs= 2 Cls=0a(data ) Sub=06 Prot=00 Driver=cdc_ether
I:  If#=0x2 Alt= 0 #EPs= 3 Cls=ff(vend.) Sub=06 Prot=10 Driver=option
I:  If#=0x3 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=06 Prot=13 Driver=option
I:  If#=0x4 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=06 Prot=12 Driver=option
I:  If#=0x5 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=06 Prot=06 Driver=option
I:  If#=0x6 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=06 Prot=1b Driver=option

3.- Run reboot command on the console, the modem is not recognized anymore after reboot, and the LED11 on the Ixora, Mini PCIe status indicator, turns on:

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 002: ID 0424:3503 Microchip Technology, Inc. (formerly SMSC) 
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

4.-Reboot again, the LED11 keeps on, and the modem is not recognized.

5.-Switch the Ixora power off and on. After that, the Ixora boots with LED 11 turned off, and the modem is recognized:

root@apalis-imx8-07028727:~# lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 003: ID 12d1:15c1 Huawei Technologies Co., Ltd. ME906s LTE M.2 Module
Bus 004 Device 002: ID 0424:3503 Microchip Technology, Inc. (formerly SMSC) 
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

To sum up: The modem is not recognized if I reboot the system, and it is recognized if I turn off and on the power. If I change the iMX8 with a TK1 the modem works flawlessly but LED 11 is always off with the TK1.

More environment information:

Ixora v1.2A
iMX8QM v1.1C (I have tried with 2 different SOMs)

root@apalis-imx8-07028727:~# uname -a
Linux apalis-imx8-07028727 5.4.154-5.5.0+git.c65f1622951c #1 SMP PREEMPT Mon Jan 3 15:58:01 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Thanks in advance!

Andermutu · February 28, 2022, 12:29pm

Hi,

Do you have any updates on this topic?

Regards, Ander.

kevin.tx · February 28, 2022, 2:33pm

Hi @Andermutu ,

Thanks for all the information.

Did you try the same process with a different modem than Huawei?

My first guess is that when you power off and on again, the PERST (PCIe Reset) signal is triggered.

In the case that you do a software reboot, it is possible that the signal sent by the module is not the right length to properly reset your Huawei device.

Below you see a section of the schematic that shows the PERST and other PCIe signals.

You could try to force the PERST pin into reset and then see if the modem gets recognized.

Best Regards
Kevin

Andermutu · February 28, 2022, 2:49pm

Hi @kevin.tx ,

How can I force the PERST pin without causing any damage to the iMX8 or Ixora board? I understand that the PERST pin (connected to the RESET_MOCI signal) is not controllable by software right?

Thanks for your answer.

Regards, Ander.

kevin.tx · February 28, 2022, 3:39pm

Hi @Andermutu ,

I think the command shutdown now should achieve this. Obviously, the module will then power down so it’s hard to see if it’s actually solved by this.

Is it mandatory for you that you have to use an upstream image?

Alternatively, you could use a downstream kernel, this should also remove the issue. I found some information about another customer reporting something similar and he was able to solve it by using a downstream kernel.

Best Regards
Kevin

Andermutu · February 28, 2022, 4:48pm

Hi @kevin.tx

If the module is powered down it is hard to test if it works or doesn’t. I have tried to run shutdown -h now and then reset the Ixora using the SW2 RESET push button. After that, modem is recognized, but I don’t think that is a good test…

What is the difference between upstream and downstream? How can I try a downstream kernel?

Best regards, Ander.

kevin.tx · March 1, 2022, 9:37am

Hi @Andermutu,

Yes that is true. There’s also the option to run the shutdown command with the -r option which simply reboots the machine. Did you try this as well?

The difference between upstream/mainline and downstream can be found here.

When you reflash the module through the Toradex Easy Installer, you can see that some images are marked with (UPSTREAM) in the title. Any image that does not have this tag is therefore downstream.

Best Regards
Kevin

Andermutu · March 1, 2022, 10:42am

Hi @kevin.tx,

I have downloaded and installed using Toradex Easy Installer “Linux Reference Minimal
Downstream SoC vendor based kernel Wayland/XWayland graphics back-end supported but not included” image from this link.

If I reboot the iMX8, the modem is not recognized and LED 11 still turns on. The results are the same with reboot or shutdown -r now commands.

What else could we try?

Regards, Ander.

Andermutu · March 1, 2022, 11:52am

Hi @kevin.tx,

We have done more tests: We have connected the Huawei PCIe modem to the Ixora USB port through a PCIe to USB adapter.

The first boot the modem is recognized, but after a reboot, it is not. If I disconnect the adapter and reconnect again the modem is recognized.

If I use that same PCIe to USB adapter in a Raspberry Pi, the Huawei modem is always recognized, even after a system reboot.

From our point of view, it seems more a software issue, not related to the PCIe PERST signal.

Best regards, Ander.

henrique.tx · March 2, 2022, 3:48pm

Hi @Andermutu !

According to the datasheet of your Huawei ME909s PCIe modem, it actually communicates via USB interface (rather than PCIe). It only uses the mini PCIe as a physical connection.

On the Ixora board, the mini PCIe has the PCIe data lines as well as the USB data lines. Your card will only make use of the USB data lines.

So, the fact that you got the same test results using “PCIe” and USB is actually expected.

Would be possible that something is wrong with the driver that you are using?

Could you please share the output of the dmesg after a “good” boot and after a “bad” boot?

Best regards,

Andermutu · March 4, 2022, 10:06am

Hi @henrique.tx ,

Here you have dmesg outputs. I don’t see any difference between a good and a bad boot apart from the modem being recognized in the good one.

Best regards, Ander.

Dmesg_BAD.txt (34.1 KB)
Dmesg_OK.txt (35.5 KB)

henrique.tx · March 7, 2022, 3:52pm

Hi @Andermutu !

Thanks for sharing the boot logs.

We can’t get any information besides the fact that it just never is recognized.

Strangely, when it works, it enumerates 5 different ttyUSB interfaces.

Out of curiosity, could you please share the dmesg from Raspberry Pi? So we can do a quick comparison?

Best regards,

Andermutu · March 9, 2022, 8:36am

Hi @henrique.tx ,

When I connect the modem to a Raspberry Pi it also enumerates 5 different USB interfaces:

[    6.312121] usb 1-1.3: new high-speed USB device number 5 using dwc_otg
[    6.452706] usb 1-1.3: New USB device found, idVendor=12d1, idProduct=15c1, bcdDevice= 1.02
[    6.452725] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    6.452736] usb 1-1.3: Product: HUAWEI Mobile V7R11
[    6.452745] usb 1-1.3: Manufacturer: Huawei Technologies Co., Ltd.
[    6.452754] usb 1-1.3: SerialNumber: 0123456789ABCDEF
[    6.876132] usbcore: registered new interface driver usbserial_generic
[    6.876206] usbserial: USB Serial support registered for generic
[    6.900117] cdc_ether 1-1.3:2.0 wwan0: register 'cdc_ether' at usb-3f980000.usb-1.3, Mobile Broadband Network Device, 02:1e:10:1f:00:00
[    6.900416] usbcore: registered new interface driver cdc_ether
[    6.918431] usbcore: registered new interface driver option
[    6.918516] usbserial: USB Serial support registered for GSM modem (1-port)
[    6.920625] option 1-1.3:2.2: GSM modem (1-port) converter detected
[    6.922864] usb 1-1.3: GSM modem (1-port) converter now attached to ttyUSB0
[    6.923631] option 1-1.3:2.3: GSM modem (1-port) converter detected
[    6.925409] usb 1-1.3: GSM modem (1-port) converter now attached to ttyUSB1
[    6.927207] option 1-1.3:2.4: GSM modem (1-port) converter detected
[    6.927845] usb 1-1.3: GSM modem (1-port) converter now attached to ttyUSB2
[    6.928257] option 1-1.3:2.5: GSM modem (1-port) converter detected
[    6.930707] usb 1-1.3: GSM modem (1-port) converter now attached to ttyUSB3
[    6.931263] option 1-1.3:2.6: GSM modem (1-port) converter detected
[    6.932788] usb 1-1.3: GSM modem (1-port) converter now attached to ttyUSB4

What also could we try to make it work on a iMX8? It is strange that works nice on a TK1 but not on a iMX8… This is the TK1 dmesg output:

[   10.591711] input: apalis-tk1-k20-ts as /devices/platform/spi-tegra114.1/spi_master/spi1/spi1.1/apalis-tk1-k20-ts/input/input1
[   10.635989] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[   10.636002] 8021q: adding VLAN 0 to HW filter on device eth2
[   10.636449] apalis-tk1-k20-can apalis-tk1-k20-can.0: probed 0
[   10.636787] apalis-tk1-k20-can apalis-tk1-k20-can.1: probed 1
[   11.067705] usbcore: registered new interface driver option
[   11.067759] usbserial: USB Serial support registered for GSM modem (1-port)
[   11.067938] option 1-1:2.2: GSM modem (1-port) converter detected
[   11.071466] usb 1-1: GSM modem (1-port) converter now attached to ttyUSB0
[   11.071585] option 1-1:2.3: GSM modem (1-port) converter detected
[   11.072273] usb 1-1: GSM modem (1-port) converter now attached to ttyUSB1
[   11.072413] option 1-1:2.4: GSM modem (1-port) converter detected
[   11.073098] usb 1-1: GSM modem (1-port) converter now attached to ttyUSB2
[   11.073215] option 1-1:2.5: GSM modem (1-port) converter detected
[   11.073849] usb 1-1: GSM modem (1-port) converter now attached to ttyUSB3
[   11.074310] option 1-1:2.6: GSM modem (1-port) converter detected
[   11.074723] usb 1-1: GSM modem (1-port) converter now attached to ttyUSB4

Best regards, Ander.

hfranco.tx · May 2, 2022, 12:14pm

Hi @Andermutu,

Hope you’re doing great.

Were you able to figure this out or are you still having this issue?

Best Regards,
Hiago.