Regression: PCI express failing to enumerate on Apalis iMX6

Technical Support
, , ,
1

This happens on the 2.8 beta branch of the Apalis iMX6 BSP, I can reproduce this error on the Apalis Evalboard. I have an Atheros WiFi module in the mini-pci slot, but I can’t bring it up, it won’t even show up. Part of the PEX switch is also missing.

Using Linux 4.9

root@apalis-imx6:~# uname -a
Linux apalis-imx6 4.9.84-2.8.2+gb2a7f2f #1 SMP Sat Mar 31 02:44:15 UTC 2018 armv7l GNU/Linux

root@apalis-imx6:~# lspci
00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01)
01:00.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)

root@apalis-imx6:~# dmesg | grep -i pci
[    0.294791] PCI: CLS 0 bytes, default 64
[    0.315117] OF: PCI: host bridge /soc/pcie@0x01000000 ranges:
[    0.315171] OF: PCI:    IO 0x01f80000..0x01f8ffff -> 0x00000000
[    0.315203] OF: PCI:   MEM 0x01000000..0x01efffff -> 0x01000000
[    0.370682] imx6q-pcie 1ffc000.pcie: Link: Gen2 disabled
[    0.370717] imx6q-pcie 1ffc000.pcie: Link up, Gen1
[    0.370957] imx6q-pcie 1ffc000.pcie: PCI host bridge to bus 0000:00
[    0.370990] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.371017] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    0.371042] pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]
[    0.371097] pci 0000:00:00.0: [16c3:abcd] type 01 class 0x060400
[    0.371120] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
[    0.371136] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    0.371199] pci 0000:00:00.0: supports D1
[    0.371208] pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold
[    0.371479] PCI: bus0: Fast back to back transfers disabled
[    0.371703] pci 0000:01:00.0: [10b5:8605] type 01 class 0x060400
[    0.371796] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff]
[    0.372383] pci 0000:01:00.0: supports D1 D2
[    0.372391] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.402768] PCI: bus1: Fast back to back transfers disabled
[    0.402812] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    0.403175] pci_bus 0000:02: busn_res: can not insert [bus 02-01] under [bus 01] (conflicts with (null) [bus 01])
[    0.403458] PCI: bus2: Fast back to back transfers enabled
[    0.403489] pci_bus 0000:02: busn_res: [bus 02-01] end is updated to 02
[    0.403498] pci_bus 0000:02: busn_res: can not insert [bus 02] under [bus 01] (conflicts with (null) [bus 01])
[    0.403518] pci_bus 0000:02: [bus 02] partially hidden behind bridge 0000:01 [bus 01]
[    0.403565] pci 0000:00:00.0: bridge has subordinate 01 but max busn 02
[    0.403808] pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff]
[    0.403844] pci 0000:00:00.0: BAR 8: assigned [mem 0x01100000-0x011fffff]
[    0.403876] pci 0000:00:00.0: BAR 6: assigned [mem 0x01200000-0x0120ffff pref]
[    0.403917] pci 0000:01:00.0: BAR 0: assigned [mem 0x01100000-0x01103fff]
[    0.403963] pci 0000:01:00.0: PCI bridge to [bus 02]
[    0.404057] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.404083] pci 0000:00:00.0:   bridge window [mem 0x01100000-0x011fffff]
[    0.404354] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    0.404386] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    0.404417] pcie_pme 0000:00:00.0:pcie001: service driver pcie_pme loaded
[    0.404578] aer 0000:00:00.0:pcie002: service driver aer loaded
[    0.404781] pcieport 0000:01:00.0: enabling device (0140 -> 0142)
[    1.956491] ehci-pci: EHCI PCI platform driver
[    8.337812] vgaarb: this pci device is not a vga device
[    8.494775] vgaarb: this pci device is not a vga device

root@apalis-imx6:~# lspci -v
00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 320
	Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: None
	Memory behind bridge: 01100000-011fffff [size=1M]
	Prefetchable memory behind bridge: None
	[virtual] Expansion ROM at 01200000 [disabled] [size=64K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
	Capabilities: [70] Express Root Port (Slot-), MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Virtual Channel
	Kernel driver in use: pcieport

01:00.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 321
	Memory at 01100000 (32-bit, non-prefetchable) [size=16K]
	Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: None
	Memory behind bridge: None
	Prefetchable memory behind bridge: None
	Capabilities: [40] Power Management version 3
	Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
	Capabilities: [68] Express Upstream Port, MSI 00
	Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch
	Capabilities: [100] Device Serial Number ab-86-02-10-b5-df-0e-00
	Capabilities: [fb4] Advanced Error Reporting
	Capabilities: [138] Power Budgeting <?>
	Capabilities: [148] Virtual Channel
	Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=028 <?>
	Kernel driver in use: pcieport

Using Linux 4.1

root@apalis-imx6:~# uname -a
Linux apalis-imx6 4.1.44-2.7.4+gb1555bf #1 SMP Wed Oct 4 22:39:51 UTC 2017 armv7l GNU/Linux

root@apalis-imx6:~# lspci
00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01)
01:00.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)
02:01.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)
02:02.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)
02:03.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)
04:00.0 Network controller: Qualcomm Atheros AR93xx Wireless Network Adapter (rev 01)

root@apalis-imx6:~# dmesg | grep -i pci
[    0.195274] PCI: CLS 0 bytes, default 64
[    1.621788] ehci-pci: EHCI PCI platform driver
[    2.879537] 1ffc000.pcie supply pcie-bus not found, using dummy regulator
[    2.946274] imx6q-pcie 1ffc000.pcie: Configuration forces GEN1
[    2.957597] imx6q-pcie 1ffc000.pcie: PCI host bridge to bus 0000:00
[    2.967884] pci_bus 0000:00: root bus resource [io  0x1000-0xffff]
[    2.978019] pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]
[    2.988786] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.998088] pci 0000:00:00.0: [16c3:abcd] type 01 class 0x060400
[    2.998113] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
[    2.998130] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    2.998186] pci 0000:00:00.0: supports D1
[    2.998196] pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold
[    2.998405] PCI: bus0: Fast back to back transfers disabled
[    3.007958] pci 0000:01:00.0: [10b5:8605] type 01 class 0x060400
[    3.058048] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff]
[    3.058512] pci 0000:01:00.0: supports D1 D2
[    3.058523] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.072760] PCI: bus1: Fast back to back transfers disabled
[    3.082189] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    3.094517] pci 0000:02:01.0: [10b5:8605] type 01 class 0x060400
[    3.145050] pci 0000:02:01.0: supports D1 D2
[    3.145060] pci 0000:02:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.145375] pci 0000:02:02.0: [10b5:8605] type 01 class 0x060400
[    3.195900] pci 0000:02:02.0: supports D1 D2
[    3.195910] pci 0000:02:02.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.196228] pci 0000:02:03.0: [10b5:8605] type 01 class 0x060400
[    3.246752] pci 0000:02:03.0: supports D1 D2
[    3.246762] pci 0000:02:03.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.247313] PCI: bus2: Fast back to back transfers disabled
[    3.256713] pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    3.268622] pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    3.280483] pci 0000:02:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    3.308065] PCI: bus3: Fast back to back transfers enabled
[    3.317280] pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
[    3.317631] pci 0000:04:00.0: [168c:0030] type 00 class 0x028000
[    3.317749] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x0001ffff 64bit]
[    3.317941] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[    3.318255] pci 0000:04:00.0: supports D1
[    3.318265] pci 0000:04:00.0: PME# supported from D0 D1 D3hot
[    3.318662] PCI: bus4: Fast back to back transfers disabled
[    3.327897] pci_bus 0000:04: busn_res: [bus 04-ff] end is updated to 04
[    3.328172] PCI: bus5: Fast back to back transfers enabled
[    3.337279] pci_bus 0000:05: busn_res: [bus 05-ff] end is updated to 05
[    3.337306] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 05
[    3.337331] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 05
[    3.337788] pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff]
[    3.348248] pci 0000:00:00.0: BAR 8: assigned [mem 0x01100000-0x014fffff]
[    3.358637] pci 0000:00:00.0: BAR 9: assigned [mem 0x01500000-0x016fffff pref]
[    3.369399] pci 0000:00:00.0: BAR 6: assigned [mem 0x01700000-0x0170ffff pref]
[    3.380105] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    3.389640] pci 0000:01:00.0: BAR 8: assigned [mem 0x01100000-0x013fffff]
[    3.399842] pci 0000:01:00.0: BAR 9: assigned [mem 0x01500000-0x016fffff 64bit pref]
[    3.410951] pci 0000:01:00.0: BAR 0: assigned [mem 0x01400000-0x01403fff]
[    3.421085] pci 0000:01:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    3.430435] pci 0000:02:01.0: BAR 8: assigned [mem 0x01100000-0x012fffff]
[    3.440509] pci 0000:02:01.0: BAR 9: assigned [mem 0x01500000-0x016fffff 64bit pref]
[    3.451558] pci 0000:02:02.0: BAR 8: assigned [mem 0x01300000-0x013fffff]
[    3.461617] pci 0000:02:01.0: BAR 7: assigned [io  0x1000-0x1fff]
[    3.470966] pci 0000:02:01.0: PCI bridge to [bus 03]
[    3.479174] pci 0000:02:01.0:   bridge window [io  0x1000-0x1fff]
[    3.488550] pci 0000:02:01.0:   bridge window [mem 0x01100000-0x012fffff]
[    3.498610] pci 0000:02:01.0:   bridge window [mem 0x01500000-0x016fffff 64bit pref]
[    3.509641] pci 0000:04:00.0: BAR 0: assigned [mem 0x01300000-0x0131ffff 64bit]
[    3.520224] pci 0000:04:00.0: BAR 6: assigned [mem 0x01320000-0x0132ffff pref]
[    3.530630] pci 0000:02:02.0: PCI bridge to [bus 04]
[    3.538753] pci 0000:02:02.0:   bridge window [mem 0x01300000-0x013fffff]
[    3.548730] pci 0000:02:03.0: PCI bridge to [bus 05]
[    3.556887] pci 0000:01:00.0: PCI bridge to [bus 02-05]
[    3.565221] pci 0000:01:00.0:   bridge window [io  0x1000-0x1fff]
[    3.574453] pci 0000:01:00.0:   bridge window [mem 0x01100000-0x013fffff]
[    3.584361] pci 0000:01:00.0:   bridge window [mem 0x01500000-0x016fffff 64bit pref]
[    3.595236] pci 0000:00:00.0: PCI bridge to [bus 01-05]
[    3.603552] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    3.612745] pci 0000:00:00.0:   bridge window [mem 0x01100000-0x014fffff]
[    3.622647] pci 0000:00:00.0:   bridge window [mem 0x01500000-0x016fffff pref]
[    3.633203] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    3.643269] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    3.652885] pci 0000:02:01.0: Signaling PME through PCIe PME interrupt
[    3.662408] pci 0000:02:02.0: Signaling PME through PCIe PME interrupt
[    3.671870] pci 0000:04:00.0: Signaling PME through PCIe PME interrupt
[    3.681345] pci 0000:02:03.0: Signaling PME through PCIe PME interrupt
[    3.690759] pcie_pme 0000:00:00.0:pcie01: service driver pcie_pme loaded
[    3.690995] aer 0000:00:00.0:pcie02: service driver aer loaded
[    3.691214] pcieport 0000:01:00.0: enabling device (0140 -> 0143)
[    3.700960] pcieport 0000:02:01.0: enabling device (0140 -> 0143)
[    3.710731] pcieport 0000:02:02.0: enabling device (0140 -> 0142)

root@apalis-imx6:~# lspci -v
00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 324
	Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
	Bus: primary=00, secondary=01, subordinate=05, sec-latency=0
	I/O behind bridge: 00001000-00001fff
	Memory behind bridge: 01100000-014fffff
	Prefetchable memory behind bridge: 01500000-016fffff
	[virtual] Expansion ROM at 01700000 [disabled] [size=64K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
	Capabilities: [70] Express Root Port (Slot-), MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Virtual Channel
	Kernel driver in use: pcieport

01:00.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 325
	Memory at 01400000 (32-bit, non-prefetchable) [size=16K]
	Bus: primary=01, secondary=02, subordinate=05, sec-latency=0
	I/O behind bridge: 00001000-00001fff
	Memory behind bridge: 01100000-013fffff
	Prefetchable memory behind bridge: 0000000001500000-00000000016fffff
	Capabilities: [40] Power Management version 3
	Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
	Capabilities: [68] Express Upstream Port, MSI 00
	Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch
	Capabilities: [100] Device Serial Number ab-86-02-10-b5-df-0e-00
	Capabilities: [fb4] Advanced Error Reporting
	Capabilities: [138] Power Budgeting <?>
	Capabilities: [148] Virtual Channel
	Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=028 <?>
	Kernel driver in use: pcieport

02:01.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 326
	Bus: primary=02, secondary=03, subordinate=03, sec-latency=0
	I/O behind bridge: 00001000-00001fff
	Memory behind bridge: 01100000-012fffff
	Prefetchable memory behind bridge: 0000000001500000-00000000016fffff
	Capabilities: [40] Power Management version 3
	Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
	Capabilities: [68] Express Downstream Port (Slot+), MSI 00
	Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch
	Capabilities: [100] Device Serial Number ab-86-02-10-b5-df-0e-00
	Capabilities: [fb4] Advanced Error Reporting
	Capabilities: [148] Virtual Channel
	Capabilities: [520] Access Control Services
	Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=028 <?>
	Kernel driver in use: pcieport

02:02.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 327
	Bus: primary=02, secondary=04, subordinate=04, sec-latency=0
	Memory behind bridge: 01300000-013fffff
	Capabilities: [40] Power Management version 3
	Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
	Capabilities: [68] Express Downstream Port (Slot+), MSI 00
	Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch
	Capabilities: [100] Device Serial Number ab-86-02-10-b5-df-0e-00
	Capabilities: [fb4] Advanced Error Reporting
	Capabilities: [148] Virtual Channel
	Capabilities: [520] Access Control Services
	Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=028 <?>
	Kernel driver in use: pcieport

02:03.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 328
	Bus: primary=02, secondary=05, subordinate=05, sec-latency=0
	Capabilities: [40] Power Management version 3
	Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
	Capabilities: [68] Express Downstream Port (Slot+), MSI 00
	Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch
	Capabilities: [100] Device Serial Number ab-86-02-10-b5-df-0e-00
	Capabilities: [fb4] Advanced Error Reporting
	Capabilities: [148] Virtual Channel
	Capabilities: [520] Access Control Services
	Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=028 <?>
	Kernel driver in use: pcieport

04:00.0 Network controller: Qualcomm Atheros AR93xx Wireless Network Adapter (rev 01)
	Subsystem: Qualcomm Atheros AR93xx Wireless Network Adapter
	Flags: bus master, fast devsel, latency 0, IRQ 358
	Memory at 01300000 (64-bit, non-prefetchable) [size=128K]
	[virtual] Expansion ROM at 01320000 [disabled] [size=64K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable- Count=1/4 Maskable+ 64bit+
	Capabilities: [70] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Virtual Channel
	Capabilities: [300] Device Serial Number 00-00-00-00-00-00-00-00
	Kernel driver in use: ath9k
	Kernel modules: ath9k
2

We were able to reproduce this on BSP 2.8b2 while it worked fine in BSP 2.8b1. So it really looks like a regression introduced by the fslc merge to 4.9.84. We are further analysing the exact cause.

3

The offending commit is the following. As a workaround, you may just revert that one for now. We keep looking into the issue. Thanks again for reporting this.

4

Even after reverting the said commit, the conflicts… line appears, but now I can list the pcie bus devices. This line has nothing to do with the issue. I have removed this from my post, since it’s irrelevant.

5

Yes, we also noticed that together with a new OF: PCI: No bus range found for /soc/pcie@0x01000000, using [bus 00-ff] message in 2.8. However, both do not seem to have any adverse effect.

6

The bus range can now be set in the device tree, and the warning message disappears if I add a bus-range = <0x00 0xff>; property to &pcie.

This makes me wonder whether I’m supposed to set this value to something other than the default.

7

Yes, I did notice upstream also setting that bus-range node nowadays. So I cherry-picked resp. commit to our downstream as well. Looks like <0x00 0xff> is what everybody else is using as well.