Apalis iMX8 PCIe

Technical Support
,
1

Apalis 1.1 eval board
iMX8 QuadMax IT
Linux apalis-imx8-14760581 5.15.129-6.4.0-212535-g6f8fd49366db-dirty

I have a PolarFire MPF300T FPGA dev board attached to Apalais X43 pci connector. The dev board shows up in lspci and the reported BAR sizes match what we have programmed. This leads me to think connectivity is good.

I added a kernel module driver to support the Polarfire. That module loads and runs but fails to “enable” the device early in the probe call while allocating BARS.

root@apalis-imx8-14760581:~# modprobe mpci
[ 223.585241] Module Inserted
[ 223.588567] Initializing 0000:03:00.0
[ 223.592424] MS_PCI_DRIVER 0000:03:00.0: can’t enable device: BAR 4 [mem 0x00000000-0x007fffff 64bit pref] not claimed
[ 223.603152] pci_enable_device() failed, err
[ 223.607474] MS_PCI_DRIVER: probe of 0000:03:00.0 failed with error -22
[ 223.614166] MS_PCI_DEV: Got Major 510
[ 223.618268] Successfully created /dev/MS_PCI_DEV
[ 223.623033]
[ 223.623033] Device Registered: MS_PCI_DEV
[ 223.628593] Major number = 510, Minor number = 0

Digging into the PCIe code reveals that the “parent” pointer in the passed down pci_dev structure is null. This causes the pci_enable_device() call to fail while processing the BARS

root@apalis-imx8-14760581:~# lspci -t
-[0000:00]—00.0-[01-ff]----00.0-[02-05]–±01.0-[03]----00.0
±02.0-[04]–
-03.0-[05]–

lspci -v shows the PEX uses the “pcieport” driver.

00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0000 (rev 01) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 573
Memory at 60000000 (32-bit, non-prefetchable) [size=16M]
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: 00000000-00000fff [size=4K]
Memory behind bridge: 62800000-62afffff [size=3M]
Prefetchable memory behind bridge: 0000000062000000-00000000627fffff [size=8M]
Expansion ROM at 61000000 [virtual] [disabled] [size=16M]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/16 Maskable- 64bit-
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Secondary PCI Express
Capabilities: [168] L1 PM Substates
Kernel driver in use: pcieport

01:00.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 573
Memory at 62a00000 (32-bit, non-prefetchable) [size=16K]
Bus: primary=01, secondary=02, subordinate=05, sec-latency=0
I/O behind bridge: 00000000-00000fff [size=4K]
Memory behind bridge: 62800000-629fffff [size=2M]
Prefetchable memory behind bridge: 0000000062000000-00000000627fffff [size=8M]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+
Capabilities: [68] Express Upstream Port, MSI 00
Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch
Capabilities: [100] Device Serial Number ab-86-02-10-b5-df-0e-00
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [138] Power Budgeting <?> Capabilities: [148] Virtual Channel Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=028 <?>
Kernel driver in use: pcieport

03:00.0 Non-VGA unclassified device: Actel Device 1556
Flags: fast devsel, IRQ 573
Memory at (64-bit, prefetchable) [virtual] [size=8M]
Capabilities: [80] Express Endpoint, MSI 00
Capabilities: [d0] MSI-X: Enable- Count=1 Masked-
Capabilities: [e0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [f8] Power Management version 3
Capabilities: [100] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
Capabilities: [108] Latency Tolerance Reporting
Capabilities: [200] Advanced Error Reporting
Kernel modules: mpci

I haven’t been able to find much info on what a null parent attribute means. Is this a driver or HW issue? Any advice would be appreciated.

2

Unfortunately we have no any experience interfacing PolarFire MPF300T FPGA. So anything below is based on generic knowledge:
Based on error messages you provided I can conclude that the Base Address Registers (BARs) not being properly claimed or allocated. This could be caused by many factors such a

  • The custom kernel module mpci for the PolarFire FPGA might not be handling the resource allocation correctly.
  • If the hardware setup or the FPGA configuration is not aligned with the system’s expectations (e.g., incorrect BAR size programmed in the FPGA), it can lead to such issues.
  • The null parent pointer in the pci_dev structure indicates a problem in the PCIe device hierarchy or enumeration process.

Please ensure that the mpci driver handles the resource allocation correctly. Pay special attention to the functions that claim and allocate memory for the BARs.
Verify that the FPGA’s configuration aligns with the PCIe specifications and the system’s expectations. This includes correct BAR size and type settings.
Investigate the PCIe device hierarchy and ensure that the PolarFire FPGA is correctly enumerated under the PCIe root complex or bridge.

Try increasing the verbosity of the kernel logging, including enabling PCIe debug messages if your kernel supports them. Also, consider using dynamic debug if possible. Check the output of the dmesg command for more details about your case. Additionally, you can examine the output of lspci -vvv for further clues about the problem.

3

Thanks for the feedback. All good ideas.

Should there be anything that I need to add in the device trees like a parent resource or something. I didn’t think it would be necessary if the driver isn’t expecting it.

4

If your driver does not require any configuration options from the Device Tree, then modifying the Device Tree is not necessary.

5

Correct me if I am wrong, but from the provided output I understand that your PolarFire FPGA PCIe ID (address 03:00.0) is connected to the downstream port of a PLX PEX 8605 switch (address 01:00.0). And, of course, the Apalis iMX8 is the RootComplex (address 00:00.0).

So, my questions to you now should be the following:

  1. Can you try to connected the FPGA as a direct endpoint, without the PCIe switch, to the Apalis iMX8?
  2. Are you sure that the complete topology is initialised properly?

The fact that PCIe enumeration has taken place does not guarantee completeness, because you have a PCIe switch in-between, which will require additional (kind of 2nd stage) initialisation. That would mean that the U-boot “firmware” should have some source code to recognise the PCIe switch (maybe load some driver that PLX provide) which will assign the register configurations (by sending a specific type of PCIe CONFIG packets). Only when the switch has become fully configured will it be able to forward all PCIe packets properly.

6

Yesterday I started looking more closely at the PLX settings thinking maybe our 8MB bar is bigger than the parent port settings would allow. We reduced the PolarFire to one 4KB, 32 bit, non-prefetchable BAR. This allows us to get though driver init and probe but when trying to read the bar we get nothing but F’s. At some point I will need to open up the setting on the PLX and was wondering where that would be, so thanks for the suggestion.

7

By the way, can you please share how you control (configure) the PLX switch?
Can you read or write to the registers?

I am simply curious…

8

That is my question also. I looked around a bit in the uboot code but didn’t see anything obvious.

If we need to adjust PCIe port bars on the PEX switch, how/where do we do that?

9

Hi,

Well, I found a topic that may be interesting:

You may also try to enter the U-Boot shell and execute a memory Read/Write to the BAR of the FPGA and figure out what the result is. This way you will identify whether the issue actually lies in pre-Linux hardware initialisation or it is something related to the DeviceTree/Linux/driver(s).

10

Had to put this size issue on the back burner for now but will update when resolved.