IMX8 PCIe Link only working on Gen1

Hi,

I am tying to connect a 5Gbs NIC to PCIe of IMX8 module. Our hardware team made a custom carrier board per guide provided. I was able to get the NIC to communicate with module by bitbaking a custom image with the driver needed.

root@apalis-imx8-07307404:~# uname -a
Linux apalis-imx8-07307404 5.4.193-5.7.0-devel+git.f78299297185

NIC used is QNAP QXG-5G1T-111C.
NIC works on Gen3 speeds but when connected to carrier board the established connection works on Gen1 speeds and max bandwith I am able to achieve is 1.5Gbs.

Are there any other hardware limitations preventing PCIe link/speed to work on higher speeds?
Any help is appreaciated.

root@apalis-imx8-07307404:~# dmesg | grep pci
[    0.000000] Kernel command line: pci=nomsi root=PARTUUID=b7d8e883-02 ro rootwait
[    0.274859] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    1.753986] ehci-pci: EHCI PCI platform driver
[    2.574517] imx6q-pcie 5f010000.pcie: pcie_ext clock source missing or invalid
[    2.582036] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    2.604469] imx6q-pcie 5f000000.pcie: No cache used with register defaults set!
[    2.727246] imx6q-pcie 5f000000.pcie: PCIe PLL locked after 0 us.
[    2.949254] imx6q-pcie 5f000000.pcie: host bridge /bus@5f000000/pcie@0x5f000000 ranges:
[    2.957338] imx6q-pcie 5f000000.pcie:    IO 0x6ff80000..0x6ff8ffff -> 0x00000000
[    2.965051] imx6q-pcie 5f000000.pcie:   MEM 0x60000000..0x6fefffff -> 0x60000000
[    3.072575] imx6q-pcie 5f000000.pcie: Link up
[    3.077610] imx6q-pcie 5f000000.pcie: Link: Gen2 disabled
[    3.083257] imx6q-pcie 5f000000.pcie: Link up, Gen1
[    3.088510] imx6q-pcie 5f000000.pcie: PCI host bridge to bus 0000:00
[    3.094880] pci_bus 0000:00: root bus resource [bus 00-ff]
[    3.100379] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    3.106572] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fefffff]
[    3.106588] pci 0000:00:00.0: [1957:0000] type 01 class 0x060400
[    3.120561] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[    3.137906] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x00ffffff pref]
[    3.144658] pci 0000:00:00.0: supports D1 D2
[    3.148934] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[    3.158432] pci 0000:01:00.0: [1d6a:11b1] type 00 class 0x020000
[    3.171044] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0000ffff 64bit]
[    3.177883] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
[    3.184705] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x003fffff 64bit]
[    3.191539] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
[    3.198266] pci 0000:01:00.0: enabling Extended Tags
[    3.198468] pci 0000:01:00.0: supports D1 D2
[    3.212344] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.223447] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:00.0 (capable of 7.876 Gb/s with 8 GT/s x1 link)
[    3.235888] imx6q-pcie 5f010000.pcie: No cache used with register defaults set!
[    3.238916] imx6q-pcie 5f010000.pcie: PCIe PLL locked after 0 us.
[    3.269630] pci 0000:00:00.0: BAR 0: assigned [mem 0x60000000-0x60ffffff]
[    3.279495] pci 0000:00:00.0: BAR 6: assigned [mem 0x61000000-0x61ffffff pref]
[    3.286744] pci 0000:00:00.0: BAR 14: assigned [mem 0x62000000-0x625fffff]
[    3.300463] pci 0000:01:00.0: BAR 4: assigned [mem 0x62000000-0x623fffff 64bit]
[    3.307806] pci 0000:01:00.0: BAR 6: assigned [mem 0x62400000-0x6243ffff pref]
[    3.318024] pci 0000:01:00.0: BAR 0: assigned [mem 0x62440000-0x6244ffff 64bit]
[    3.328177] pci 0000:01:00.0: BAR 2: assigned [mem 0x62450000-0x62450fff 64bit]
[    3.339436] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    3.344672] pci 0000:00:00.0:   bridge window [mem 0x62000000-0x625fffff]
[    3.351753] pcieport 0000:00:00.0: PME: Signaling with IRQ 568
[    3.449288] imx6q-pcie 5f010000.pcie: host bridge /bus@5f000000/pcie@0x5f010000 ranges:
[    3.457394] imx6q-pcie 5f010000.pcie:    IO 0x7ff80000..0x7ff8ffff -> 0x00000000
[    3.465089] imx6q-pcie 5f010000.pcie:   MEM 0x70000000..0x7fefffff -> 0x70000000
[    4.472957] imx6q-pcie 5f010000.pcie: Phy link never came up
[    4.478719] imx6q-pcie 5f010000.pcie: failed to initialize host
[    4.484711] imx6q-pcie 5f010000.pcie: unable to add pcie port.
root@apalis-imx8-07307404:~# lspci
00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0000 (rev 01)
01:00.0 Ethernet controller: Aquantia Corp. AQC111 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)

modprobing the driver shows no output. Tried modprobing other drivers aswell and got no output aswell.

root@apalis-imx8-07307404:~# lspci -x
00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0000 (rev 01)
00: 57 19 00 00 07 01 10 00 01 00 04 06 00 00 01 00
10: 00 00 00 60 00 00 00 00 00 01 ff 00 f1 01 00 00
20: 00 62 50 62 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 38 01 02 00

01:00.0 Ethernet controller: Aquantia Corp. AQC111 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
00: 6a 1d b1 11 06 00 10 00 02 00 00 02 00 00 00 00
10: 04 00 44 62 00 00 00 00 04 00 45 62 00 00 00 00
20: 04 00 00 62 00 00 00 00 00 00 00 00 aa 1b b3 87
30: 00 00 00 00 40 00 00 00 00 00 00 00 38 01 00 00

root@apalis-imx8-07307404:~# lspci -vv
00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0000 (rev 01) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 568
        Region 0: Memory at 60000000 (32-bit, non-prefetchable) [size=16M]
        Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
        I/O behind bridge: [disabled]
        Memory behind bridge: 62000000-625fffff [size=6M]
        Prefetchable memory behind bridge: [disabled]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        Expansion ROM at 61000000 [virtual] [disabled] [size=16M]
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/16 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x2, ASPM L0s, Exit Latency L0s unlimited
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s (ok), Width x1 (downgraded)
                        TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
                RootCap: CRSVisible-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP+, LTR-
                         10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-, LN System CLS Not Supported, TPHComp-, ExtTPHComp-, ARIFwd-
                         AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                         AtomicOpsCtl: ReqEn- EgressBlck-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
                RootCmd: CERptEn- NFERptEn- FERptEn-
                RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                         FirstFatal- NonFatalMsg- FatalMsg- IntMsg 1
                ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
        Capabilities: [148 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
                LaneErrStat: 0
        Capabilities: [168 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=10us
                L1SubCtl2: T_PwrOn=10us
        Kernel driver in use: pcieport

01:00.0 Ethernet controller: Aquantia Corp. AQC111 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
        Subsystem: QNAP Systems, Inc. AQC111 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 568
        Region 0: Memory at 62440000 (64-bit, non-prefetchable) [size=64K]
        Region 2: Memory at 62450000 (64-bit, non-prefetchable) [size=4K]
        Region 4: Memory at 62000000 (64-bit, non-prefetchable) [size=4M]
        Expansion ROM at 62400000 [virtual] [disabled] [size=256K]
        Capabilities: [40] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s (downgraded), Width x1 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+, NROPrPrP-, LTR-
                         10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-, TPHComp-, ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [80] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [90] MSI-X: Enable- Count=32 Masked-
                Vector table: BAR=2 offset=00000000
                PBA: BAR=2 offset=00000200
        Capabilities: [a0] MSI: Enable- Count=1/32 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [c0] Vital Product Data
                Product Name: Atlantic
                Read-only fields:
                        [PN] Part number: 3290495095
                        [EC] Engineering changes: 0
                        [FG] Unknown: 61 62 63
                        [LC] Unknown: 64 65 66
                        [MN] Manufacture ID: AFDSWEWEBSFD
                        [PG] Unknown: 49 49 49
                        [SN] Serial number: CPL5938TLKMY
                        [V0] Vendor specific: wfewfe
                        [V1] Vendor specific: fwewfe
                        [V2] Vendor specific: SDFWI
                        [RV] Reserved: checksum good, 0 byte(s) reserved
                Read/write fields:
                        [YA] Asset tag: 9495829
                        [V0] Vendor specific: f34ge4rsg
                        [V1] Vendor specific: ger35g5rthghgsa3
                        [Y0] System specific: bsdfvbxcz
                        [Y1] System specific: fwefewwfe
                        [RW] Read-write area: 11 byte(s) free
                End
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [150 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [180 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
                LaneErrStat: 0
        Kernel driver in use: atlantic
        Kernel modules: atlantic

Hi @Ales !

Could you please answer the following questions, so we can better understand your setup?

  • Which exact Toradex model are you using? Please share its full name and version.
  • Which version of BSP are you using?
    • You can share the output of cat /etc/os-release and cat /etc/issue commands from the module.
  • How did you perform the setup on your device tree?
    • Please share the changes on the source code of .dts/.dtsi files and/or your device tree overlays’ source codes.
  • How was carried out the driver part?
    • Was its source code already available in the kernel?
    • If it was already there, did you need to enable the CONFIG option?
  • Please share the carrier board schematics for the PCIe
    • If you don’t feel comfortable sharing the schematics here, please send them via email: support@toradex.com
      • Please, don’t forget to reference the link to this Community thread in the email body.

Best regards,

Hi @henrique.tx ,

Toradex model I am using is Apalis iMX8QM 4GB IT v1.1d.
BSP is v5.

root@apalis-imx8-07307404:~# cat /etc/os-release 
ID=tdx-xwayland
NAME="TDX Wayland with XWayland"
VERSION="5.7.0-devel-20221013091530+build.0 (dunfell)"
VERSION_ID=5.7.0-devel-20221013091530-build.0
PRETTY_NAME="TDX Wayland with XWayland 5.7.0-devel-20221013091530+build.0 (dunfell)"
DISTRO_CODENAME="dunfell"
root@apalis-imx8-07307404:~# cat /etc/issue      
TDX Wayland with XWayland 5.7.0-devel-20221013091530+build.0 (dunfell) \n \l
Apalis-iMX8_Reference-Minimal-Image

Device tree was left as default.
Image on the module was built with Yocto using reference minimal image and installed using toradex easy installer.
Driver was first crosscompiled with kernel crosscompile toolchain (gcc-linaro) and tested to work with module and kernel and then a bitbake recipe was added to extend the yocto build base image to integrate the driver.
Driver source code is provided by manufacturer.

Schematics will be sent on email. I only have access to sch files, if pcb files are also needed let me know so i can fetch those from our hardware team.
Subject: IMX8 PCIe Link only working on Gen1 - Schematics

Best regards,
Aleš