PCIe devices not enumerated on imx8mp-verdin

cperry · July 15, 2024, 5:59pm

Hello,

My end goal is to bring up a custom board using an imx8mp-verdin SoM and a TW6865 which is a PCIe video encoder. I’m having issues enumerating the device, the kernel logs a “Phy link never came up” error message.

I also have a Mallow carrier board equipped with an i.MX8MP Verdin SoM which has an M.2 connector. Plugging a PCIe WiFi card in that connector also exhibits a “no enumeration” problem, which I do believe is the same problem as I see on my custom use case.

Here’s a summary of my two imx8mp-verdin setups:

Setup 1 uses a v1.1a mallow carrier board and a QCNFA335 WiFi card on the M.2 connector. This M.2 card does use PCIe, I’ve validated this by plugging it in another machine.
Setup 2 uses a custom carrier board where PCIe is hardwired to a TW6865 video decoder.

Both setups exhibits the same behavior on the different software releases I’ve tried:

On a 5.15.148 kernel built with Yocto, I’m seeing the following driver messages:

    root@verdin-imx8mp:~# dmesg | grep pci
    [    1.312892] ehci-pci: EHCI PCI platform driver
    [    1.329136] ohci-pci: OHCI PCI platform driver
    [    1.883796] imx6q-pcie 33800000.pcie: PLL REF_CLK is used!.
    [    1.890010] imx6q-pcie 33800000.pcie: PCIe PHY PLL clock is locked.
    [    1.924330] imx6q-pcie 33800000.pcie: PCIe PLL is locked.
    [    2.148578] imx6q-pcie 33800000.pcie: iATU unroll: enabled
    [    2.154099] imx6q-pcie 33800000.pcie: Detected iATU regions: 4 outbound, 4 inbound
    [    2.161739] imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000 ranges:
    [    2.169031] imx6q-pcie 33800000.pcie:       IO 0x001ff80000..0x001ff8ffff -> 0x0000000000
    [    2.177305] imx6q-pcie 33800000.pcie:      MEM 0x0018000000..0x001fefffff -> 0x0018000000
    [    2.185601] imx6q-pcie 33800000.pcie: iATU unroll: enabled
    [    2.191226] imx6q-pcie 33800000.pcie: Detected iATU regions: 4 outbound, 4 inbound
    [    3.193906] imx6q-pcie 33800000.pcie: Phy link never came up
    [    3.205070] imx6q-pcie: probe of 33800000.pcie failed with error -110

The link is detected as down so the bus driver bails out. lspci shows an empty output:

    root@verdin-imx8mp:~# lspci
    root@verdin-imx8mp:~#

I’ve also tried the reference and multimedia image by downloading them with TEZI and saw the same issue/output. I can’t recall which version I’ve tried but I remember they were using a 5.15 kernel. I can try again if you need version numbers.

I’ve also tried a 6.6.20 kernel built with Yocto where I get a slightly different result:

    root@verdin-imx8mp:~# dmesg | grep pci
    [    1.870568] imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000 ranges:
    [    1.877895] imx6q-pcie 33800000.pcie:       IO 0x001ff80000..0x001ff8ffff -> 0x0000000000
    [    1.891517] imx6q-pcie 33800000.pcie:      MEM 0x0018000000..0x001fefffff -> 0x0018000000
    [    2.122498] imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib, align 64K, limit 16G
    [    3.130617] imx6q-pcie 33800000.pcie: Phy link never came up
    [    4.136434] imx6q-pcie 33800000.pcie: Phy link never came up
    [    4.142469] imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00
    [    4.148850] pci_bus 0000:00: root bus resource [bus 00-ff]
    [    4.154357] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
    [    4.160554] pci_bus 0000:00: root bus resource [mem 0x18000000-0x1fefffff]
    [    4.167471] pci 0000:00:00.0: [16c3:abcd] type 01 class 0x060400
    [    4.173506] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
    [    4.179803] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
    [    4.186571] pci 0000:00:00.0: supports D1
    [    4.190609] pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold
    [    4.200088] pci 0000:00:00.0: BAR 0: assigned [mem 0x18000000-0x180fffff]
    [    4.206911] pci 0000:00:00.0: BAR 6: assigned [mem 0x18100000-0x1810ffff pref]
    [    4.214165] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
    [    4.219765] pcieport 0000:00:00.0: PME: Signaling with IRQ 206
    [    4.225907] pcieport 0000:00:00.0: AER: enabled with IRQ 206

The link is still down but the bus driver doesn’t bail out this time. lspci shows the root complex node but no devices:

    root@verdin-imx8mp:~# lspci -vvv
    00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 206
        Region 0: Memory at 18000000 (32-bit, non-prefetchable) [size=1M]
        Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
        I/O behind bridge: f000-0fff [disabled] [16-bit]
        Memory behind bridge: fff00000-000fffff [disabled] [32-bit]
        Prefetchable memory behind bridge: fff00000-000fffff [disabled] [32-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        Expansion ROM at 18100000 [virtual] [disabled] [size=64K]
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
            PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
            Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
            Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
            Address: 00000000400bd000  Data: 0000
            Masking: 00000000  Pending: 00000000
        Capabilities: [70] Express (v2) Root Port (Slot-), IntMsgNum 0
            DevCap:    MaxPayload 128 bytes, PhantFunc 0
                ExtTag- RBE+
            DevCtl:    CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                MaxPayload 128 bytes, MaxReadReq 512 bytes
            DevSta:    CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
            LnkCap:    Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <8us
                ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
            LnkCtl:    ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
                ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
            LnkSta:    Speed 2.5GT/s, Width x1
                TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
            RootCap: CRSVisible+
            RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
            RootSta: PME ReqID 0000, PMEStatus- PMEPending-
            DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP+ LTR-
                 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
                 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
            DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
                 AtomicOpsCtl: ReqEn- EgressBlck-
                 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
            LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
            LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
            LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
                 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                 Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v2] Advanced Error Reporting
            UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
            UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
            UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
            CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
            CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
            AERCap:    First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
            HeaderLog: 00000000 00000000 00000000 00000000
            RootCmd: CERptEn+ NFERptEn+ FERptEn+
            RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                 FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
            ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
        Capabilities: [148 v1] Secondary PCI Express
            LnkCtl3: LnkEquIntrruptEn- PerformEqu-
            LaneErrStat: 0
        Capabilities: [158 v1] L1 PM Substates
            L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+ L1_PM_Substates+
                  PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
            L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                   T_CommonMode=10us
            L1SubCtl2: T_PwrOn=10us
        Kernel driver in use: pcieport

In hex:

    root@verdin-imx8mp:~# lspci -xxx
    00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
    00: c3 16 cd ab 07 05 10 00 01 00 04 06 00 00 01 00
    10: 00 00 00 18 00 00 00 00 00 01 ff 00 f0 00 00 00
    20: f0 ff 00 00 f0 ff 00 00 00 00 00 00 00 00 00 00
    30: 00 00 00 00 40 00 00 00 00 00 00 00 cd 01 02 00
    40: 01 50 c3 db 00 00 00 00 00 00 00 00 00 00 00 00
    50: 05 70 81 01 00 d0 0b 40 00 00 00 00 00 00 00 00
    60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    70: 10 00 42 00 00 80 00 00 1f 20 10 00 11 cc 71 00
    80: 00 00 11 10 00 00 00 00 c0 03 40 00 18 00 01 00
    90: 00 00 00 00 1f 04 00 00 00 00 00 00 02 00 00 00
    a0: 03 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
    b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

I’ve also tried the latest and greatest 6.10-rc7 and have had similar results to 6.6.20.

Regards,
Charles

ToradexAI · July 15, 2024, 6:00pm

Hi, I’m the Toradex AI Assistant. I’m in early beta, and I would appreciate your feedback.
The Toradex Support team will follow up with you shortly!

Did you like my answer?

Hello,

The issue you’re encountering with PCIe devices not being enumerated on the iMX8MP Verdin platform, as indicated by the “Phy link never came up” error, suggests a problem with the PCIe link establishment. This can be due to several factors, including hardware design, power supply stability, and software configuration. Here are some steps and considerations to help troubleshoot and potentially resolve the issue:

Hardware Design and Power Supply:
- Ensure that the PCIe reference clock is stable and within the specifications required by the iMX8MP SoC and the PCIe devices you are trying to enumerate.
- Verify the power supply to the PCIe devices is stable and meets the specifications. Insufficient power can cause the PCIe link to fail.
- Check the physical connections and layout of the PCIe lines on your custom carrier board for any issues that might affect signal integrity.
Device Tree Configuration:
- Review the device tree settings related to PCIe on your custom carrier board. Ensure that the PCIe controller is correctly configured and that the device tree reflects the actual hardware design.
- For the second Ethernet PHY powered through a GPIO expander, as mentioned in your context, ensure that the power control is correctly implemented in the device tree.
Kernel Configuration:
- Ensure that your Linux kernel configuration includes support for the PCIe hardware on the iMX8MP SoC. This includes the PCIe controller driver and any other relevant configurations.
- If you are using a custom Linux kernel or a significantly modified version of the Toradex provided kernel, compare your configuration with the default configuration provided by Toradex for the iMX8MP platform to identify any missing or altered settings that might affect PCIe functionality.
Software and Firmware Updates:
- Update to the latest version of the U-Boot bootloader and Linux kernel provided by Toradex for the iMX8MP platform. Sometimes, issues with PCIe enumeration are resolved in newer software releases.
- Ensure that any firmware required by the PCIe devices (e.g., for the TW6865 video encoder or the WiFi card) is correctly installed and loaded by the Linux kernel.
Debugging and Logs:
- Use the lspci command to check if the PCIe devices are detected at the hardware level. If lspci does not show your devices, the issue is likely at the hardware or lower software (kernel/bootloader) level.
- Examine the kernel log (`dmesg

Please, also check if these links can help you solve your problem:

Network on Toradex Computer on Modules | Toradex Developer Center.