IMX8 PCIe Link only working on Gen1

Technical Support
, ,
1

Hi,

I am tying to connect a 5Gbs NIC to PCIe of IMX8 module. Our hardware team made a custom carrier board per guide provided. I was able to get the NIC to communicate with module by bitbaking a custom image with the driver needed.

root@apalis-imx8-07307404:~# uname -a
Linux apalis-imx8-07307404 5.4.193-5.7.0-devel+git.f78299297185

NIC used is QNAP QXG-5G1T-111C.
NIC works on Gen3 speeds but when connected to carrier board the established connection works on Gen1 speeds and max bandwith I am able to achieve is 1.5Gbs.

Are there any other hardware limitations preventing PCIe link/speed to work on higher speeds?
Any help is appreaciated.

root@apalis-imx8-07307404:~# dmesg | grep pci
[    0.000000] Kernel command line: pci=nomsi root=PARTUUID=b7d8e883-02 ro rootwait
[    0.274859] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    1.753986] ehci-pci: EHCI PCI platform driver
[    2.574517] imx6q-pcie 5f010000.pcie: pcie_ext clock source missing or invalid
[    2.582036] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    2.604469] imx6q-pcie 5f000000.pcie: No cache used with register defaults set!
[    2.727246] imx6q-pcie 5f000000.pcie: PCIe PLL locked after 0 us.
[    2.949254] imx6q-pcie 5f000000.pcie: host bridge /bus@5f000000/pcie@0x5f000000 ranges:
[    2.957338] imx6q-pcie 5f000000.pcie:    IO 0x6ff80000..0x6ff8ffff -> 0x00000000
[    2.965051] imx6q-pcie 5f000000.pcie:   MEM 0x60000000..0x6fefffff -> 0x60000000
[    3.072575] imx6q-pcie 5f000000.pcie: Link up
[    3.077610] imx6q-pcie 5f000000.pcie: Link: Gen2 disabled
[    3.083257] imx6q-pcie 5f000000.pcie: Link up, Gen1
[    3.088510] imx6q-pcie 5f000000.pcie: PCI host bridge to bus 0000:00
[    3.094880] pci_bus 0000:00: root bus resource [bus 00-ff]
[    3.100379] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    3.106572] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fefffff]
[    3.106588] pci 0000:00:00.0: [1957:0000] type 01 class 0x060400
[    3.120561] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[    3.137906] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x00ffffff pref]
[    3.144658] pci 0000:00:00.0: supports D1 D2
[    3.148934] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[    3.158432] pci 0000:01:00.0: [1d6a:11b1] type 00 class 0x020000
[    3.171044] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0000ffff 64bit]
[    3.177883] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
[    3.184705] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x003fffff 64bit]
[    3.191539] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
[    3.198266] pci 0000:01:00.0: enabling Extended Tags
[    3.198468] pci 0000:01:00.0: supports D1 D2
[    3.212344] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.223447] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:00.0 (capable of 7.876 Gb/s with 8 GT/s x1 link)
[    3.235888] imx6q-pcie 5f010000.pcie: No cache used with register defaults set!
[    3.238916] imx6q-pcie 5f010000.pcie: PCIe PLL locked after 0 us.
[    3.269630] pci 0000:00:00.0: BAR 0: assigned [mem 0x60000000-0x60ffffff]
[    3.279495] pci 0000:00:00.0: BAR 6: assigned [mem 0x61000000-0x61ffffff pref]
[    3.286744] pci 0000:00:00.0: BAR 14: assigned [mem 0x62000000-0x625fffff]
[    3.300463] pci 0000:01:00.0: BAR 4: assigned [mem 0x62000000-0x623fffff 64bit]
[    3.307806] pci 0000:01:00.0: BAR 6: assigned [mem 0x62400000-0x6243ffff pref]
[    3.318024] pci 0000:01:00.0: BAR 0: assigned [mem 0x62440000-0x6244ffff 64bit]
[    3.328177] pci 0000:01:00.0: BAR 2: assigned [mem 0x62450000-0x62450fff 64bit]
[    3.339436] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    3.344672] pci 0000:00:00.0:   bridge window [mem 0x62000000-0x625fffff]
[    3.351753] pcieport 0000:00:00.0: PME: Signaling with IRQ 568
[    3.449288] imx6q-pcie 5f010000.pcie: host bridge /bus@5f000000/pcie@0x5f010000 ranges:
[    3.457394] imx6q-pcie 5f010000.pcie:    IO 0x7ff80000..0x7ff8ffff -> 0x00000000
[    3.465089] imx6q-pcie 5f010000.pcie:   MEM 0x70000000..0x7fefffff -> 0x70000000
[    4.472957] imx6q-pcie 5f010000.pcie: Phy link never came up
[    4.478719] imx6q-pcie 5f010000.pcie: failed to initialize host
[    4.484711] imx6q-pcie 5f010000.pcie: unable to add pcie port.

root@apalis-imx8-07307404:~# lspci
00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0000 (rev 01)
01:00.0 Ethernet controller: Aquantia Corp. AQC111 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)

modprobing the driver shows no output. Tried modprobing other drivers aswell and got no output aswell.

root@apalis-imx8-07307404:~# lspci -x
00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0000 (rev 01)
00: 57 19 00 00 07 01 10 00 01 00 04 06 00 00 01 00
10: 00 00 00 60 00 00 00 00 00 01 ff 00 f1 01 00 00
20: 00 62 50 62 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 38 01 02 00

01:00.0 Ethernet controller: Aquantia Corp. AQC111 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
00: 6a 1d b1 11 06 00 10 00 02 00 00 02 00 00 00 00
10: 04 00 44 62 00 00 00 00 04 00 45 62 00 00 00 00
20: 04 00 00 62 00 00 00 00 00 00 00 00 aa 1b b3 87
30: 00 00 00 00 40 00 00 00 00 00 00 00 38 01 00 00


root@apalis-imx8-07307404:~# lspci -vv
00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0000 (rev 01) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 568
        Region 0: Memory at 60000000 (32-bit, non-prefetchable) [size=16M]
        Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
        I/O behind bridge: [disabled]
        Memory behind bridge: 62000000-625fffff [size=6M]
        Prefetchable memory behind bridge: [disabled]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        Expansion ROM at 61000000 [virtual] [disabled] [size=16M]
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/16 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x2, ASPM L0s, Exit Latency L0s unlimited
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s (ok), Width x1 (downgraded)
                        TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
                RootCap: CRSVisible-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP+, LTR-
                         10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-, LN System CLS Not Supported, TPHComp-, ExtTPHComp-, ARIFwd-
                         AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                         AtomicOpsCtl: ReqEn- EgressBlck-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
                RootCmd: CERptEn- NFERptEn- FERptEn-
                RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                         FirstFatal- NonFatalMsg- FatalMsg- IntMsg 1
                ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
        Capabilities: [148 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
                LaneErrStat: 0
        Capabilities: [168 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=10us
                L1SubCtl2: T_PwrOn=10us
        Kernel driver in use: pcieport

01:00.0 Ethernet controller: Aquantia Corp. AQC111 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
        Subsystem: QNAP Systems, Inc. AQC111 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 568
        Region 0: Memory at 62440000 (64-bit, non-prefetchable) [size=64K]
        Region 2: Memory at 62450000 (64-bit, non-prefetchable) [size=4K]
        Region 4: Memory at 62000000 (64-bit, non-prefetchable) [size=4M]
        Expansion ROM at 62400000 [virtual] [disabled] [size=256K]
        Capabilities: [40] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s (downgraded), Width x1 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+, NROPrPrP-, LTR-
                         10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-, TPHComp-, ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [80] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [90] MSI-X: Enable- Count=32 Masked-
                Vector table: BAR=2 offset=00000000
                PBA: BAR=2 offset=00000200
        Capabilities: [a0] MSI: Enable- Count=1/32 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [c0] Vital Product Data
                Product Name: Atlantic
                Read-only fields:
                        [PN] Part number: 3290495095
                        [EC] Engineering changes: 0
                        [FG] Unknown: 61 62 63
                        [LC] Unknown: 64 65 66
                        [MN] Manufacture ID: AFDSWEWEBSFD
                        [PG] Unknown: 49 49 49
                        [SN] Serial number: CPL5938TLKMY
                        [V0] Vendor specific: wfewfe
                        [V1] Vendor specific: fwewfe
                        [V2] Vendor specific: SDFWI
                        [RV] Reserved: checksum good, 0 byte(s) reserved
                Read/write fields:
                        [YA] Asset tag: 9495829
                        [V0] Vendor specific: f34ge4rsg
                        [V1] Vendor specific: ger35g5rthghgsa3
                        [Y0] System specific: bsdfvbxcz
                        [Y1] System specific: fwefewwfe
                        [RW] Read-write area: 11 byte(s) free
                End
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [150 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [180 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
                LaneErrStat: 0
        Kernel driver in use: atlantic
        Kernel modules: atlantic
2

Hi @Ales !

Could you please answer the following questions, so we can better understand your setup?

  • Which exact Toradex model are you using? Please share its full name and version.
  • Which version of BSP are you using?
    • You can share the output of cat /etc/os-release and cat /etc/issue commands from the module.
  • How did you perform the setup on your device tree?
    • Please share the changes on the source code of .dts/.dtsi files and/or your device tree overlays’ source codes.
  • How was carried out the driver part?
    • Was its source code already available in the kernel?
    • If it was already there, did you need to enable the CONFIG option?
  • Please share the carrier board schematics for the PCIe
    • If you don’t feel comfortable sharing the schematics here, please send them via email: support@toradex.com
      • Please, don’t forget to reference the link to this Community thread in the email body.

Best regards,

3

Hi @henrique.tx ,

Toradex model I am using is Apalis iMX8QM 4GB IT v1.1d.
BSP is v5.

root@apalis-imx8-07307404:~# cat /etc/os-release 
ID=tdx-xwayland
NAME="TDX Wayland with XWayland"
VERSION="5.7.0-devel-20221013091530+build.0 (dunfell)"
VERSION_ID=5.7.0-devel-20221013091530-build.0
PRETTY_NAME="TDX Wayland with XWayland 5.7.0-devel-20221013091530+build.0 (dunfell)"
DISTRO_CODENAME="dunfell"

root@apalis-imx8-07307404:~# cat /etc/issue      
TDX Wayland with XWayland 5.7.0-devel-20221013091530+build.0 (dunfell) \n \l
Apalis-iMX8_Reference-Minimal-Image

Device tree was left as default.
Image on the module was built with Yocto using reference minimal image and installed using toradex easy installer.
Driver was first crosscompiled with kernel crosscompile toolchain (gcc-linaro) and tested to work with module and kernel and then a bitbake recipe was added to extend the yocto build base image to integrate the driver.
Driver source code is provided by manufacturer.

Schematics will be sent on email. I only have access to sch files, if pcb files are also needed let me know so i can fetch those from our hardware team.
Subject: IMX8 PCIe Link only working on Gen1 - Schematics

Best regards,
Aleš

4

Hi @Ales !

Researching internally, I found some tests realized with Apalis iMX8QM and BSP 5.7.0 with hdparm and some SSD devices over PCIe (NVMe).

All tests were carried out with fsl,max-link-speed = <3> and with Gen3 enabled according to dmesg.

Tests on Ixora reached almost 290MB/s with Gen 3 enabled.

On Apalis Evaluation Board, the test reached around 260MB/s.

On NXP’s Evaluation Kit the result was around 300MB/s.

Unfortunately, I do not have a PCIe device that I could use to run tests.

Best regards,

5

Hi @henrique.tx ,

Thanks for your research.

  • fsl,max-link-speed: Specify PCI gen for link capability. Must be ‘2’ for
    gen2, otherwise will default to gen1. Note that the IMX6 LVDS clock outputs
    do not meet gen2 jitter requirements and thus for gen2 capability a gen2
    compliant clock generator should be used and configured.

By docs it appears a hardware clock that is compatible with gen2/3 needs to be added?

I have tested with the suggestion provided.
Here is the following overlay apalis-imx8_pcie-gen2.dts

/dts-v1/;
/plugin/;

&pciea {
    status = "okay";
    fsl,max-link-speed = <2>;
};

Compiled with procedure described here Building the Device Tree Overlays and then transfered over to module. Moved under /boot/overlays/ and entry added to /boot/overlays.txt followed with sync and reboot command.

After boot the PCIe link connected with Gen2 speeds. Success, sorts of.
In order to prevent NIC driver from overflowing and freezing the module(atleast I think it overflow kernel memory) requiring a reset the U-Boot defargs were set to fw_setenv defargs '', looks like it didn’t like nomsi. Tried few other args(dma allocation and few other pci ones) but it only made it worse or equal.

Speed test on Gen2 is around 1.56Gbs, it spikes a bit higher.
1 CPU core was at 60% and 1 at 20% out of 6 available.

Accepted connection from 172.25.1.12, port 49963
[  5] local 172.25.1.10 port 8008 connected to 172.25.1.12 port 49307
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   177 MBytes  1.48 Gbits/sec
[  5]   1.00-2.00   sec   184 MBytes  1.55 Gbits/sec
[  5]   2.00-3.00   sec   186 MBytes  1.56 Gbits/sec
[  5]   3.00-4.00   sec   186 MBytes  1.56 Gbits/sec
[  5]   4.00-5.00   sec   186 MBytes  1.56 Gbits/sec
[  5]   5.00-6.00   sec   184 MBytes  1.54 Gbits/sec
[  5]   6.00-7.00   sec   187 MBytes  1.57 Gbits/sec
[  5]   7.00-8.00   sec   187 MBytes  1.57 Gbits/sec
[  5]   8.00-9.00   sec   187 MBytes  1.57 Gbits/sec
[  5]   9.00-10.00  sec   184 MBytes  1.54 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.80 GBytes  1.55 Gbits/sec                  receiver
-----------------------------------------------------------
Accepted connection from 172.25.1.12, port 50395
[  5] local 172.25.1.10 port 8008 connected to 172.25.1.12 port 43367
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   178 MBytes  1.49 Gbits/sec
[  5]   1.00-2.00   sec   186 MBytes  1.56 Gbits/sec
[  5]   2.00-3.00   sec   186 MBytes  1.56 Gbits/sec
[  5]   3.00-4.00   sec   188 MBytes  1.58 Gbits/sec
[  5]   4.00-5.00   sec   189 MBytes  1.58 Gbits/sec
[  5]   5.00-6.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   6.00-7.00   sec   188 MBytes  1.58 Gbits/sec
[  5]   7.00-8.00   sec   189 MBytes  1.58 Gbits/sec
[  5]   8.00-9.00   sec   188 MBytes  1.58 Gbits/sec
[  5]   9.00-10.00  sec   187 MBytes  1.57 Gbits/sec
[  5]  10.00-10.00  sec   242 KBytes  1.48 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.82 GBytes  1.56 Gbits/sec                  receiver

Rerunning speed test on Gen1 with changed defargs yielded 1.54Gbs.
1 CPU core was at 60% and 1 at 16% out of 6 available.

Accepted connection from 172.25.1.12, port 40313
[  5] local 172.25.1.10 port 8008 connected to 172.25.1.12 port 49155
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   175 MBytes  1.47 Gbits/sec
[  5]   1.00-2.00   sec   183 MBytes  1.54 Gbits/sec
[  5]   2.00-3.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   3.00-4.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   4.00-5.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   5.00-6.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   6.00-7.00   sec   184 MBytes  1.55 Gbits/sec
[  5]   7.00-8.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   8.00-9.00   sec   185 MBytes  1.56 Gbits/sec
[  5]   9.00-10.00  sec   182 MBytes  1.53 Gbits/sec
[  5]  10.00-10.00  sec  78.4 KBytes  1.27 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.79 GBytes  1.54 Gbits/sec                  receiver
-----------------------------------------------------------
Accepted connection from 172.25.1.12, port 39717
[  5] local 172.25.1.10 port 8008 connected to 172.25.1.12 port 41607
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   179 MBytes  1.50 Gbits/sec
[  5]   1.00-2.00   sec   181 MBytes  1.52 Gbits/sec
[  5]   2.00-3.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   3.00-4.00   sec   184 MBytes  1.55 Gbits/sec
[  5]   4.00-5.00   sec   184 MBytes  1.55 Gbits/sec
[  5]   5.00-6.00   sec   186 MBytes  1.56 Gbits/sec
[  5]   6.00-7.00   sec   184 MBytes  1.54 Gbits/sec
[  5]   7.00-8.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   8.00-9.00   sec   184 MBytes  1.55 Gbits/sec
[  5]   9.00-10.00  sec   185 MBytes  1.55 Gbits/sec
[  5]  10.00-10.00  sec  85.5 KBytes  1.22 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.79 GBytes  1.54 Gbits/sec                  receiver

How did you perform the test to get this throughput?

Used iperf3 server and module acted as client. Server was hosted on my dev machine with 10Gbs NIC with direct connection from NIC to NIC. I have also tried iperf2 but it did not yield different results. Ipks were gotten from yocto build.

Could you please share the device tree configuration you did for your NIC?

There was no device tree configuration. The only device tree overlay in overlays.txt is default hdmi overlay and now the pci-gen2 one.

Grep of dmesg after enabling Gen2.

root@apalis-imx8-07307404:~# dmesg | grep pci
[    0.270748] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    1.775875] ehci-pci: EHCI PCI platform driver
[    2.568210] imx6q-pcie 5f010000.pcie: pcie_ext clock source missing or invalid
[    2.568682] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    2.568896] imx6q-pcie 5f000000.pcie: pcie_ext clock source missing or invalid
[    3.215525] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    3.216740] imx6q-pcie 5f010000.pcie: No cache used with register defaults set!
[    3.220477] imx6q-pcie 5f010000.pcie: PCIe PLL locked after 0 us.
[    3.224955] imx6q-pcie 5f000000.pcie: No cache used with register defaults set!
[    3.365734] imx6q-pcie 5f000000.pcie: PCIe PLL locked after 0 us.
[    3.435783] imx6q-pcie 5f010000.pcie: host bridge /bus@5f000000/pcie@0x5f010000 ranges:
[    3.443878] imx6q-pcie 5f010000.pcie:    IO 0x7ff80000..0x7ff8ffff -> 0x00000000
[    3.451599] imx6q-pcie 5f010000.pcie:   MEM 0x70000000..0x7fefffff -> 0x70000000
[    3.587805] imx6q-pcie 5f000000.pcie: host bridge /bus@5f000000/pcie@0x5f000000 ranges:
[    3.595932] imx6q-pcie 5f000000.pcie:    IO 0x6ff80000..0x6ff8ffff -> 0x00000000
[    3.603384] imx6q-pcie 5f000000.pcie:   MEM 0x60000000..0x6fefffff -> 0x60000000
[    3.703716] imx6q-pcie 5f000000.pcie: Link up
[    3.807740] imx6q-pcie 5f000000.pcie: Link up
[    3.812164] imx6q-pcie 5f000000.pcie: Link up, Gen2
[    3.817447] imx6q-pcie 5f000000.pcie: PCI host bridge to bus 0000:00
[    3.823900] pci_bus 0000:00: root bus resource [bus 00-ff]
[    3.829500] pci_bus 0000:00: root bus resource [io  0x10000-0x1ffff] (bus address [0x0000-0xffff])
[    3.838544] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fefffff]
[    3.845648] pci 0000:00:00.0: [1957:0000] type 01 class 0x060400
[    3.851772] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[    3.858107] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x00ffffff pref]
[    3.865013] pci 0000:00:00.0: supports D1 D2
[    3.869337] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[    3.882780] pci 0000:01:00.0: [1d6a:11b1] type 00 class 0x020000
[    3.888905] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0000ffff 64bit]
[    3.895740] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
[    3.902588] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x003fffff 64bit]
[    3.909412] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
[    3.916161] pci 0000:01:00.0: enabling Extended Tags
[    3.921318] pci 0000:01:00.0: supports D1 D2
[    3.925601] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.932341] pci 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0000:00:00.0 (capable of 7.876 Gb/s with 8 GT/s x1 link)
[    3.962140] pci 0000:00:00.0: BAR 0: assigned [mem 0x60000000-0x60ffffff]
[    3.968985] pci 0000:00:00.0: BAR 6: assigned [mem 0x61000000-0x61ffffff pref]
[    3.976564] pci 0000:00:00.0: BAR 14: assigned [mem 0x62000000-0x625fffff]
[    3.983476] pci 0000:01:00.0: BAR 4: assigned [mem 0x62000000-0x623fffff 64bit]
[    3.990860] pci 0000:01:00.0: BAR 6: assigned [mem 0x62400000-0x6243ffff pref]
[    3.998111] pci 0000:01:00.0: BAR 0: assigned [mem 0x62440000-0x6244ffff 64bit]
[    4.005478] pci 0000:01:00.0: BAR 2: assigned [mem 0x62450000-0x62450fff 64bit]
[    4.012834] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    4.018076] pci 0000:00:00.0:   bridge window [mem 0x62000000-0x625fffff]
[    4.025295] pcieport 0000:00:00.0: PME: Signaling with IRQ 568
[    4.031423] pcieport 0000:00:00.0: AER: enabled with IRQ 568
[    4.459482] imx6q-pcie 5f010000.pcie: Phy link never came up
[    4.465246] imx6q-pcie 5f010000.pcie: failed to initialize host
[    4.471228] imx6q-pcie 5f010000.pcie: unable to add pcie port.
6

Hi @Ales !

Thanks for the update.

Could you please test using fsl,max-link-speed = <3> on your device tree?

Although this is undocumented on NXP side, seems like it might be the key to reaching Gen3.
Some of the NXP .dtsi use it:
https://git.toradex.com/cgit/linux-toradex.git/tree/arch/arm64/boot/dts/freescale/imx8qm-ss-hsio.dtsi?h=toradex_5.4-2.3.x-imx

I am still trying to get more information about it.

Also, do you have an Ixora carrier board that you could use to test? This way I could compare your results with some internal tests.

Best regards,

7

Hi @henrique.tx !

I do not have a Ixora board. I do have a Apalis Evaluation Board v1.1A.

I have looked at NXP .dts and tried using some of their configuration. Overlay wouldnt compile with values that are set using variables (ex. GIC_SPI, IRQ_TYPE_LEVEL_HIGH, IMX_SC_R_PCIE_A). After commenting those out i was greeted by some warnings.

make[1]: Entering directory '/home/ales/toradex/kernel64/apalis/imx8/bsp5/linux-toradex'
  DTC     /home/ales/toradex/images/device-tree-overlays/overlays/apalis-imx8_pcie-gen3.dtbo
/home/ales/toradex/images/device-tree-overlays/overlays/apalis-imx8_pcie-gen3.dts:18.5-19.63: Warning (ranges_format): /fragment@0/__overlay__:ranges: "ranges" property has invalid length (48 bytes) (parent #address-cells == 2, child #address-cells == 3, #size-cells == 2)
/home/ales/toradex/images/device-tree-overlays/overlays/apalis-imx8_pcie-gen3.dts:6.8-41.3: Warning (pci_bridge): /fragment@0/__overlay__: node name is not "pci" or "pcie"
/home/ales/toradex/images/device-tree-overlays/overlays/apalis-imx8_pcie-gen3.dtbo: Warning (pci_device_bus_num): Failed prerequisite 'pci_bridge'
/home/ales/toradex/images/device-tree-overlays/overlays/apalis-imx8_pcie-gen3.dts:6.8-41.3: Warning (avoid_default_addr_size): /fragment@0/__overlay__: Relying on default #address-cells value
/home/ales/toradex/images/device-tree-overlays/overlays/apalis-imx8_pcie-gen3.dts:6.8-41.3: Warning (avoid_default_addr_size): /fragment@0/__overlay__: Relying on default #size-cells value
  Building modules, stage 2.
  MODPOST 0 modules
make[1]: Leaving directory '/home/ales/toradex/kernel64/apalis/imx8/bsp5/linux-toradex'

It did produce a .dtbo but loading it on the module resulted in pciea not even showing in dmesg and no devices were found under lspci.

Here is the following overlay that produced that.

// try to force pcie gen 3

/dts-v1/;
/plugin/;

&pciea {
    status = "okay";

    compatible = "fsl,imx8qm-pcie", "snps,dw-pcie";
    reg = <0x5f000000 0x10000>, // ctrl
          <0x6ff00000 0x80000>, // cfg
          <0x5f080000 0xf0000>; // lpcg, csr, msic, gpio
    reg-names = "dbi", "config", "hsio";
    #address-cells = <3>;
    #size-cells = <2>;
    device_type = "pci";
    bus-range = <0x00 0xff>;
    ranges = <0x81000000 0 0x00000000 0x6ff80000 0 0x00010000 // DS io
             0x82000000 0 0x60000000 0x60000000 0 0x0ff00000>; // non-prefetchable memory
    num-lanes = <1>;
    //interrupts = <GIC_SPI 70 IRQ_TYPE_LEVEL_HIGH>, <GIC_SPI 72 IRQ_TYPE_LEVEL_HIGH>;
    //interrupt-names = "msi", "dma";
    //interrupt-map-mask = <0 0 0 0x7>;
    //interrupt-map = <0 0 0 1 &gic 0 73 4>,
    //                <0 0 0 2 &gic 0 74 4>,
    //                <0 0 0 3 &gic 0 75 4>,
    //                <0 0 0 4 &gic 0 76 4>;
    clocks = <&pciea_lpcg 0>,
             <&pciea_lpcg 1>,
             <&pciea_lpcg 2>,
             <&phyx2_lpcg 0>,
             <&phyx2_crr0_lpcg 0>,
             <&pciea_crr2_lpcg 0>,
             <&misc_crr5_lpcg 0>;
    clock-names = "pcie", "pcie_bus", "pcie_inbound_axi", "pcie_phy", "phy_per", "pcie_per", "misc_per";
    //power-domains = <&pd IMX_SC_R_PCIE_A>, <&pd IMX_SC_R_SERDES_0>, <&pd IMX_SC_R_HSIO_GPIO>;
    //power-domain-names = "pcie", "pcie_phy", "hsio_gpio";
    fsl,max-link-speed = <3>;
    //hsio-cfg = <PCIEAX1PCIEBX1SATA>;
    local-addr = <0x40000000>;
};

Then I just tested with a simple overlay that compiled and kernel showed that pcie is working on Gen3.

/dts-v1/;
/plugin/;

&pciea {
    status = "okay";

    fsl,max-link-speed = <3>;
};

root@apalis-imx8-07307404:~# lspci -vv
00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0000 (rev 01) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 568
        Region 0: Memory at 60000000 (32-bit, non-prefetchable) [size=16M]
        Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
        I/O behind bridge: [disabled]
        Memory behind bridge: 62000000-625fffff [size=6M]
        Prefetchable memory behind bridge: [disabled]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        Expansion ROM at 61000000 [virtual] [disabled] [size=16M]
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/16 Maskable- 64bit-
                Address: 00000000  Data: 0000
        Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x2, ASPM L0s, Exit Latency L0s unlimited
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (ok), Width x1 (downgraded)
                        TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+
                RootCap: CRSVisible-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP+, LTR-
                         10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-, LN System CLS Not Supported, TPHComp-, ExtTPHComp-, ARIFwd-
                         AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                         AtomicOpsCtl: ReqEn- EgressBlck-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
                         EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
                RootCmd: CERptEn+ NFERptEn+ FERptEn+
                RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                         FirstFatal- NonFatalMsg- FatalMsg- IntMsg 1
                ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
        Capabilities: [148 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
                LaneErrStat: 0
        Capabilities: [168 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=10us
                L1SubCtl2: T_PwrOn=10us
        Kernel driver in use: pcieport

01:00.0 Ethernet controller: Aquantia Corp. AQC111 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
        Subsystem: QNAP Systems, Inc. AQC111 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 568
        Region 0: Memory at 62440000 (64-bit, non-prefetchable) [size=64K]
        Region 2: Memory at 62450000 (64-bit, non-prefetchable) [size=4K]
        Region 4: Memory at 62000000 (64-bit, non-prefetchable) [size=4M]
        Expansion ROM at 62400000 [virtual] [disabled] [size=256K]
        Capabilities: [40] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (ok), Width x1 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+, NROPrPrP-, LTR-
                         10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-, TPHComp-, ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
                         EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
        Capabilities: [80] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [90] MSI-X: Enable+ Count=32 Masked-
                Vector table: BAR=2 offset=00000000
                PBA: BAR=2 offset=00000200
        Capabilities: [a0] MSI: Enable- Count=1/32 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [c0] Vital Product Data
                Product Name: Atlantic
                Read-only fields:
                        [PN] Part number: 3290495095
                        [EC] Engineering changes: 0
                        [FG] Unknown: 61 62 63
                        [LC] Unknown: 64 65 66
                        [MN] Manufacture ID: AFDSWEWEBSFD
                        [PG] Unknown: 49 49 49
                        [SN] Serial number: CPL5938TLKMY
                        [V0] Vendor specific: wfewfe
                        [V1] Vendor specific: fwewfe
                        [V2] Vendor specific: SDFWI
                        [RV] Reserved: checksum good, 0 byte(s) reserved
                Read/write fields:
                        [YA] Asset tag: 9495829
                        [V0] Vendor specific: f34ge4rsg
                        [V1] Vendor specific: ger35g5rthghgsa3
                        [Y0] System specific: bsdfvbxcz
                        [Y1] System specific: fwefewwfe
                        [RW] Read-write area: 11 byte(s) free
                End
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [150 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [180 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
                LaneErrStat: 0
        Kernel driver in use: atlantic
        Kernel modules: atlantic

root@apalis-imx8-07307404:~# dmesg | grep pci
[    0.269861] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    1.775232] ehci-pci: EHCI PCI platform driver
[    2.521656] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    2.522160] imx6q-pcie 5f010000.pcie: pcie_ext clock source missing or invalid
[    2.539318] imx6q-pcie 5f000000.pcie: No cache used with register defaults set!
[    2.661768] imx6q-pcie 5f000000.pcie: PCIe PLL locked after 0 us.
[    2.883811] imx6q-pcie 5f000000.pcie: host bridge /bus@5f000000/pcie@0x5f000000 ranges:
[    2.891896] imx6q-pcie 5f000000.pcie:    IO 0x6ff80000..0x6ff8ffff -> 0x00000000
[    2.899617] imx6q-pcie 5f000000.pcie:   MEM 0x60000000..0x6fefffff -> 0x60000000
[    3.004059] imx6q-pcie 5f000000.pcie: Link up
[    3.108451] imx6q-pcie 5f000000.pcie: Link up
[    3.112834] imx6q-pcie 5f000000.pcie: Link up, Gen3
[    3.117811] imx6q-pcie 5f000000.pcie: PCI host bridge to bus 0000:00
[    3.124177] pci_bus 0000:00: root bus resource [bus 00-ff]
[    3.124181] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    3.138737] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fefffff]
[    3.151632] pci 0000:00:00.0: [1957:0000] type 01 class 0x060400
[    3.163839] imx6q-pcie 5f010000.pcie: No cache used with register defaults set!
[    3.164280] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[    3.167241] imx6q-pcie 5f010000.pcie: PCIe PLL locked after 0 us.
[    3.187290] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x00ffffff pref]
[    3.208072] pci 0000:00:00.0: supports D1 D2
[    3.217755] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[    3.231867] pci 0000:01:00.0: [1d6a:11b1] type 00 class 0x020000
[    3.240926] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0000ffff 64bit]
[    3.247766] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
[    3.254722] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x003fffff 64bit]
[    3.261603] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
[    3.268496] pci 0000:01:00.0: enabling Extended Tags
[    3.273833] pci 0000:01:00.0: supports D1 D2
[    3.278131] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.304044] pci 0000:00:00.0: BAR 0: assigned [mem 0x60000000-0x60ffffff]
[    3.311113] pci 0000:00:00.0: BAR 6: assigned [mem 0x61000000-0x61ffffff pref]
[    3.318356] pci 0000:00:00.0: BAR 14: assigned [mem 0x62000000-0x625fffff]
[    3.325304] pci 0000:01:00.0: BAR 4: assigned [mem 0x62000000-0x623fffff 64bit]
[    3.332652] pci 0000:01:00.0: BAR 6: assigned [mem 0x62400000-0x6243ffff pref]
[    3.339955] pci 0000:01:00.0: BAR 0: assigned [mem 0x62440000-0x6244ffff 64bit]
[    3.347308] pci 0000:01:00.0: BAR 2: assigned [mem 0x62450000-0x62450fff 64bit]
[    3.354668] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    3.359916] pci 0000:00:00.0:   bridge window [mem 0x62000000-0x625fffff]
[    3.367225] pcieport 0000:00:00.0: PME: Signaling with IRQ 568
[    3.373386] pcieport 0000:00:00.0: AER: enabled with IRQ 568
[    3.379766] imx6q-pcie 5f010000.pcie: host bridge /bus@5f000000/pcie@0x5f010000 ranges:
[    3.387823] imx6q-pcie 5f010000.pcie:    IO 0x7ff80000..0x7ff8ffff -> 0x00000000
[    3.395250] imx6q-pcie 5f010000.pcie:   MEM 0x70000000..0x7fefffff -> 0x70000000
[    4.387758] imx6q-pcie 5f010000.pcie: Phy link never came up
[    4.393510] imx6q-pcie 5f010000.pcie: failed to initialize host
[    4.399477] imx6q-pcie 5f010000.pcie: unable to add pcie port.

But after starting a speedtest i got greeted by stderr and module froze. After reverting back to Gen1, got the exact message few more times and then it stoped.

Broadcast message from systemd-journald@apalis-imx8-07307404 (Wed 2022-12-14 06:36:51 UTC):

kernel[608]: [   29.189105] Internal error: Oops: 96000004 [#1] PREEMPT SMP


Broadcast message from systemd-journald@apalis-imx8-07307404 (Wed 2022-12-14 06:36:51 UTC):

kernel[608]: [   29.396123] Code: f94006d3 b40000d3 d503201f aa1303e0 (f9400273)

As far the speed test goes I tried with different settings and parameters but it did not improve or it made it way worse. I even replaced a cable for cat6 sftp, prior tests were done on short length 5e which i confirmed that its max transfer was 6.2Gbs. But just in case.

Updated tests using cat6:

  • Gen 2
Accepted connection from 172.25.1.12, port 49265
[  5] local 172.25.1.10 port 8008 connected to 172.25.1.12 port 38587
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   167 MBytes  1.40 Gbits/sec
[  5]   1.00-2.00   sec   181 MBytes  1.52 Gbits/sec
[  5]   2.00-3.00   sec   187 MBytes  1.57 Gbits/sec
[  5]   3.00-4.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   4.00-5.00   sec   188 MBytes  1.57 Gbits/sec
[  5]   5.00-6.00   sec   188 MBytes  1.58 Gbits/sec
[  5]   6.00-7.00   sec   189 MBytes  1.58 Gbits/sec
[  5]   7.00-8.00   sec   186 MBytes  1.56 Gbits/sec
[  5]   8.00-9.00   sec   188 MBytes  1.58 Gbits/sec
[  5]   9.00-10.00  sec   189 MBytes  1.58 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.80 GBytes  1.55 Gbits/sec                  receiver
-----------------------------------------------------------
Accepted connection from 172.25.1.12, port 37415
[  5] local 172.25.1.10 port 8008 connected to 172.25.1.12 port 59463
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   177 MBytes  1.49 Gbits/sec
[  5]   1.00-2.00   sec   186 MBytes  1.56 Gbits/sec
[  5]   2.00-3.00   sec   188 MBytes  1.57 Gbits/sec
[  5]   3.00-4.00   sec   189 MBytes  1.58 Gbits/sec
[  5]   4.00-5.00   sec   187 MBytes  1.56 Gbits/sec
[  5]   5.00-6.00   sec   186 MBytes  1.56 Gbits/sec
[  5]   6.00-7.00   sec   189 MBytes  1.58 Gbits/sec
[  5]   7.00-8.00   sec   189 MBytes  1.59 Gbits/sec
[  5]   8.00-9.00   sec   189 MBytes  1.59 Gbits/sec
[  5]   9.00-10.00  sec   185 MBytes  1.55 Gbits/sec
[  5]  10.00-10.00  sec  99.8 KBytes  1.29 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.82 GBytes  1.56 Gbits/sec                  receiver
  • Gen 1
-----------------------------------------------------------
Accepted connection from 172.25.1.12, port 33989
[  5] local 172.25.1.10 port 8008 connected to 172.25.1.12 port 48555
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   157 MBytes  1.31 Gbits/sec
[  5]   1.00-2.00   sec   171 MBytes  1.43 Gbits/sec
[  5]   2.00-3.00   sec   181 MBytes  1.52 Gbits/sec
[  5]   3.00-4.00   sec   185 MBytes  1.56 Gbits/sec
[  5]   4.00-5.00   sec   182 MBytes  1.53 Gbits/sec
[  5]   5.00-6.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   6.00-7.00   sec   179 MBytes  1.50 Gbits/sec
[  5]   7.00-8.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   8.00-9.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   9.00-10.00  sec   184 MBytes  1.54 Gbits/sec
[  5]  10.00-10.03  sec  5.92 MBytes  1.56 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.03  sec  1.76 GBytes  1.50 Gbits/sec                  receiver
-----------------------------------------------------------
Accepted connection from 172.25.1.12, port 38013
[  5] local 172.25.1.10 port 8008 connected to 172.25.1.12 port 35999
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   180 MBytes  1.51 Gbits/sec
[  5]   1.00-2.00   sec   181 MBytes  1.52 Gbits/sec
[  5]   2.00-3.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   3.00-4.00   sec   183 MBytes  1.53 Gbits/sec
[  5]   4.00-5.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   5.00-6.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   6.00-7.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   7.00-8.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   8.00-9.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   9.00-10.00  sec   184 MBytes  1.54 Gbits/sec
[  5]  10.00-10.01  sec  1.38 MBytes  1.54 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  1.80 GBytes  1.54 Gbits/sec                  receiver

Also tried compiling imx8qm-ss-hsio.dtsi as a whole and got a syntax error on lines 7 to 13 which is the start of definitions. Confirmed that the STAGING_KERNEL_DIR path is correct.

Best regards,
Aleš

8

Hi @Ales !

According to the Apalis Evaluation Board datasheet, it has a Gen2 PCIe bridge. This might be the limitation you are facing.

Ixora Carrier Board does not have it. So, for better throughput, you need to use another carrier board than Apalis Evaluation Board.

image
image837×148 20.1 KB

(Upcoming version of the Apalis Evaluation Board will not have this limitation).

Best regards,

9

Hi @henrique.tx !

I am aware of limitation in Apalis Eval Board for PCIe. Thats why I am using custom carrier baord designed by us which had Gen3 specifications in mind.

Nevertheless, here are test results on Apalis Eval Board(Gen2 one).

root@apalis-imx8-07307404:~# lspci
00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0000 (rev 01)
01:00.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)
02:01.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)
02:02.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)
02:03.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)
05:00.0 Ethernet controller: Aquantia Corp. AQC111 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
  • Gen 1
root@apalis-imx8-07307404:~# dmesg | grep pci
[    0.252914] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    1.717130] ehci-pci: EHCI PCI platform driver
[    2.524767] imx6q-pcie 5f010000.pcie: pcie_ext clock source missing or invalid
[    2.524911] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    2.525010] imx6q-pcie 5f000000.pcie: pcie_ext clock source missing or invalid
[    3.177240] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    3.178061] imx6q-pcie 5f010000.pcie: No cache used with register defaults set!
[    3.181546] imx6q-pcie 5f010000.pcie: PCIe PLL locked after 0 us.
[    3.186868] imx6q-pcie 5f000000.pcie: No cache used with register defaults set!
[    3.327145] imx6q-pcie 5f000000.pcie: PCIe PLL locked after 0 us.
[    3.397193] imx6q-pcie 5f010000.pcie: host bridge /bus@5f000000/pcie@0x5f010000 ranges:
[    3.405292] imx6q-pcie 5f010000.pcie:    IO 0x7ff80000..0x7ff8ffff -> 0x00000000
[    3.413003] imx6q-pcie 5f010000.pcie:   MEM 0x70000000..0x7fefffff -> 0x70000000
[    3.549165] imx6q-pcie 5f000000.pcie: host bridge /bus@5f000000/pcie@0x5f000000 ranges:
[    3.557265] imx6q-pcie 5f000000.pcie:    IO 0x6ff80000..0x6ff8ffff -> 0x00000000
[    3.564983] imx6q-pcie 5f000000.pcie:   MEM 0x60000000..0x6fefffff -> 0x60000000
[    3.672534] imx6q-pcie 5f000000.pcie: Link up
[    3.676922] imx6q-pcie 5f000000.pcie: Link: Gen2 disabled
[    3.682523] imx6q-pcie 5f000000.pcie: Link up, Gen1
[    3.687812] imx6q-pcie 5f000000.pcie: PCI host bridge to bus 0000:00
[    3.694187] pci_bus 0000:00: root bus resource [bus 00-ff]
[    3.699703] pci_bus 0000:00: root bus resource [io  0x10000-0x1ffff] (bus address [0x0000-0xffff])
[    3.708694] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fefffff]
[    3.715626] pci 0000:00:00.0: [1957:0000] type 01 class 0x060400
[    3.721672] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[    3.727976] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x00ffffff pref]
[    3.734735] pci 0000:00:00.0: supports D1 D2
[    3.739029] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[    3.751658] pci 0000:01:00.0: [10b5:8605] type 01 class 0x060400
[    3.757767] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff]
[    3.764414] pci 0000:01:00.0: supports D1 D2
[    3.768711] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.775496] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:00.0 (capable of 4.000 Gb/s with 5 GT/s x1 link)
[    3.812586] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    3.821260] pci 0000:02:01.0: [10b5:8605] type 01 class 0x060400
[    3.827730] pci 0000:02:01.0: supports D1 D2
[    3.832016] pci 0000:02:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.839238] pci 0000:02:02.0: [10b5:8605] type 01 class 0x060400
[    3.845689] pci 0000:02:02.0: supports D1 D2
[    3.849986] pci 0000:02:02.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.857142] pci 0000:02:03.0: [10b5:8605] type 01 class 0x060400
[    3.863597] pci 0000:02:03.0: supports D1 D2
[    3.867888] pci 0000:02:03.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.880041] pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    3.888099] pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    3.896144] pci 0000:02:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    3.910109] pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
[    3.923263] pci_bus 0000:04: busn_res: [bus 04-ff] end is updated to 04
[    3.930135] pci 0000:05:00.0: [1d6a:11b1] type 00 class 0x020000
[    3.936314] pci 0000:05:00.0: reg 0x10: [mem 0x00000000-0x0000ffff 64bit]
[    3.943198] pci 0000:05:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
[    3.950063] pci 0000:05:00.0: reg 0x20: [mem 0x00000000-0x003fffff 64bit]
[    3.956909] pci 0000:05:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
[    3.963650] pci 0000:05:00.0: enabling Extended Tags
[    3.968880] pci 0000:05:00.0: supports D1 D2
[    3.973162] pci 0000:05:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.979959] pci 0000:05:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:00.0 (capable of 7.876 Gb/s with 8 GT/s x1 link)
[    3.999723] pci_bus 0000:05: busn_res: [bus 05-ff] end is updated to 05
[    4.006382] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 05
[    4.013101] pci 0000:00:00.0: BAR 0: assigned [mem 0x60000000-0x60ffffff]
[    4.019921] pci 0000:00:00.0: BAR 6: assigned [mem 0x61000000-0x61ffffff pref]
[    4.027162] pci 0000:00:00.0: BAR 14: assigned [mem 0x62000000-0x628fffff]
[    4.034072] pci 0000:00:00.0: BAR 15: assigned [mem 0x62900000-0x62afffff 64bit pref]
[    4.041937] pci 0000:00:00.0: BAR 13: assigned [io  0x10000-0x10fff]
[    4.048351] pci 0000:01:00.0: BAR 14: assigned [mem 0x62000000-0x627fffff]
[    4.055270] pci 0000:01:00.0: BAR 15: assigned [mem 0x62900000-0x62afffff 64bit pref]
[    4.063150] pci 0000:01:00.0: BAR 0: assigned [mem 0x62800000-0x62803fff]
[    4.069972] pci 0000:01:00.0: BAR 13: assigned [io  0x10000-0x10fff]
[    4.080278] pci 0000:02:03.0: BAR 14: assigned [mem 0x62000000-0x625fffff]
[    4.080284] pci 0000:02:01.0: BAR 14: assigned [mem 0x62600000-0x627fffff]
[    4.098309] pci 0000:02:01.0: BAR 15: assigned [mem 0x62900000-0x62afffff 64bit pref]
[    4.106189] pci 0000:02:01.0: BAR 13: assigned [io  0x10000-0x10fff]
[    4.112595] pci 0000:02:01.0: PCI bridge to [bus 03]
[    4.117608] pci 0000:02:01.0:   bridge window [io  0x10000-0x10fff]
[    4.123933] pci 0000:02:01.0:   bridge window [mem 0x62600000-0x627fffff]
[    4.130766] pci 0000:02:01.0:   bridge window [mem 0x62900000-0x62afffff 64bit pref]
[    4.138569] pci 0000:02:02.0: PCI bridge to [bus 04]
[    4.143586] pci 0000:05:00.0: BAR 4: assigned [mem 0x62000000-0x623fffff 64bit]
[    4.150954] pci 0000:05:00.0: BAR 6: assigned [mem 0x62400000-0x6243ffff pref]
[    4.158207] pci 0000:05:00.0: BAR 0: assigned [mem 0x62440000-0x6244ffff 64bit]
[    4.165575] pci 0000:05:00.0: BAR 2: assigned [mem 0x62450000-0x62450fff 64bit]
[    4.172929] pci 0000:02:03.0: PCI bridge to [bus 05]
[    4.177930] pci 0000:02:03.0:   bridge window [mem 0x62000000-0x625fffff]
[    4.184758] pci 0000:01:00.0: PCI bridge to [bus 02-05]
[    4.190023] pci 0000:01:00.0:   bridge window [io  0x10000-0x10fff]
[    4.196320] pci 0000:01:00.0:   bridge window [mem 0x62000000-0x627fffff]
[    4.203138] pci 0000:01:00.0:   bridge window [mem 0x62900000-0x62afffff 64bit pref]
[    4.210913] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    4.216165] pci 0000:00:00.0:   bridge window [io  0x10000-0x10fff]
[    4.222459] pci 0000:00:00.0:   bridge window [mem 0x62000000-0x628fffff]
[    4.229274] pci 0000:00:00.0:   bridge window [mem 0x62900000-0x62afffff 64bit pref]
[    4.237513] pcieport 0000:00:00.0: PME: Signaling with IRQ 568
[    4.243750] pcieport 0000:00:00.0: AER: enabled with IRQ 568
[    4.249671] pcieport 0000:01:00.0: enabling device (0000 -> 0003)
[    4.256621] pcieport 0000:02:01.0: enabling device (0000 -> 0003)
[    4.263298] pcieport 0000:02:01.0: pciehp: Slot #1 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl- LLActRep+
[    4.277320] pcieport 0000:02:03.0: enabling device (0000 -> 0002)
[    4.420796] imx6q-pcie 5f010000.pcie: Phy link never came up
[    4.426563] imx6q-pcie 5f010000.pcie: failed to initialize host
[    4.432556] imx6q-pcie 5f010000.pcie: unable to add pcie port.

Accepted connection from 172.25.1.12, port 40245
[  5] local 172.25.1.10 port 8008 connected to 172.25.1.12 port 45645
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   149 MBytes  1.25 Gbits/sec
[  5]   1.00-2.00   sec   165 MBytes  1.39 Gbits/sec
[  5]   2.00-3.00   sec   171 MBytes  1.43 Gbits/sec
[  5]   3.00-4.00   sec   171 MBytes  1.44 Gbits/sec
[  5]   4.00-5.00   sec   169 MBytes  1.42 Gbits/sec
[  5]   5.00-6.00   sec   171 MBytes  1.43 Gbits/sec
[  5]   6.00-7.00   sec   169 MBytes  1.42 Gbits/sec
[  5]   7.00-8.00   sec   168 MBytes  1.41 Gbits/sec
[  5]   8.00-9.00   sec   168 MBytes  1.41 Gbits/sec
[  5]   9.00-10.00  sec   170 MBytes  1.43 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.63 GBytes  1.40 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 8008
-----------------------------------------------------------
Accepted connection from 172.25.1.12, port 47425
[  5] local 172.25.1.10 port 8008 connected to 172.25.1.12 port 39279
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   154 MBytes  1.29 Gbits/sec
[  5]   1.00-2.00   sec   163 MBytes  1.36 Gbits/sec
[  5]   2.00-3.00   sec   165 MBytes  1.38 Gbits/sec
[  5]   3.00-4.00   sec   165 MBytes  1.38 Gbits/sec
[  5]   4.00-5.00   sec   165 MBytes  1.38 Gbits/sec
[  5]   5.00-6.00   sec   171 MBytes  1.43 Gbits/sec
[  5]   6.00-7.00   sec   171 MBytes  1.43 Gbits/sec
[  5]   7.00-8.00   sec   170 MBytes  1.43 Gbits/sec
[  5]   8.00-9.00   sec   171 MBytes  1.44 Gbits/sec
[  5]   9.00-10.00  sec   171 MBytes  1.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.63 GBytes  1.40 Gbits/sec                  receiver
  • Gen 2
    Ran this test twice. After reboot I got lower speeds.
root@apalis-imx8-07307404:~# dmesg | grep pci 
[    0.275090] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    1.779165] ehci-pci: EHCI PCI platform driver
[    2.536355] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    2.536885] imx6q-pcie 5f010000.pcie: pcie_ext clock source missing or invalid
[    2.554335] imx6q-pcie 5f000000.pcie: No cache used with register defaults set!
[    2.675319] imx6q-pcie 5f000000.pcie: PCIe PLL locked after 0 us.
[    2.897376] imx6q-pcie 5f000000.pcie: host bridge /bus@5f000000/pcie@0x5f000000 ranges:
[    2.905464] imx6q-pcie 5f000000.pcie:    IO 0x6ff80000..0x6ff8ffff -> 0x00000000
[    2.913188] imx6q-pcie 5f000000.pcie:   MEM 0x60000000..0x6fefffff -> 0x60000000
[    3.017619] imx6q-pcie 5f000000.pcie: Link up
[    3.122020] imx6q-pcie 5f000000.pcie: Link up
[    3.126450] imx6q-pcie 5f000000.pcie: Link up, Gen2
[    3.131442] imx6q-pcie 5f000000.pcie: PCI host bridge to bus 0000:00
[    3.137812] pci_bus 0000:00: root bus resource [bus 00-ff]
[    3.143315] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    3.158556] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fefffff]
[    3.172157] pci 0000:00:00.0: [1957:0000] type 01 class 0x060400
[    3.183204] imx6q-pcie 5f010000.pcie: No cache used with register defaults set!
[    3.183947] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[    3.186567] imx6q-pcie 5f010000.pcie: PCIe PLL locked after 0 us.
[    3.213296] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x00ffffff pref]
[    3.228422] pci 0000:00:00.0: supports D1 D2
[    3.232705] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[    3.236174] pci 0000:01:00.0: [10b5:8605] type 01 class 0x060400
[    3.247835] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff]
[    3.257172] pci 0000:01:00.0: supports D1 D2
[    3.265075] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.281901] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    3.290327] pci 0000:02:01.0: [10b5:8605] type 01 class 0x060400
[    3.297009] pci 0000:02:01.0: supports D1 D2
[    3.301290] pci 0000:02:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.308367] pci 0000:02:02.0: [10b5:8605] type 01 class 0x060400
[    3.314788] pci 0000:02:02.0: supports D1 D2
[    3.319083] pci 0000:02:02.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.326103] pci 0000:02:03.0: [10b5:8605] type 01 class 0x060400
[    3.332528] pci 0000:02:03.0: supports D1 D2
[    3.336807] pci 0000:02:03.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.346545] pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    3.354587] pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    3.362619] pci 0000:02:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    3.373818] pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
[    3.383594] pci_bus 0000:04: busn_res: [bus 04-ff] end is updated to 04
[    3.390401] pci 0000:05:00.0: [1d6a:11b1] type 00 class 0x020000
[    3.396529] pci 0000:05:00.0: reg 0x10: [mem 0x00000000-0x0000ffff 64bit]
[    3.397407] imx6q-pcie 5f010000.pcie: host bridge /bus@5f000000/pcie@0x5f010000 ranges:
[    3.403366] pci 0000:05:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
[    3.418220] pci 0000:05:00.0: reg 0x20: [mem 0x00000000-0x003fffff 64bit]
[    3.424917] imx6q-pcie 5f010000.pcie:    IO 0x7ff80000..0x7ff8ffff -> 0x00000000
[    3.431477] pci 0000:05:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
[    3.438938] imx6q-pcie 5f010000.pcie:   MEM 0x70000000..0x7fefffff -> 0x70000000
[    3.445613] pci 0000:05:00.0: enabling Extended Tags
[    3.458237] pci 0000:05:00.0: supports D1 D2
[    3.462516] pci 0000:05:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.469299] pci 0000:05:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0000:00:00.0 (capable of 7.876 Gb/s with 8 GT/s x1 link)
[    3.486248] pci_bus 0000:05: busn_res: [bus 05-ff] end is updated to 05
[    3.492881] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 05
[    3.499541] pci 0000:00:00.0: BAR 0: assigned [mem 0x60000000-0x60ffffff]
[    3.506339] pci 0000:00:00.0: BAR 6: assigned [mem 0x61000000-0x61ffffff pref]
[    3.513584] pci 0000:00:00.0: BAR 14: assigned [mem 0x62000000-0x628fffff]
[    3.520471] pci 0000:00:00.0: BAR 15: assigned [mem 0x62900000-0x62afffff 64bit pref]
[    3.528317] pci 0000:00:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    3.534507] pci 0000:01:00.0: BAR 14: assigned [mem 0x62000000-0x627fffff]
[    3.541404] pci 0000:01:00.0: BAR 15: assigned [mem 0x62900000-0x62afffff 64bit pref]
[    3.549247] pci 0000:01:00.0: BAR 0: assigned [mem 0x62800000-0x62803fff]
[    3.556053] pci 0000:01:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    3.562244] pci 0000:02:03.0: BAR 14: assigned [mem 0x62000000-0x625fffff]
[    3.569140] pci 0000:02:01.0: BAR 14: assigned [mem 0x62600000-0x627fffff]
[    3.576018] pci 0000:02:01.0: BAR 15: assigned [mem 0x62900000-0x62afffff 64bit pref]
[    3.583867] pci 0000:02:01.0: BAR 13: assigned [io  0x1000-0x1fff]
[    3.590058] pci 0000:02:01.0: PCI bridge to [bus 03]
[    3.595038] pci 0000:02:01.0:   bridge window [io  0x1000-0x1fff]
[    3.601154] pci 0000:02:01.0:   bridge window [mem 0x62600000-0x627fffff]
[    3.607964] pci 0000:02:01.0:   bridge window [mem 0x62900000-0x62afffff 64bit pref]
[    3.619496] pci 0000:02:02.0: PCI bridge to [bus 04]
[    3.628532] pci 0000:05:00.0: BAR 4: assigned [mem 0x62000000-0x623fffff 64bit]
[    3.635885] pci 0000:05:00.0: BAR 6: assigned [mem 0x62400000-0x6243ffff pref]
[    3.643116] pci 0000:05:00.0: BAR 0: assigned [mem 0x62440000-0x6244ffff 64bit]
[    3.650472] pci 0000:05:00.0: BAR 2: assigned [mem 0x62450000-0x62450fff 64bit]
[    3.657817] pci 0000:02:03.0: PCI bridge to [bus 05]
[    3.662806] pci 0000:02:03.0:   bridge window [mem 0x62000000-0x625fffff]
[    3.669623] pci 0000:01:00.0: PCI bridge to [bus 02-05]
[    3.674868] pci 0000:01:00.0:   bridge window [io  0x1000-0x1fff]
[    3.680980] pci 0000:01:00.0:   bridge window [mem 0x62000000-0x627fffff]
[    3.687785] pci 0000:01:00.0:   bridge window [mem 0x62900000-0x62afffff 64bit pref]
[    3.695553] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    3.700795] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    3.706895] pci 0000:00:00.0:   bridge window [mem 0x62000000-0x628fffff]
[    3.713703] pci 0000:00:00.0:   bridge window [mem 0x62900000-0x62afffff 64bit pref]
[    3.721714] pcieport 0000:00:00.0: PME: Signaling with IRQ 568
[    3.727764] pcieport 0000:00:00.0: AER: enabled with IRQ 568
[    3.733612] pcieport 0000:01:00.0: enabling device (0000 -> 0003)
[    3.740366] pcieport 0000:02:01.0: enabling device (0000 -> 0003)
[    3.746868] pcieport 0000:02:01.0: pciehp: Slot #1 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl- LLActRep+
[    3.760533] pcieport 0000:02:03.0: enabling device (0000 -> 0002)
[    4.453557] imx6q-pcie 5f010000.pcie: Phy link never came up
[    4.459337] imx6q-pcie 5f010000.pcie: failed to initialize host
[    4.465331] imx6q-pcie 5f010000.pcie: unable to add pcie port.

Accepted connection from 172.25.1.12, port 34015
[  5] local 172.25.1.10 port 8008 connected to 172.25.1.12 port 39469
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   169 MBytes  1.41 Gbits/sec
[  5]   1.00-2.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   2.00-3.00   sec   189 MBytes  1.59 Gbits/sec
[  5]   3.00-4.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   4.00-5.00   sec   186 MBytes  1.56 Gbits/sec
[  5]   5.00-6.00   sec   188 MBytes  1.58 Gbits/sec
[  5]   6.00-7.00   sec   189 MBytes  1.58 Gbits/sec
[  5]   7.00-8.00   sec   187 MBytes  1.57 Gbits/sec
[  5]   8.00-9.00   sec   189 MBytes  1.59 Gbits/sec
[  5]   9.00-10.00  sec   187 MBytes  1.57 Gbits/sec
[  5]  10.00-10.00  sec  99.8 KBytes  1.41 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.81 GBytes  1.56 Gbits/sec                  receiver
-----------------------------------------------------------
Accepted connection from 172.25.1.12, port 60077
[  5] local 172.25.1.10 port 8008 connected to 172.25.1.12 port 58833
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   178 MBytes  1.49 Gbits/sec
[  5]   1.00-2.00   sec   187 MBytes  1.57 Gbits/sec
[  5]   2.00-3.00   sec   189 MBytes  1.58 Gbits/sec
[  5]   3.00-4.00   sec   187 MBytes  1.57 Gbits/sec
[  5]   4.00-5.00   sec   190 MBytes  1.59 Gbits/sec
[  5]   5.00-6.00   sec   190 MBytes  1.59 Gbits/sec
[  5]   6.00-7.00   sec   190 MBytes  1.60 Gbits/sec
[  5]   7.00-8.00   sec   187 MBytes  1.57 Gbits/sec
[  5]   8.00-9.00   sec   191 MBytes  1.60 Gbits/sec
[  5]   9.00-10.00  sec   189 MBytes  1.59 Gbits/sec
[  5]  10.00-10.00  sec   128 KBytes  1.25 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.83 GBytes  1.58 Gbits/sec                  receiver

Accepted connection from 172.25.1.12, port 38581
[  5] local 172.25.1.10 port 8008 connected to 172.25.1.12 port 40519
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   159 MBytes  1.33 Gbits/sec
[  5]   1.00-2.00   sec   169 MBytes  1.42 Gbits/sec
[  5]   2.00-3.00   sec   169 MBytes  1.42 Gbits/sec
[  5]   3.00-4.00   sec   168 MBytes  1.40 Gbits/sec
[  5]   4.00-5.00   sec   168 MBytes  1.42 Gbits/sec
[  5]   5.00-6.00   sec   169 MBytes  1.42 Gbits/sec
[  5]   6.00-7.00   sec   170 MBytes  1.43 Gbits/sec
[  5]   7.00-8.00   sec   168 MBytes  1.41 Gbits/sec
[  5]   8.00-9.00   sec   167 MBytes  1.40 Gbits/sec
[  5]   9.00-10.00  sec   170 MBytes  1.43 Gbits/sec
[  5]  10.00-10.00  sec   413 KBytes  1.40 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.64 GBytes  1.41 Gbits/sec                  receiver
-----------------------------------------------------------
Accepted connection from 172.25.1.12, port 43549
[  5] local 172.25.1.10 port 8008 connected to 172.25.1.12 port 42209
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   160 MBytes  1.34 Gbits/sec
[  5]   1.00-2.00   sec   168 MBytes  1.41 Gbits/sec
[  5]   2.00-3.00   sec   171 MBytes  1.43 Gbits/sec
[  5]   3.00-4.00   sec   168 MBytes  1.41 Gbits/sec
[  5]   4.00-5.00   sec   169 MBytes  1.42 Gbits/sec
[  5]   5.00-6.00   sec   170 MBytes  1.43 Gbits/sec
[  5]   6.00-7.00   sec   169 MBytes  1.42 Gbits/sec
[  5]   7.00-8.00   sec   168 MBytes  1.41 Gbits/sec
[  5]   8.00-9.00   sec   166 MBytes  1.39 Gbits/sec
[  5]   9.00-10.00  sec   169 MBytes  1.42 Gbits/sec
[  5]  10.00-10.00  sec   358 KBytes  1.33 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.64 GBytes  1.41 Gbits/sec                  receiver
  • Gen 3
    Even though its set to Gen3 the interface negotiated Gen2, as per specification.
root@apalis-imx8-07307404:~# dmesg | grep pci 
[    0.276348] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    1.785318] ehci-pci: EHCI PCI platform driver
[    2.597894] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    2.607266] imx6q-pcie 5f000000.pcie: pcie_ext clock source missing or invalid
[    2.615913] imx6q-pcie 5f010000.pcie: pcie_ext clock source missing or invalid
[    3.238589] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    3.239729] imx6q-pcie 5f010000.pcie: No cache used with register defaults set!
[    3.243453] imx6q-pcie 5f010000.pcie: PCIe PLL locked after 0 us.
[    3.248244] imx6q-pcie 5f000000.pcie: No cache used with register defaults set!
[    3.379219] imx6q-pcie 5f000000.pcie: PCIe PLL locked after 0 us.
[    3.457260] imx6q-pcie 5f010000.pcie: host bridge /bus@5f000000/pcie@0x5f010000 ranges:
[    3.465364] imx6q-pcie 5f010000.pcie:    IO 0x7ff80000..0x7ff8ffff -> 0x00000000
[    3.473078] imx6q-pcie 5f010000.pcie:   MEM 0x70000000..0x7fefffff -> 0x70000000
[    3.601221] imx6q-pcie 5f000000.pcie: host bridge /bus@5f000000/pcie@0x5f000000 ranges:
[    3.609312] imx6q-pcie 5f000000.pcie:    IO 0x6ff80000..0x6ff8ffff -> 0x00000000
[    3.617004] imx6q-pcie 5f000000.pcie:   MEM 0x60000000..0x6fefffff -> 0x60000000
[    3.724577] imx6q-pcie 5f000000.pcie: Link up
[    3.829023] imx6q-pcie 5f000000.pcie: Link up
[    3.833463] imx6q-pcie 5f000000.pcie: Link up, Gen2
[    3.838589] imx6q-pcie 5f000000.pcie: PCI host bridge to bus 0000:00
[    3.845035] pci_bus 0000:00: root bus resource [bus 00-ff]
[    3.850773] pci_bus 0000:00: root bus resource [io  0x10000-0x1ffff] (bus address [0x0000-0xffff])
[    3.859767] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fefffff]
[    3.866682] pci 0000:00:00.0: [1957:0000] type 01 class 0x060400
[    3.872741] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[    3.879038] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x00ffffff pref]
[    3.885809] pci 0000:00:00.0: supports D1 D2
[    3.890098] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[    3.902709] pci 0000:01:00.0: [10b5:8605] type 01 class 0x060400
[    3.908831] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff]
[    3.915441] pci 0000:01:00.0: supports D1 D2
[    3.919741] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    4.020483] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    4.029193] pci 0000:02:01.0: [10b5:8605] type 01 class 0x060400
[    4.035616] pci 0000:02:01.0: supports D1 D2
[    4.039918] pci 0000:02:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[    4.047072] pci 0000:02:02.0: [10b5:8605] type 01 class 0x060400
[    4.053509] pci 0000:02:02.0: supports D1 D2
[    4.057789] pci 0000:02:02.0: PME# supported from D0 D1 D2 D3hot D3cold
[    4.064929] pci 0000:02:03.0: [10b5:8605] type 01 class 0x060400
[    4.074793] pci 0000:02:03.0: supports D1 D2
[    4.082898] pci 0000:02:03.0: PME# supported from D0 D1 D2 D3hot D3cold
[    4.095020] pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    4.103079] pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    4.111119] pci 0000:02:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    4.124944] pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
[    4.137378] pci_bus 0000:04: busn_res: [bus 04-ff] end is updated to 04
[    4.144232] pci 0000:05:00.0: [1d6a:11b1] type 00 class 0x020000
[    4.150362] pci 0000:05:00.0: reg 0x10: [mem 0x00000000-0x0000ffff 64bit]
[    4.157203] pci 0000:05:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
[    4.164040] pci 0000:05:00.0: reg 0x20: [mem 0x00000000-0x003fffff 64bit]
[    4.170873] pci 0000:05:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
[    4.177616] pci 0000:05:00.0: enabling Extended Tags
[    4.182815] pci 0000:05:00.0: supports D1 D2
[    4.187106] pci 0000:05:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    4.193894] pci 0000:05:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0000:00:00.0 (capable of 7.876 Gb/s with 8 GT/s x1 link)
[    4.213473] pci_bus 0000:05: busn_res: [bus 05-ff] end is updated to 05
[    4.220133] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 05
[    4.232400] pci 0000:00:00.0: bridge configuration invalid ([bus 00-ff]), reconfiguring
[    4.268800] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 05
[    4.275449] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 05
[    4.282112] pci 0000:00:00.0: BAR 0: assigned [mem 0x60000000-0x60ffffff]
[    4.288933] pci 0000:00:00.0: BAR 6: assigned [mem 0x61000000-0x61ffffff pref]
[    4.296179] pci 0000:00:00.0: BAR 14: assigned [mem 0x62000000-0x628fffff]
[    4.303089] pci 0000:00:00.0: BAR 15: assigned [mem 0x62900000-0x62afffff 64bit pref]
[    4.310939] pci 0000:00:00.0: BAR 13: assigned [io  0x10000-0x10fff]
[    4.317331] pci 0000:01:00.0: BAR 14: assigned [mem 0x62000000-0x627fffff]
[    4.324221] pci 0000:01:00.0: BAR 15: assigned [mem 0x62900000-0x62afffff 64bit pref]
[    4.332087] pci 0000:01:00.0: BAR 0: assigned [mem 0x62800000-0x62803fff]
[    4.338902] pci 0000:01:00.0: BAR 13: assigned [io  0x10000-0x10fff]
[    4.345284] pci 0000:02:03.0: BAR 14: assigned [mem 0x62000000-0x625fffff]
[    4.352177] pci 0000:02:01.0: BAR 14: assigned [mem 0x62600000-0x627fffff]
[    4.359074] pci 0000:02:01.0: BAR 15: assigned [mem 0x62900000-0x62afffff 64bit pref]
[    4.366921] pci 0000:02:01.0: BAR 13: assigned [io  0x10000-0x10fff]
[    4.373296] pci 0000:02:01.0: PCI bridge to [bus 03]
[    4.378276] pci 0000:02:01.0:   bridge window [io  0x10000-0x10fff]
[    4.384584] pci 0000:02:01.0:   bridge window [mem 0x62600000-0x627fffff]
[    4.391401] pci 0000:02:01.0:   bridge window [mem 0x62900000-0x62afffff 64bit pref]
[    4.399187] pci 0000:02:02.0: PCI bridge to [bus 04]
[    4.404203] pci 0000:05:00.0: BAR 4: assigned [mem 0x62000000-0x623fffff 64bit]
[    4.411570] pci 0000:05:00.0: BAR 6: assigned [mem 0x62400000-0x6243ffff pref]
[    4.418813] pci 0000:05:00.0: BAR 0: assigned [mem 0x62440000-0x6244ffff 64bit]
[    4.426183] pci 0000:05:00.0: BAR 2: assigned [mem 0x62450000-0x62450fff 64bit]
[    4.433540] pci 0000:02:03.0: PCI bridge to [bus 05]
[    4.438540] pci 0000:02:03.0:   bridge window [mem 0x62000000-0x625fffff]
[    4.445360] pci 0000:01:00.0: PCI bridge to [bus 02-05]
[    4.450618] pci 0000:01:00.0:   bridge window [io  0x10000-0x10fff]
[    4.456913] pci 0000:01:00.0:   bridge window [mem 0x62000000-0x627fffff]
[    4.457192] imx6q-pcie 5f010000.pcie: Phy link never came up
[    4.463739] pci 0000:01:00.0:   bridge window [mem 0x62900000-0x62afffff 64bit pref]
[    4.469461] imx6q-pcie 5f010000.pcie: failed to initialize host
[    4.477177] pci 0000:00:00.0: PCI bridge to [bus 01-05]
[    4.483116] imx6q-pcie 5f010000.pcie: unable to add pcie port.
[    4.488343] pci 0000:00:00.0:   bridge window [io  0x10000-0x10fff]
[    4.488350] pci 0000:00:00.0:   bridge window [mem 0x62000000-0x628fffff]
[    4.507300] pci 0000:00:00.0:   bridge window [mem 0x62900000-0x62afffff 64bit pref]
[    4.515368] pcieport 0000:00:00.0: enabling device (0000 -> 0003)
[    4.521648] pcieport 0000:00:00.0: PME: Signaling with IRQ 568
[    4.527839] pcieport 0000:00:00.0: AER: enabled with IRQ 568
[    4.533766] pcieport 0000:01:00.0: enabling device (0000 -> 0003)
[    4.540689] pcieport 0000:02:01.0: enabling device (0000 -> 0003)
[    4.547338] pcieport 0000:02:01.0: pciehp: Slot #1 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl- LLActRep+
[    4.561310] pcieport 0000:02:03.0: enabling device (0000 -> 0002)

I believe achieving Gen2(4Gb/s) is enough. The problem is that the practical bandwith is in no way close to theoretical one.

Best regards,
Aleš

11

Hi @Ales !

Sorry for letting you hanging for so long.

I got back to your issue and asked internally for help.

I will let you know as soon as I have any news.

Do you perhaps have any updates about it since your last message?

Best regards,

12

Hi @henrique.tx !

Tried few more pcie startup args but it didn’t improve anything. Had some Emulex x8 cards which do work on pcie x1 on desktop unfortunately didn’t want to communicate with module, lspci didn’t show anything.

And the solution… sorry it didn’t come to my mind before but increasing the number of connection for iperf definetly helps. Found that 4 connections gave highest stable speed with some high peaks. Going more connections didn’t do any better.

Now I am just facing the instability of module, its freezing alot during speed tests. I am unable to narrow it down as it seems random.

Got greeted by this message when starting the test.

Broadcast message from systemd-journald@apalis-imx8-07307404 (Thu 2023-01-19 07:01:21 UTC):

kernel[485]: [   50.479846] Internal error: Oops: 96000005 [#1] PREEMPT SMP

I am guessing its because its pushing max allowed speed of hardware. Tried limiting max bandwith with -b flag in iperf3 but that didn’t limit it. Using 3 connections instead of 4 did improve stability, yet to see a crash.

Test with 4 connections got 3.1 Gb/s, 1 CPU core at 97% and other at 30%.

root@apalis-imx8-07307404:~# iperf3 -B 172.25.1.12 -p 8008 -c 172.25.1.10 -t 20 -P 4
Connecting to host 172.25.1.10, port 8008
[  5] local 172.25.1.12 port 36101 connected to 172.25.1.10 port 8008
[  7] local 172.25.1.12 port 55813 connected to 172.25.1.10 port 8008
[  9] local 172.25.1.12 port 49535 connected to 172.25.1.10 port 8008
[ 11] local 172.25.1.12 port 32921 connected to 172.25.1.10 port 8008
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  64.8 MBytes   543 Mbits/sec    0    372 KBytes       
[  7]   0.00-1.00   sec   111 MBytes   930 Mbits/sec   32   1.16 MBytes       
[  9]   0.00-1.00   sec  68.1 MBytes   571 Mbits/sec    0    469 KBytes       
[ 11]   0.00-1.00   sec   111 MBytes   932 Mbits/sec  102   1.24 MBytes       
[SUM]   0.00-1.00   sec   355 MBytes  2.98 Gbits/sec  134             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  64.5 MBytes   541 Mbits/sec    0    372 KBytes       
[  7]   1.00-2.00   sec   121 MBytes  1.02 Gbits/sec    0   1.22 MBytes       
[  9]   1.00-2.00   sec  66.1 MBytes   555 Mbits/sec    0    469 KBytes       
[ 11]   1.00-2.00   sec   122 MBytes  1.03 Gbits/sec    0   1.30 MBytes       
[SUM]   1.00-2.00   sec   374 MBytes  3.14 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  61.6 MBytes   517 Mbits/sec    0    372 KBytes       
[  7]   2.00-3.00   sec   125 MBytes  1.05 Gbits/sec    0   1.22 MBytes       
[  9]   2.00-3.00   sec  63.1 MBytes   529 Mbits/sec    0    493 KBytes       
[ 11]   2.00-3.00   sec   124 MBytes  1.04 Gbits/sec    0   1.30 MBytes       
[SUM]   2.00-3.00   sec   373 MBytes  3.13 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  62.2 MBytes   522 Mbits/sec    0    372 KBytes       
[  7]   3.00-4.00   sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]   3.00-4.00   sec  62.1 MBytes   521 Mbits/sec    0    499 KBytes       
[ 11]   3.00-4.00   sec   125 MBytes  1.05 Gbits/sec    0   1.30 MBytes       
[SUM]   3.00-4.00   sec   373 MBytes  3.13 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  61.1 MBytes   513 Mbits/sec    0    389 KBytes       
[  7]   4.00-5.00   sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]   4.00-5.00   sec  61.9 MBytes   520 Mbits/sec    0    499 KBytes       
[ 11]   4.00-5.00   sec   124 MBytes  1.04 Gbits/sec    0   1.30 MBytes       
[SUM]   4.00-5.00   sec   370 MBytes  3.11 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  60.5 MBytes   507 Mbits/sec    0    415 KBytes       
[  7]   5.00-6.00   sec   125 MBytes  1.05 Gbits/sec    0   1.22 MBytes       
[  9]   5.00-6.00   sec  64.8 MBytes   543 Mbits/sec    0    499 KBytes       
[ 11]   5.00-6.00   sec   124 MBytes  1.04 Gbits/sec    0   1.30 MBytes       
[SUM]   5.00-6.00   sec   374 MBytes  3.13 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  59.6 MBytes   500 Mbits/sec    0    415 KBytes       
[  7]   6.00-7.00   sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]   6.00-7.00   sec  64.4 MBytes   541 Mbits/sec    0    530 KBytes       
[ 11]   6.00-7.00   sec   124 MBytes  1.04 Gbits/sec    0   1.30 MBytes       
[SUM]   6.00-7.00   sec   371 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  59.7 MBytes   500 Mbits/sec    0    415 KBytes       
[  7]   7.00-8.00   sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]   7.00-8.00   sec  64.4 MBytes   540 Mbits/sec    0    530 KBytes       
[ 11]   7.00-8.00   sec   125 MBytes  1.05 Gbits/sec    0   1.30 MBytes       
[SUM]   7.00-8.00   sec   373 MBytes  3.13 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  58.6 MBytes   492 Mbits/sec    0    415 KBytes       
[  7]   8.00-9.00   sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]   8.00-9.00   sec  65.4 MBytes   549 Mbits/sec    0    530 KBytes       
[ 11]   8.00-9.00   sec   124 MBytes  1.04 Gbits/sec    0   1.30 MBytes       
[SUM]   8.00-9.00   sec   372 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  59.7 MBytes   500 Mbits/sec    0    415 KBytes       
[  7]   9.00-10.00  sec   125 MBytes  1.05 Gbits/sec    0   1.22 MBytes       
[  9]   9.00-10.00  sec  64.8 MBytes   543 Mbits/sec    0    530 KBytes       
[ 11]   9.00-10.00  sec   124 MBytes  1.04 Gbits/sec    0   1.30 MBytes       
[SUM]   9.00-10.00  sec   373 MBytes  3.13 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  10.00-11.00  sec  58.6 MBytes   493 Mbits/sec    0    415 KBytes       
[  7]  10.00-11.00  sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]  10.00-11.00  sec  65.1 MBytes   547 Mbits/sec    0    530 KBytes       
[ 11]  10.00-11.00  sec   124 MBytes  1.04 Gbits/sec    0   1.35 MBytes       
[SUM]  10.00-11.00  sec   371 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  11.00-12.00  sec  59.2 MBytes   496 Mbits/sec    0    415 KBytes       
[  7]  11.00-12.00  sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]  11.00-12.00  sec  65.0 MBytes   545 Mbits/sec    0    530 KBytes       
[ 11]  11.00-12.00  sec   125 MBytes  1.05 Gbits/sec    0   1.35 MBytes       
[SUM]  11.00-12.00  sec   373 MBytes  3.13 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  12.00-13.00  sec  58.1 MBytes   488 Mbits/sec    0    415 KBytes       
[  7]  12.00-13.00  sec   125 MBytes  1.05 Gbits/sec    0   1.22 MBytes       
[  9]  12.00-13.00  sec  65.9 MBytes   552 Mbits/sec    0    530 KBytes       
[ 11]  12.00-13.00  sec   122 MBytes  1.03 Gbits/sec    0   1.59 MBytes       
[SUM]  12.00-13.00  sec   371 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  13.00-14.00  sec  58.1 MBytes   488 Mbits/sec    0    415 KBytes       
[  7]  13.00-14.00  sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]  13.00-14.00  sec  66.2 MBytes   556 Mbits/sec    0    530 KBytes       
[ 11]  13.00-14.00  sec   124 MBytes  1.04 Gbits/sec    0   1.59 MBytes       
[SUM]  13.00-14.00  sec   372 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  14.00-15.00  sec  58.6 MBytes   492 Mbits/sec    0    415 KBytes       
[  7]  14.00-15.00  sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]  14.00-15.00  sec  65.1 MBytes   546 Mbits/sec    0    530 KBytes       
[ 11]  14.00-15.00  sec   124 MBytes  1.04 Gbits/sec    0   1.59 MBytes       
[SUM]  14.00-15.00  sec   371 MBytes  3.11 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  15.00-16.00  sec  58.4 MBytes   490 Mbits/sec    0    415 KBytes       
[  7]  15.00-16.00  sec   125 MBytes  1.05 Gbits/sec    0   1.22 MBytes       
[  9]  15.00-16.00  sec  66.2 MBytes   555 Mbits/sec    0    530 KBytes       
[ 11]  15.00-16.00  sec   124 MBytes  1.04 Gbits/sec    0   1.59 MBytes       
[SUM]  15.00-16.00  sec   373 MBytes  3.13 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  16.00-17.00  sec  58.8 MBytes   493 Mbits/sec    0    415 KBytes       
[  7]  16.00-17.00  sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]  16.00-17.00  sec  65.0 MBytes   545 Mbits/sec    0    530 KBytes       
[ 11]  16.00-17.00  sec   125 MBytes  1.05 Gbits/sec    0   1.59 MBytes       
[SUM]  16.00-17.00  sec   373 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  17.00-18.00  sec  58.6 MBytes   492 Mbits/sec    0    415 KBytes       
[  7]  17.00-18.00  sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]  17.00-18.00  sec  65.7 MBytes   551 Mbits/sec    0    530 KBytes       
[ 11]  17.00-18.00  sec   124 MBytes  1.04 Gbits/sec    0   1.59 MBytes       
[SUM]  17.00-18.00  sec   372 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  18.00-19.00  sec  58.8 MBytes   493 Mbits/sec    0    415 KBytes       
[  7]  18.00-19.00  sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]  18.00-19.00  sec  66.2 MBytes   555 Mbits/sec    0    530 KBytes       
[ 11]  18.00-19.00  sec   124 MBytes  1.04 Gbits/sec    0   1.59 MBytes       
[SUM]  18.00-19.00  sec   372 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  19.00-20.00  sec  58.3 MBytes   489 Mbits/sec    0    415 KBytes       
[  7]  19.00-20.00  sec   125 MBytes  1.05 Gbits/sec    0   1.22 MBytes       
[  9]  19.00-20.00  sec  64.8 MBytes   543 Mbits/sec    0    530 KBytes       
[ 11]  19.00-20.00  sec   124 MBytes  1.04 Gbits/sec    0   1.59 MBytes       
[SUM]  19.00-20.00  sec   372 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  1.17 GBytes   502 Mbits/sec    0             sender
[  5]   0.00-20.01  sec  1.17 GBytes   501 Mbits/sec                  receiver
[  7]   0.00-20.00  sec  2.41 GBytes  1.03 Gbits/sec   32             sender
[  7]   0.00-20.01  sec  2.41 GBytes  1.03 Gbits/sec                  receiver
[  9]   0.00-20.00  sec  1.27 GBytes   545 Mbits/sec    0             sender
[  9]   0.00-20.01  sec  1.27 GBytes   544 Mbits/sec                  receiver
[ 11]   0.00-20.00  sec  2.41 GBytes  1.03 Gbits/sec  102             sender
[ 11]   0.00-20.01  sec  2.40 GBytes  1.03 Gbits/sec                  receiver
[SUM]   0.00-20.00  sec  7.26 GBytes  3.12 Gbits/sec  134             sender
[SUM]   0.00-20.01  sec  7.25 GBytes  3.11 Gbits/sec                  receiver

Test with 3 connections, 1 CPU core at 87% other at 25%. Here i noticed that most of the tim eit will run on 2.4Gb/s but sometimes all 3 connections give 1Gb/s and it runs at 3Gb/s.

root@apalis-imx8-07307404:~# iperf3 -B 172.25.1.12 -p 8008 -c 172.25.1.10 -t 20 -P 3
Connecting to host 172.25.1.10, port 8008
[  5] local 172.25.1.12 port 57561 connected to 172.25.1.10 port 8008
[  7] local 172.25.1.12 port 47465 connected to 172.25.1.10 port 8008
[  9] local 172.25.1.12 port 59161 connected to 172.25.1.10 port 8008
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  94.2 MBytes   790 Mbits/sec   83    903 KBytes       
[  7]   0.00-1.00   sec  38.3 MBytes   321 Mbits/sec    1    365 KBytes       
[  9]   0.00-1.00   sec   135 MBytes  1.13 Gbits/sec   47   1.15 MBytes       
[SUM]   0.00-1.00   sec   268 MBytes  2.25 Gbits/sec  131             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  96.2 MBytes   807 Mbits/sec    0    998 KBytes       
[  7]   1.00-2.00   sec  39.1 MBytes   328 Mbits/sec    0    439 KBytes       
[  9]   1.00-2.00   sec   138 MBytes  1.15 Gbits/sec    0   1.26 MBytes       
[SUM]   1.00-2.00   sec   273 MBytes  2.29 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   104 MBytes   870 Mbits/sec    0   1.04 MBytes       
[  7]   2.00-3.00   sec  35.2 MBytes   295 Mbits/sec    2    375 KBytes       
[  9]   2.00-3.00   sec   141 MBytes  1.18 Gbits/sec    0   1.34 MBytes       
[SUM]   2.00-3.00   sec   280 MBytes  2.35 Gbits/sec    2             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   110 MBytes   923 Mbits/sec    0   1.09 MBytes       
[  7]   3.00-4.00   sec  31.3 MBytes   263 Mbits/sec   10    328 KBytes       
[  9]   3.00-4.00   sec   144 MBytes  1.21 Gbits/sec    0   1.41 MBytes       
[SUM]   3.00-4.00   sec   285 MBytes  2.39 Gbits/sec   10             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   111 MBytes   933 Mbits/sec    1    843 KBytes       
[  7]   4.00-5.00   sec  27.6 MBytes   231 Mbits/sec    6    297 KBytes       
[  9]   4.00-5.00   sec   142 MBytes  1.20 Gbits/sec    4   1.05 MBytes       
[SUM]   4.00-5.00   sec   281 MBytes  2.36 Gbits/sec   11             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   102 MBytes   860 Mbits/sec    0    951 KBytes       
[  7]   5.00-6.00   sec  37.2 MBytes   312 Mbits/sec    0    382 KBytes       
[  9]   5.00-6.00   sec   142 MBytes  1.20 Gbits/sec    0   1.14 MBytes       
[SUM]   5.00-6.00   sec   282 MBytes  2.37 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   102 MBytes   860 Mbits/sec    0   1.01 MBytes       
[  7]   6.00-7.00   sec  39.8 MBytes   334 Mbits/sec    2    319 KBytes       
[  9]   6.00-7.00   sec   146 MBytes  1.23 Gbits/sec    0   1.23 MBytes       
[SUM]   6.00-7.00   sec   289 MBytes  2.42 Gbits/sec    2             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   109 MBytes   912 Mbits/sec    0   1.07 MBytes       
[  7]   7.00-8.00   sec  33.1 MBytes   278 Mbits/sec    2    278 KBytes       
[  9]   7.00-8.00   sec   146 MBytes  1.23 Gbits/sec    0   1.32 MBytes       
[SUM]   7.00-8.00   sec   288 MBytes  2.42 Gbits/sec    2             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   115 MBytes   965 Mbits/sec    0   1.14 MBytes       
[  7]   8.00-9.00   sec  27.7 MBytes   232 Mbits/sec    2    250 KBytes       
[  9]   8.00-9.00   sec   146 MBytes  1.23 Gbits/sec    0   1.40 MBytes       
[SUM]   8.00-9.00   sec   289 MBytes  2.42 Gbits/sec    2             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   118 MBytes   986 Mbits/sec    1    874 KBytes       
[  7]   9.00-10.00  sec  23.8 MBytes   199 Mbits/sec    1    240 KBytes       
[  9]   9.00-10.00  sec   144 MBytes  1.21 Gbits/sec    1   1.07 MBytes       
[SUM]   9.00-10.00  sec   285 MBytes  2.39 Gbits/sec    3             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  10.00-11.00  sec   110 MBytes   923 Mbits/sec    0    984 KBytes       
[  7]  10.00-11.00  sec  32.3 MBytes   271 Mbits/sec    0    327 KBytes       
[  9]  10.00-11.00  sec   145 MBytes  1.22 Gbits/sec    0   1.18 MBytes       
[SUM]  10.00-11.00  sec   287 MBytes  2.41 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  11.00-12.00  sec   102 MBytes   859 Mbits/sec    3    749 KBytes       
[  7]  11.00-12.00  sec  36.0 MBytes   302 Mbits/sec    0    403 KBytes       
[  9]  11.00-12.00  sec   145 MBytes  1.22 Gbits/sec    0   1.27 MBytes       
[SUM]  11.00-12.00  sec   283 MBytes  2.38 Gbits/sec    3             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  12.00-13.00  sec  92.5 MBytes   776 Mbits/sec    0    837 KBytes       
[  7]  12.00-13.00  sec  50.9 MBytes   427 Mbits/sec    0    489 KBytes       
[  9]  12.00-13.00  sec   145 MBytes  1.22 Gbits/sec    0   1.33 MBytes       
[SUM]  12.00-13.00  sec   288 MBytes  2.42 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  13.00-14.00  sec  88.8 MBytes   744 Mbits/sec   14    750 KBytes       
[  7]  13.00-14.00  sec  51.6 MBytes   433 Mbits/sec    0    566 KBytes       
[  9]  13.00-14.00  sec   145 MBytes  1.22 Gbits/sec    0   1.39 MBytes       
[SUM]  13.00-14.00  sec   285 MBytes  2.39 Gbits/sec   14             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  14.00-15.00  sec  76.2 MBytes   640 Mbits/sec    0    731 KBytes       
[  7]  14.00-15.00  sec  66.1 MBytes   554 Mbits/sec    0    649 KBytes       
[  9]  14.00-15.00  sec   144 MBytes  1.21 Gbits/sec    1   1.05 MBytes       
[SUM]  14.00-15.00  sec   286 MBytes  2.40 Gbits/sec    1             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  15.00-16.00  sec  71.2 MBytes   598 Mbits/sec    7    572 KBytes       
[  7]  15.00-16.00  sec  71.2 MBytes   598 Mbits/sec    0    727 KBytes       
[  9]  15.00-16.00  sec   146 MBytes  1.23 Gbits/sec    0   1.15 MBytes       
[SUM]  15.00-16.00  sec   289 MBytes  2.42 Gbits/sec    7             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  16.00-17.00  sec  62.5 MBytes   524 Mbits/sec    0    653 KBytes       
[  7]  16.00-17.00  sec  78.8 MBytes   661 Mbits/sec    0    808 KBytes       
[  9]  16.00-17.00  sec   146 MBytes  1.23 Gbits/sec    0   1.24 MBytes       
[SUM]  16.00-17.00  sec   288 MBytes  2.41 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  17.00-18.00  sec  51.2 MBytes   430 Mbits/sec   14    526 KBytes       
[  7]  17.00-18.00  sec  88.8 MBytes   744 Mbits/sec    0    888 KBytes       
[  9]  17.00-18.00  sec   144 MBytes  1.21 Gbits/sec    0   1.32 MBytes       
[SUM]  17.00-18.00  sec   284 MBytes  2.38 Gbits/sec   14             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  18.00-19.00  sec  58.8 MBytes   493 Mbits/sec    0    606 KBytes       
[  7]  18.00-19.00  sec  82.5 MBytes   692 Mbits/sec   14    694 KBytes       
[  9]  18.00-19.00  sec   144 MBytes  1.21 Gbits/sec    0   1.40 MBytes       
[SUM]  18.00-19.00  sec   285 MBytes  2.39 Gbits/sec   14             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  19.00-20.00  sec  66.2 MBytes   556 Mbits/sec    0    684 KBytes       
[  7]  19.00-20.00  sec  75.0 MBytes   629 Mbits/sec    0    777 KBytes       
[  9]  19.00-20.00  sec   144 MBytes  1.21 Gbits/sec    5   1.07 MBytes       
[SUM]  19.00-20.00  sec   285 MBytes  2.39 Gbits/sec    5             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  1.80 GBytes   772 Mbits/sec  123             sender
[  5]   0.00-20.01  sec  1.80 GBytes   771 Mbits/sec                  receiver
[  7]   0.00-20.00  sec   966 MBytes   405 Mbits/sec   40             sender
[  7]   0.00-20.01  sec   963 MBytes   404 Mbits/sec                  receiver
[  9]   0.00-20.00  sec  2.81 GBytes  1.20 Gbits/sec   58             sender
[  9]   0.00-20.01  sec  2.80 GBytes  1.20 Gbits/sec                  receiver
[SUM]   0.00-20.00  sec  5.55 GBytes  2.38 Gbits/sec  221             sender
[SUM]   0.00-20.01  sec  5.54 GBytes  2.38 Gbits/sec                  receiver

Occasional 3Gb/s

root@apalis-imx8-07307404:~# iperf3 -B 172.25.1.12 -p 8008 -c 172.25.1.10 -t 20 -P 3
Connecting to host 172.25.1.10, port 8008
[  5] local 172.25.1.12 port 48073 connected to 172.25.1.10 port 8008
[  7] local 172.25.1.12 port 39815 connected to 172.25.1.10 port 8008
[  9] local 172.25.1.12 port 48615 connected to 172.25.1.10 port 8008
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   125 MBytes  1.04 Gbits/sec    0    599 KBytes       
[  7]   0.00-1.00   sec   125 MBytes  1.05 Gbits/sec    0    586 KBytes       
[  9]   0.00-1.00   sec   123 MBytes  1.03 Gbits/sec    0    557 KBytes       
[SUM]   0.00-1.00   sec   373 MBytes  3.13 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   125 MBytes  1.05 Gbits/sec    0    599 KBytes       
[  7]   1.00-2.00   sec   124 MBytes  1.04 Gbits/sec    0    586 KBytes       
[  9]   1.00-2.00   sec   124 MBytes  1.04 Gbits/sec    0    557 KBytes       
[SUM]   1.00-2.00   sec   372 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   125 MBytes  1.05 Gbits/sec    0    768 KBytes       
[  7]   2.00-3.00   sec   122 MBytes  1.03 Gbits/sec    0    674 KBytes       
[  9]   2.00-3.00   sec   126 MBytes  1.06 Gbits/sec    0    763 KBytes       
[SUM]   2.00-3.00   sec   374 MBytes  3.13 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   125 MBytes  1.05 Gbits/sec    0    887 KBytes       
[  7]   3.00-4.00   sec   122 MBytes  1.03 Gbits/sec    0    786 KBytes       
[  9]   3.00-4.00   sec   125 MBytes  1.05 Gbits/sec    0    932 KBytes       
[SUM]   3.00-4.00   sec   372 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   125 MBytes  1.05 Gbits/sec    0    887 KBytes       
[  7]   4.00-5.00   sec   124 MBytes  1.04 Gbits/sec    0    786 KBytes       
[  9]   4.00-5.00   sec   124 MBytes  1.04 Gbits/sec    0    932 KBytes       
[SUM]   4.00-5.00   sec   372 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   125 MBytes  1.05 Gbits/sec    0   1.03 MBytes       
[  7]   5.00-6.00   sec   126 MBytes  1.06 Gbits/sec    0   1017 KBytes       
[  9]   5.00-6.00   sec   121 MBytes  1.02 Gbits/sec    0   1.06 MBytes       
[SUM]   5.00-6.00   sec   372 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   125 MBytes  1.05 Gbits/sec    0   1.03 MBytes       
[  7]   6.00-7.00   sec   124 MBytes  1.04 Gbits/sec    0   1017 KBytes       
[  9]   6.00-7.00   sec   124 MBytes  1.04 Gbits/sec    0   1.06 MBytes       
[SUM]   6.00-7.00   sec   372 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   124 MBytes  1.04 Gbits/sec    0   1.03 MBytes       
[  7]   7.00-8.00   sec   124 MBytes  1.04 Gbits/sec    0   1017 KBytes       
[  9]   7.00-8.00   sec   124 MBytes  1.04 Gbits/sec    0   1.06 MBytes       
[SUM]   7.00-8.00   sec   371 MBytes  3.11 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   125 MBytes  1.05 Gbits/sec    0   1.03 MBytes       
[  7]   8.00-9.00   sec   125 MBytes  1.05 Gbits/sec    0   1017 KBytes       
[  9]   8.00-9.00   sec   124 MBytes  1.04 Gbits/sec    0   1.18 MBytes       
[SUM]   8.00-9.00   sec   374 MBytes  3.14 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   120 MBytes  1.01 Gbits/sec    0   1.17 MBytes       
[  7]   9.00-10.00  sec   125 MBytes  1.05 Gbits/sec    0   1.22 MBytes       
[  9]   9.00-10.00  sec   125 MBytes  1.05 Gbits/sec    0   1.40 MBytes       
[SUM]   9.00-10.00  sec   370 MBytes  3.10 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  10.00-11.00  sec   125 MBytes  1.05 Gbits/sec    0   1.17 MBytes       
[  7]  10.00-11.00  sec   125 MBytes  1.05 Gbits/sec    0   1.22 MBytes       
[  9]  10.00-11.00  sec   125 MBytes  1.05 Gbits/sec    0   1.40 MBytes       
[SUM]  10.00-11.00  sec   375 MBytes  3.14 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  11.00-12.00  sec   124 MBytes  1.04 Gbits/sec    0   1.17 MBytes       
[  7]  11.00-12.00  sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]  11.00-12.00  sec   124 MBytes  1.04 Gbits/sec    0   1.40 MBytes       
[SUM]  11.00-12.00  sec   371 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  12.00-13.00  sec   124 MBytes  1.04 Gbits/sec    0   1.17 MBytes       
[  7]  12.00-13.00  sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]  12.00-13.00  sec   124 MBytes  1.04 Gbits/sec    0   1.40 MBytes       
[SUM]  12.00-13.00  sec   371 MBytes  3.11 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  13.00-14.00  sec   125 MBytes  1.05 Gbits/sec    0   1.17 MBytes       
[  7]  13.00-14.00  sec   125 MBytes  1.05 Gbits/sec    0   1.22 MBytes       
[  9]  13.00-14.00  sec   125 MBytes  1.05 Gbits/sec    0   1.40 MBytes       
[SUM]  13.00-14.00  sec   375 MBytes  3.14 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  14.00-15.00  sec   124 MBytes  1.04 Gbits/sec    0   1.17 MBytes       
[  7]  14.00-15.00  sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]  14.00-15.00  sec   124 MBytes  1.04 Gbits/sec    0   1.40 MBytes       
[SUM]  14.00-15.00  sec   371 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  15.00-16.00  sec   124 MBytes  1.04 Gbits/sec    0   1.17 MBytes       
[  7]  15.00-16.00  sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]  15.00-16.00  sec   124 MBytes  1.04 Gbits/sec    0   1.40 MBytes       
[SUM]  15.00-16.00  sec   371 MBytes  3.11 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  16.00-17.00  sec   124 MBytes  1.04 Gbits/sec    0   1.17 MBytes       
[  7]  16.00-17.00  sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]  16.00-17.00  sec   125 MBytes  1.05 Gbits/sec    0   1.40 MBytes       
[SUM]  16.00-17.00  sec   372 MBytes  3.12 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  17.00-18.00  sec   125 MBytes  1.05 Gbits/sec    0   1.17 MBytes       
[  7]  17.00-18.00  sec   125 MBytes  1.05 Gbits/sec    0   1.22 MBytes       
[  9]  17.00-18.00  sec   124 MBytes  1.04 Gbits/sec    0   1.40 MBytes       
[SUM]  17.00-18.00  sec   374 MBytes  3.14 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  18.00-19.00  sec   124 MBytes  1.04 Gbits/sec    0   1.17 MBytes       
[  7]  18.00-19.00  sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]  18.00-19.00  sec   124 MBytes  1.04 Gbits/sec    0   1.40 MBytes       
[SUM]  18.00-19.00  sec   371 MBytes  3.11 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  19.00-20.00  sec   124 MBytes  1.04 Gbits/sec    0   1.17 MBytes       
[  7]  19.00-20.00  sec   124 MBytes  1.04 Gbits/sec    0   1.22 MBytes       
[  9]  19.00-20.00  sec   124 MBytes  1.04 Gbits/sec    0   1.40 MBytes       
[SUM]  19.00-20.00  sec   371 MBytes  3.11 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  2.43 GBytes  1.04 Gbits/sec    0             sender
[  5]   0.00-20.00  sec  2.42 GBytes  1.04 Gbits/sec                  receiver
[  7]   0.00-20.00  sec  2.42 GBytes  1.04 Gbits/sec    0             sender
[  7]   0.00-20.00  sec  2.42 GBytes  1.04 Gbits/sec                  receiver
[  9]   0.00-20.00  sec  2.42 GBytes  1.04 Gbits/sec    0             sender
[  9]   0.00-20.00  sec  2.42 GBytes  1.04 Gbits/sec                  receiver
[SUM]   0.00-20.00  sec  7.27 GBytes  3.12 Gbits/sec    0             sender
[SUM]   0.00-20.00  sec  7.27 GBytes  3.12 Gbits/sec                  receiver

2 connections were pushing 2.3Gb/s.

And just to make sure 1 connection pushed 1.4Gb/s as I had test before.

I will continue testing the stability but i had a continous test with 2 connections running for a good 30min now without a crash. 3 connections seem to crash on 3Gb/s but seem stable on 2.4Gb/s.
For now that is stable 300MB/s troughput.

Edit: Just as I typed that it was stable it crashed on 2 connections, nothing out of ordinary. Speed dropped with followed freeze up of the module requiring a hard reset.
Looks like its caused by consecutive runs. Google returned that it is a common issue in kernel. When i can allocate more time to this project will try updating the image to latest, if that doesnt fix it will try a older release.

Best regards,
Aleš

13

Hi @Ales !

Thanks for all the information in your feedback! This is great!

I see that you also marked this thread as solved. Thanks for this as well :slight_smile:

When you are back to this, you can create a new thread about this specific issue.

I can’t promise that we will be able to solve it, but we can try.

Some questions

  • Your results from your last message came from your custom carrier board or from Apalis Evaluation Board V1.1A that you have?
  • Did you manage to see Gen3 working? (e.g. advertised in kernel logs - dmesg)

Could you please help me to fill/correct this table with the summary of your results? It will be very helpful for others to have an overview of the results of your experiments :slight_smile:

(Once you fill/correct it in your post, I will edit this message and remove my table)

EDIT: incomplete table removed

Have a great day!

Best regards,

14

Hi @henrique.tx !

Results are from our custom carrier board.

Test was done on Gen2. See bellow for Gen3.

Summary of test results for Gen2:

# of Con BW per Connection (Gbs) Avg. BW (Gbs) CPU (%) Stability
1 1.4 1.4 60/20 No prolonged testing
2 1.1/1.1 2.4 80/20 Random kernel page allocation error on consecutive runs
3 0.9/0.3/1.2 2.4 80/20 Random kernel page allocation error on consecutive runs
3 1/1/1 3.1 90/30 Random kernel page allocation error on consecutive runs, sometimes at start of test
4 0.5/1/0.5/1 3.1 97/30 Highly unstable

Summary of test results for Gen3:
Gen3 on higher number of connection was even more unstable than Gen2, didn’t finish a single test with 5 retries.

# of Con BW per Connection (Gbs) Avg. BW (Gbs) CPU (%) Stability
1 1.48 1.48 55/20 No prolonged testing
2 1.3/1.3 2.7 85/30 Random kernel page allocation error on consecutive runs and first run
3 - - - Highly unstable
4 - - - Highly unstable

Best regards,
Aleš

15

Hi @Ales !

Thanks a lot for all the detailed information.

Best regards,

16

Hi @henrique.tx !

Tried with different kernel version but that didn’t help at the end it was a problem with used NIC.

NIC was from Aquantia which was acquired by Marvell in Sep of 2019. Since then the drivers were not updated and faulty for ARM platform. They seemed to work fine under windows/ubuntu.

Since then I replaced it with Intel X520 based 10G card (Ethernet controller: Intel Corporation 82599 10 Gigabit Network Connection (rev 01)). Sadly its only Gen2 x4 but it does work on Gen2 x1 with reported speed of 5GT/s (4Gb/s).

Stable speed measured with 4 connections over the course of a day was 2.9Gb/s with no crashes or lockups.

Card was attached to 1x slot by using a 16x to 1x adapter with external power delivery.

Best regards,
Aleš

1 Like
18

Hi @Ales !

That’s a good piece of information! It proves that the PCIe driver/firmware plays a major role here (not that anyone would doubt it).

Anyway, thanks again for all your debugging effort, @Ales!

This topic will certainly be helpful for other developers :slight_smile:

Best regards,