iMX8 PCIe Read/Down Speed

Hello Toradex team,

We are testing iMX8QM PCIe bandwith to see if its sufficient for our testing suite.
The problem we are trying so solve is connecting a >1GBe network card.

This is the continuation of imx8-pcie-link-only-working-on-gen1.
The difference from previous post is that we now have x2 lanes on PCIEA.

//apalis-imx8_pcie-gen3-x2.dts

/dts-v1/;
/plugin/;

#include <dt-bindings/soc/imx8_hsio.h>

&pciea {
    status = "okay";

    hsio-cfg = <PCIEAX2PCIEBX1>;
    num-lanes = <2>;

    fsl,max-link-speed = <3>;
};

&sata {
    status = "disabled";
};

The following tests are done on 2 seperate NICs.

(Intel) 10Gtek X520-10G-1S
(Aquantia) OWC OWCPCIE10GB

Intel card was connected using fibre and Aquantia card using CAT6 RJ45.

Results are captured using iperf3 average speeds after 10 second test with 5 parallel connections. Loopback is done by running the server and client on iMX8.

Test Case Intel Aquantia
Up 4.25Gb/s (iperf3 100%, CPU0 78%) 4.38 Gb/s (iperf3 100%, CPU0 70%)
Down 1.58Gb/s (iperf3 30%, CPU0 100%) 1.20Gb/s (iperf3 36%, CPU0 100%)
Loopback Up 6.52Gb/s (iperf3 85%, CPU0 50%) 6.46Gb/s (iperf3 84%, CPU0 50%)
Loopback Down 6.66Gb/s (iperf3 85%, CPU0 50%) 6.89Gb/s (iperf3 84%, CPU0 50%)
root@apalis-imx8-07307404:~# uname -a
Linux apalis-imx8-07307404 5.4.193-5.7.1-devel+git.f78299297185 #1 SMP PREEMPT Mon Jul 11 14:42:03 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
root@apalis-imx8-07307404:~# fw_printenv defargs
defargs=pci=pcie_bus_perf,nommconf,nomsi

Intel dmesg.

root@apalis-imx8-07307404:~# dmesg | grep pci
[    0.000000] Kernel command line: pci=pcie_bus_perf,nommconf,nomsi root=PARTUUID=98821cdc-02 ro rootwait
[    0.274715] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    1.790020] ehci-pci: EHCI PCI platform driver
[    2.577932] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    2.595697] imx6q-pcie 5f000000.pcie: No cache used with register defaults set!
[    2.726252] imx6q-pcie 5f000000.pcie: PCIe PLL locked after 0 us.
[    2.945171] imx6q-pcie 5f000000.pcie: host bridge /bus@5f000000/pcie@0x5f000000 ranges:
[    2.953227] imx6q-pcie 5f000000.pcie:    IO 0x6ff80000..0x6ff8ffff -> 0x00000000
[    2.960743] imx6q-pcie 5f000000.pcie:   MEM 0x60000000..0x6fefffff -> 0x60000000
[    3.068430] imx6q-pcie 5f000000.pcie: Link up
[    3.172881] imx6q-pcie 5f000000.pcie: Link up
[    3.177291] imx6q-pcie 5f000000.pcie: Link up, Gen2
[    3.182424] imx6q-pcie 5f000000.pcie: PCI host bridge to bus 0000:00
[    3.188863] pci_bus 0000:00: root bus resource [bus 00-ff]
[    3.194593] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    3.200805] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fefffff]
[    3.207728] pci 0000:00:00.0: [1957:0000] type 01 class 0x060400
[    3.213775] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[    3.220074] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x00ffffff pref]
[    3.226841] pci 0000:00:00.0: supports D1 D2
[    3.231128] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[    3.243698] pci 0000:01:00.0: [8086:1557] type 00 class 0x020000
[    3.249811] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0007ffff 64bit pref]
[    3.257080] pci 0000:01:00.0: reg 0x18: [io  0x0000-0x001f]
[    3.262722] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref]
[    3.269982] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[    3.276912] pci 0000:01:00.0: PME# supported from D0 D3hot
[    3.282500] pci 0000:01:00.0: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref]
[    3.289843] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR0 for 64 VFs)
[    3.300189] pci 0000:01:00.0: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref]
[    3.307523] pci 0000:01:00.0: VF(n) BAR3 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR3 for 64 VFs)
[    3.318483] pci 0000:01:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x2 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5 GT/s x8 link)
[    3.352699] pci 0000:00:00.0: BAR 0: assigned [mem 0x60000000-0x60ffffff]
[    3.359727] pci 0000:00:00.0: BAR 6: assigned [mem 0x61000000-0x61ffffff pref]
[    3.366987] pci 0000:00:00.0: BAR 14: assigned [mem 0x62000000-0x620fffff]
[    3.373908] pci 0000:00:00.0: BAR 15: assigned [mem 0x62100000-0x623fffff 64bit pref]
[    3.381959] pci 0000:00:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    3.388180] pci 0000:01:00.0: BAR 0: assigned [mem 0x62100000-0x6217ffff 64bit pref]
[    3.395966] pci 0000:01:00.0: BAR 6: assigned [mem 0x62000000-0x6207ffff pref]
[    3.403218] pci 0000:01:00.0: BAR 4: assigned [mem 0x62180000-0x62183fff 64bit pref]
[    3.411004] pci 0000:01:00.0: BAR 7: assigned [mem 0x62184000-0x62283fff 64bit pref]
[    3.418787] pci 0000:01:00.0: BAR 10: assigned [mem 0x62284000-0x62383fff 64bit pref]
[    3.426645] pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x101f]
[    3.432779] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    3.438042] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    3.444170] pci 0000:00:00.0:   bridge window [mem 0x62000000-0x620fffff]
[    3.450983] pci 0000:00:00.0:   bridge window [mem 0x62100000-0x623fffff 64bit pref]
[    3.458768] pci 0000:00:00.0: Max Payload Size set to  256/ 256 (was  128), Max Read Rq  256
[    3.467260] pci 0000:01:00.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  256
[    3.476168] pcieport 0000:00:00.0: PME: Signaling with IRQ 565

Aquantia dmesg.

root@apalis-imx8-07307404:~# dmesg | grep pci
[    0.000000] Kernel command line: pci=pcie_bus_perf,nommconf,nomsi root=PARTUUID=98821cdc-02 ro rootwait
[    0.271244] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    1.756256] ehci-pci: EHCI PCI platform driver
[    2.545351] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[    2.567933] imx6q-pcie 5f000000.pcie: No cache used with register defaults set!
[    2.693275] imx6q-pcie 5f000000.pcie: PCIe PLL locked after 0 us.
[    2.912687] imx6q-pcie 5f000000.pcie: host bridge /bus@5f000000/pcie@0x5f000000 ranges:
[    2.920813] imx6q-pcie 5f000000.pcie:    IO 0x6ff80000..0x6ff8ffff -> 0x00000000
[    2.928261] imx6q-pcie 5f000000.pcie:   MEM 0x60000000..0x6fefffff -> 0x60000000
[    3.036126] imx6q-pcie 5f000000.pcie: Link up
[    3.140580] imx6q-pcie 5f000000.pcie: Link up
[    3.145005] imx6q-pcie 5f000000.pcie: Link up, Gen3
[    3.150140] imx6q-pcie 5f000000.pcie: PCI host bridge to bus 0000:00
[    3.156582] pci_bus 0000:00: root bus resource [bus 00-ff]
[    3.162324] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    3.168535] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fefffff]
[    3.175455] pci 0000:00:00.0: [1957:0000] type 01 class 0x060400
[    3.181504] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[    3.187796] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x00ffffff pref]
[    3.194563] pci 0000:00:00.0: supports D1 D2
[    3.198854] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[    3.211445] pci 0000:01:00.0: [1d6a:94c0] type 00 class 0x020000
[    3.217595] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0007ffff 64bit]
[    3.224430] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
[    3.231280] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x003fffff 64bit]
[    3.238101] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0001ffff pref]
[    3.245100] pci 0000:01:00.0: supports D1 D2
[    3.249384] pci 0000:01:00.0: PME# supported from D0 D1 D3hot D3cold
[    3.255944] pci 0000:01:00.0: 15.752 Gb/s available PCIe bandwidth, limited by 8 GT/s x2 link at 0000:00:00.0 (capable of 31.506 Gb/s with 16 GT/s x2 link)
[    3.291683] pci 0000:00:00.0: BAR 0: assigned [mem 0x60000000-0x60ffffff]
[    3.298499] pci 0000:00:00.0: BAR 6: assigned [mem 0x61000000-0x61ffffff pref]
[    3.305762] pci 0000:00:00.0: BAR 14: assigned [mem 0x62000000-0x625fffff]
[    3.312663] pci 0000:01:00.0: BAR 4: assigned [mem 0x62000000-0x623fffff 64bit]
[    3.320028] pci 0000:01:00.0: BAR 0: assigned [mem 0x62400000-0x6247ffff 64bit]
[    3.327384] pci 0000:01:00.0: BAR 6: assigned [mem 0x62480000-0x6249ffff pref]
[    3.334635] pci 0000:01:00.0: BAR 2: assigned [mem 0x624a0000-0x624a0fff 64bit]
[    3.341987] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    3.347240] pci 0000:00:00.0:   bridge window [mem 0x62000000-0x625fffff]
[    3.354060] pci 0000:00:00.0: Max Payload Size set to  256/ 256 (was  128), Max Read Rq  256
[    3.362564] pci 0000:01:00.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  256
[    3.371453] pcieport 0000:00:00.0: PME: Signaling with IRQ 565

We were wondering why the difference between upload and download is so huge.
Found few other articles about similar tests.
PCIe-Bandwidth Speed table
i-MX6Q-PCIe-EP-RC-Validation-System

When asked on NXP side of things they said the problem is the device connected to PCIe.
iMX8QM-PCIe-speed-troubleshooting

We also observed weird behaviour with CPU while doing these tests. Everytime iperf3 would start the CPU0 would be utilized or be maxed out while htop does not show any process that would match the CPU usage.
Its always CPU0. We tested that by setting affinity of iperf3 to CPU0 and it drasctically affected resulting speeds. CPU0 utilization was not shown in htop process list.

We are trying to increase download speed to atleast 2.5Gb/s. We also tried playing around with MTU size and it does help in case of upload but has no effect on download.
It appears that the CPU0 is limiting the transfer speed but we are unable to find out which process is using it.

Best regards,
Aleš

Hello @Ales,

does your download source provide the throughput?
Can you provide more information on the setup?

Best regards,

Matthias Gohlke

Hello @matthias.tx ,

Thanks for your swift response.

Yes. Both, intel and aquantia cards are capable of troughputing their advertised speeds. Tested by replacing iMX8 with a PC and running the same test.

Server for iperf3 is running localy on my workstation with Intel converged X520-DA2 (same chipset as Intel card used on client side). Server NIC is a seperate network which is not connected anywhere and has assigned static IP.
Client (iMX8) is directly connected to server and has static IP assigned to interface.

Server is started with:

D:\programs\iperf3.13_64> .\iperf3.exe -s -p 8006 -B 172.25.1.10

Client runs test with following commands:
Upload

root@apalis-imx8-07307404:~# iperf3 -p 8008 -B 172.25.1.12 -c 172.25.1.10 -P 5

Download

root@apalis-imx8-07307404:~# iperf3 -p 8008 -B 172.25.1.12 -c 172.25.1.10 -P 5 -R

Best regards,
Aleš

Hi @Ales ,

just for the record. I am not sure if that has any effect.

But do you see the same behaviour if you use a Toradex carrier board? As I understand you are using your own carrier board correct?

Best Regards
Kevin

Hello @kevin.tx ,

Yes. We do have a apalis eval board with Gen2 PCIe switcher on it. I tried with Intel NIC and it is the same behavior. The only difference is that the upload speed reaches 3.4Gb/s, download is the same with 1.6Gb/s with 100% on CPU0.

While doing tests we also noticed if we decrease iperf3 connection count the speed would obviously not reach the maximum but CPU0 would also not be at 100%. The speed only increased while CPU0 had breathing room, when it reached 100% the speed would not increase further.

Yes. We have a board with PCIEA x1 Gen3 design that was used for testing in previous post. Thinking that maybe another lane would help we put together a simple test board with PCIEA x2 Gen3.

Hi @Ales !

Could you please try to launch two iperf3 on your Apalis iMX8 instances instead of only one?

Please share the results of this test. Maybe it can give us some new clues.

Best regards,

Hello @henrique.tx ,

Test was done on Atlantic NIC.

Up [Gb/s] Down [Gb/s]
Server 1 2.73 0.491
Server 2 2.62 0.507
SUM 5.35 0.998

See results bellow,

4 parallel connections per, Upload
./run_parallel.sh "iperf3 -p 8006 -B 172.25.1.12 -c 172.25.1.10 -P 4" "iperf3 -p 8007 -B 172.25.1.12 -c 172.25.1.10 -P 4"
Server 1:

Accepted connection from 172.25.1.12, port 39377
[  5] local 172.25.1.10 port 8006 connected to 172.25.1.12 port 39605
[  8] local 172.25.1.10 port 8006 connected to 172.25.1.12 port 48759
[ 10] local 172.25.1.10 port 8006 connected to 172.25.1.12 port 55395
[ 12] local 172.25.1.10 port 8006 connected to 172.25.1.12 port 55485
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  71.4 MBytes   599 Mbits/sec
[  8]   0.00-1.00   sec  69.4 MBytes   582 Mbits/sec
[ 10]   0.00-1.00   sec  95.3 MBytes   799 Mbits/sec
[ 12]   0.00-1.00   sec  69.7 MBytes   585 Mbits/sec
[SUM]   0.00-1.00   sec   306 MBytes  2.56 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  77.4 MBytes   650 Mbits/sec
[  8]   1.00-2.00   sec  76.9 MBytes   645 Mbits/sec
[ 10]   1.00-2.00   sec   121 MBytes  1.01 Gbits/sec
[ 12]   1.00-2.00   sec   101 MBytes   846 Mbits/sec
[SUM]   1.00-2.00   sec   376 MBytes  3.15 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  59.5 MBytes   499 Mbits/sec
[  8]   2.00-3.00   sec  49.3 MBytes   414 Mbits/sec
[ 10]   2.00-3.00   sec  82.3 MBytes   691 Mbits/sec
[ 12]   2.00-3.00   sec   120 MBytes  1.01 Gbits/sec
[SUM]   2.00-3.00   sec   311 MBytes  2.61 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  74.8 MBytes   627 Mbits/sec
[  8]   3.00-4.00   sec  65.3 MBytes   547 Mbits/sec
[ 10]   3.00-4.00   sec  79.3 MBytes   665 Mbits/sec
[ 12]   3.00-4.00   sec  93.1 MBytes   781 Mbits/sec
[SUM]   3.00-4.00   sec   312 MBytes  2.62 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  66.1 MBytes   555 Mbits/sec
[  8]   4.00-5.00   sec  60.6 MBytes   509 Mbits/sec
[ 10]   4.00-5.00   sec  82.2 MBytes   690 Mbits/sec
[ 12]   4.00-5.00   sec  93.9 MBytes   788 Mbits/sec
[SUM]   4.00-5.00   sec   303 MBytes  2.54 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  69.3 MBytes   581 Mbits/sec
[  8]   5.00-6.00   sec  71.9 MBytes   603 Mbits/sec
[ 10]   5.00-6.00   sec  93.9 MBytes   787 Mbits/sec
[ 12]   5.00-6.00   sec  86.9 MBytes   728 Mbits/sec
[SUM]   5.00-6.00   sec   322 MBytes  2.70 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  74.0 MBytes   621 Mbits/sec
[  8]   6.00-7.00   sec  77.8 MBytes   653 Mbits/sec
[ 10]   6.00-7.00   sec  79.8 MBytes   669 Mbits/sec
[ 12]   6.00-7.00   sec  79.2 MBytes   664 Mbits/sec
[SUM]   6.00-7.00   sec   311 MBytes  2.61 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  73.7 MBytes   618 Mbits/sec
[  8]   7.00-8.00   sec  82.5 MBytes   692 Mbits/sec
[ 10]   7.00-8.00   sec  80.3 MBytes   674 Mbits/sec
[ 12]   7.00-8.00   sec  85.5 MBytes   717 Mbits/sec
[SUM]   7.00-8.00   sec   322 MBytes  2.70 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  83.5 MBytes   700 Mbits/sec
[  8]   8.00-9.00   sec  81.1 MBytes   681 Mbits/sec
[ 10]   8.00-9.00   sec  83.0 MBytes   696 Mbits/sec
[ 12]   8.00-9.00   sec  83.7 MBytes   702 Mbits/sec
[SUM]   8.00-9.00   sec   331 MBytes  2.78 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  88.7 MBytes   744 Mbits/sec
[  8]   9.00-10.00  sec  89.2 MBytes   749 Mbits/sec
[ 10]   9.00-10.00  sec  97.0 MBytes   814 Mbits/sec
[ 12]   9.00-10.00  sec  84.8 MBytes   711 Mbits/sec
[SUM]   9.00-10.00  sec   360 MBytes  3.02 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  10.00-10.01  sec   600 KBytes   670 Mbits/sec
[  8]  10.00-10.01  sec   473 KBytes   528 Mbits/sec
[ 10]  10.00-10.01  sec   483 KBytes   539 Mbits/sec
[ 12]  10.00-10.01  sec   700 KBytes   781 Mbits/sec
[SUM]  10.00-10.01  sec  2.20 MBytes  2.52 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   739 MBytes   619 Mbits/sec                  receiver
[  8]   0.00-10.01  sec   724 MBytes   607 Mbits/sec                  receiver
[ 10]   0.00-10.01  sec   894 MBytes   749 Mbits/sec                  receiver
[ 12]   0.00-10.01  sec   898 MBytes   753 Mbits/sec                  receiver
[SUM]   0.00-10.01  sec  3.18 GBytes  2.73 Gbits/sec                  receiver

Server 2:

Accepted connection from 172.25.1.12, port 48479
[  5] local 172.25.1.10 port 8007 connected to 172.25.1.12 port 46731
[  8] local 172.25.1.10 port 8007 connected to 172.25.1.12 port 45659
[ 10] local 172.25.1.10 port 8007 connected to 172.25.1.12 port 46045
[ 12] local 172.25.1.10 port 8007 connected to 172.25.1.12 port 44237
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  71.0 MBytes   596 Mbits/sec
[  8]   0.00-1.00   sec  70.2 MBytes   589 Mbits/sec
[ 10]   0.00-1.00   sec  65.3 MBytes   548 Mbits/sec
[ 12]   0.00-1.00   sec  70.0 MBytes   587 Mbits/sec
[SUM]   0.00-1.00   sec   277 MBytes  2.32 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  90.9 MBytes   763 Mbits/sec
[  8]   1.00-2.00   sec  65.8 MBytes   552 Mbits/sec
[ 10]   1.00-2.00   sec  64.7 MBytes   543 Mbits/sec
[ 12]   1.00-2.00   sec  80.8 MBytes   677 Mbits/sec
[SUM]   1.00-2.00   sec   302 MBytes  2.53 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  84.1 MBytes   705 Mbits/sec
[  8]   2.00-3.00   sec  54.2 MBytes   455 Mbits/sec
[ 10]   2.00-3.00   sec  51.2 MBytes   430 Mbits/sec
[ 12]   2.00-3.00   sec  80.1 MBytes   672 Mbits/sec
[SUM]   2.00-3.00   sec   270 MBytes  2.26 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  83.2 MBytes   698 Mbits/sec
[  8]   3.00-4.00   sec  69.7 MBytes   585 Mbits/sec
[ 10]   3.00-4.00   sec  70.3 MBytes   590 Mbits/sec
[ 12]   3.00-4.00   sec  81.7 MBytes   685 Mbits/sec
[SUM]   3.00-4.00   sec   305 MBytes  2.56 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  82.2 MBytes   689 Mbits/sec
[  8]   4.00-5.00   sec  79.9 MBytes   671 Mbits/sec
[ 10]   4.00-5.00   sec  77.2 MBytes   648 Mbits/sec
[ 12]   4.00-5.00   sec  83.3 MBytes   699 Mbits/sec
[SUM]   4.00-5.00   sec   323 MBytes  2.71 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  98.9 MBytes   830 Mbits/sec
[  8]   5.00-6.00   sec  79.4 MBytes   666 Mbits/sec
[ 10]   5.00-6.00   sec  84.8 MBytes   711 Mbits/sec
[ 12]   5.00-6.00   sec  86.5 MBytes   726 Mbits/sec
[SUM]   5.00-6.00   sec   350 MBytes  2.93 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  83.9 MBytes   704 Mbits/sec
[  8]   6.00-7.00   sec  81.0 MBytes   680 Mbits/sec
[ 10]   6.00-7.00   sec  82.5 MBytes   692 Mbits/sec
[ 12]   6.00-7.00   sec  81.7 MBytes   685 Mbits/sec
[SUM]   6.00-7.00   sec   329 MBytes  2.76 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  81.5 MBytes   684 Mbits/sec
[  8]   7.00-8.00   sec  76.8 MBytes   645 Mbits/sec
[ 10]   7.00-8.00   sec  72.6 MBytes   609 Mbits/sec
[ 12]   7.00-8.00   sec  83.5 MBytes   700 Mbits/sec
[SUM]   7.00-8.00   sec   314 MBytes  2.64 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  85.2 MBytes   715 Mbits/sec
[  8]   8.00-9.00   sec  82.7 MBytes   693 Mbits/sec
[ 10]   8.00-9.00   sec  76.8 MBytes   644 Mbits/sec
[ 12]   8.00-9.00   sec  78.6 MBytes   660 Mbits/sec
[SUM]   8.00-9.00   sec   323 MBytes  2.71 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  89.6 MBytes   751 Mbits/sec
[  8]   9.00-10.00  sec  77.6 MBytes   651 Mbits/sec
[ 10]   9.00-10.00  sec  79.5 MBytes   667 Mbits/sec
[ 12]   9.00-10.00  sec  83.1 MBytes   697 Mbits/sec
[SUM]   9.00-10.00  sec   330 MBytes  2.77 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  10.00-10.01  sec   505 KBytes   583 Mbits/sec
[  8]  10.00-10.01  sec   489 KBytes   565 Mbits/sec
[ 10]  10.00-10.01  sec   607 KBytes   701 Mbits/sec
[ 12]  10.00-10.01  sec   498 KBytes   574 Mbits/sec
[SUM]  10.00-10.01  sec  2.05 MBytes  2.42 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   851 MBytes   713 Mbits/sec                  receiver
[  8]   0.00-10.01  sec   738 MBytes   619 Mbits/sec                  receiver
[ 10]   0.00-10.01  sec   726 MBytes   608 Mbits/sec                  receiver
[ 12]   0.00-10.01  sec   810 MBytes   679 Mbits/sec                  receiver
[SUM]   0.00-10.01  sec  3.05 GBytes  2.62 Gbits/sec                  receiver

htop usage:

4 parallel connections per, Download
./run_parallel.sh "iperf3 -p 8006 -B 172.25.1.12 -c 172.25.1.10 -P 4 -R" "iperf3 -p 8007 -B 172.25.1.12 -c 172.25.1.10 -P 4 -R"

Server 1:

Accepted connection from 172.25.1.12, port 60853
[  5] local 172.25.1.10 port 8006 connected to 172.25.1.12 port 38981
[  8] local 172.25.1.10 port 8006 connected to 172.25.1.12 port 42037
[ 10] local 172.25.1.10 port 8006 connected to 172.25.1.12 port 34293
[ 12] local 172.25.1.10 port 8006 connected to 172.25.1.12 port 41477
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  15.4 MBytes   129 Mbits/sec
[  8]   0.00-1.00   sec  12.2 MBytes   103 Mbits/sec
[ 10]   0.00-1.00   sec  13.6 MBytes   114 Mbits/sec
[ 12]   0.00-1.00   sec  16.4 MBytes   137 Mbits/sec
[SUM]   0.00-1.00   sec  57.6 MBytes   483 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  12.9 MBytes   108 Mbits/sec
[  8]   1.00-2.00   sec  12.0 MBytes   101 Mbits/sec
[ 10]   1.00-2.00   sec  15.6 MBytes   131 Mbits/sec
[ 12]   1.00-2.00   sec  16.4 MBytes   137 Mbits/sec
[SUM]   1.00-2.00   sec  56.9 MBytes   477 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  16.6 MBytes   139 Mbits/sec
[  8]   2.00-3.00   sec  15.4 MBytes   129 Mbits/sec
[ 10]   2.00-3.00   sec  11.8 MBytes  98.3 Mbits/sec
[ 12]   2.00-3.00   sec  18.2 MBytes   153 Mbits/sec
[SUM]   2.00-3.00   sec  62.0 MBytes   519 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  13.9 MBytes   116 Mbits/sec
[  8]   3.00-4.00   sec  13.5 MBytes   113 Mbits/sec
[ 10]   3.00-4.00   sec  13.1 MBytes   110 Mbits/sec
[ 12]   3.00-4.00   sec  16.8 MBytes   141 Mbits/sec
[SUM]   3.00-4.00   sec  57.2 MBytes   481 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  15.9 MBytes   133 Mbits/sec
[  8]   4.00-5.00   sec  14.4 MBytes   120 Mbits/sec
[ 10]   4.00-5.00   sec  13.2 MBytes   111 Mbits/sec
[ 12]   4.00-5.00   sec  13.8 MBytes   115 Mbits/sec
[SUM]   4.00-5.00   sec  57.2 MBytes   480 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  13.2 MBytes   111 Mbits/sec
[  8]   5.00-6.00   sec  14.4 MBytes   120 Mbits/sec
[ 10]   5.00-6.00   sec  13.9 MBytes   116 Mbits/sec
[ 12]   5.00-6.00   sec  16.5 MBytes   138 Mbits/sec
[SUM]   5.00-6.00   sec  58.0 MBytes   486 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  12.6 MBytes   106 Mbits/sec
[  8]   6.00-7.00   sec  12.9 MBytes   108 Mbits/sec
[ 10]   6.00-7.00   sec  14.5 MBytes   122 Mbits/sec
[ 12]   6.00-7.00   sec  15.8 MBytes   132 Mbits/sec
[SUM]   6.00-7.00   sec  55.8 MBytes   469 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  16.2 MBytes   136 Mbits/sec
[  8]   7.00-8.00   sec  13.0 MBytes   109 Mbits/sec
[ 10]   7.00-8.00   sec  14.4 MBytes   121 Mbits/sec
[ 12]   7.00-8.00   sec  17.2 MBytes   145 Mbits/sec
[SUM]   7.00-8.00   sec  60.9 MBytes   511 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  16.1 MBytes   135 Mbits/sec
[  8]   8.00-9.00   sec  16.1 MBytes   135 Mbits/sec
[ 10]   8.00-9.00   sec  15.2 MBytes   128 Mbits/sec
[ 12]   8.00-9.00   sec  17.0 MBytes   143 Mbits/sec
[SUM]   8.00-9.00   sec  64.5 MBytes   541 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  14.6 MBytes   123 Mbits/sec
[  8]   9.00-10.00  sec  13.1 MBytes   110 Mbits/sec
[ 10]   9.00-10.00  sec  13.0 MBytes   109 Mbits/sec
[ 12]   9.00-10.00  sec  15.2 MBytes   128 Mbits/sec
[SUM]   9.00-10.00  sec  56.0 MBytes   470 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   148 MBytes   124 Mbits/sec                  sender
[  8]   0.00-10.00  sec   137 MBytes   115 Mbits/sec                  sender
[ 10]   0.00-10.00  sec   138 MBytes   116 Mbits/sec                  sender
[ 12]   0.00-10.00  sec   163 MBytes   137 Mbits/sec                  sender
[SUM]   0.00-10.00  sec   586 MBytes   491 Mbits/sec                  sender

Server 2:

Accepted connection from 172.25.1.12, port 59381
[  5] local 172.25.1.10 port 8007 connected to 172.25.1.12 port 48411
[  8] local 172.25.1.10 port 8007 connected to 172.25.1.12 port 54663
[ 10] local 172.25.1.10 port 8007 connected to 172.25.1.12 port 52669
[ 12] local 172.25.1.10 port 8007 connected to 172.25.1.12 port 42289
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  19.5 MBytes   163 Mbits/sec
[  8]   0.00-1.00   sec  15.6 MBytes   131 Mbits/sec
[ 10]   0.00-1.00   sec  12.0 MBytes   101 Mbits/sec
[ 12]   0.00-1.00   sec  17.0 MBytes   142 Mbits/sec
[SUM]   0.00-1.00   sec  64.1 MBytes   537 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  14.2 MBytes   120 Mbits/sec
[  8]   1.00-2.00   sec  18.0 MBytes   151 Mbits/sec
[ 10]   1.00-2.00   sec  13.2 MBytes   111 Mbits/sec
[ 12]   1.00-2.00   sec  15.4 MBytes   129 Mbits/sec
[SUM]   1.00-2.00   sec  60.9 MBytes   511 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  14.9 MBytes   125 Mbits/sec
[  8]   2.00-3.00   sec  14.5 MBytes   121 Mbits/sec
[ 10]   2.00-3.00   sec  12.0 MBytes   101 Mbits/sec
[ 12]   2.00-3.00   sec  14.2 MBytes   119 Mbits/sec
[SUM]   2.00-3.00   sec  55.6 MBytes   466 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  15.8 MBytes   132 Mbits/sec
[  8]   3.00-4.00   sec  14.1 MBytes   119 Mbits/sec
[ 10]   3.00-4.00   sec  14.4 MBytes   121 Mbits/sec
[ 12]   3.00-4.00   sec  15.0 MBytes   126 Mbits/sec
[SUM]   3.00-4.00   sec  59.2 MBytes   497 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  14.0 MBytes   117 Mbits/sec
[  8]   4.00-5.00   sec  15.0 MBytes   126 Mbits/sec
[ 10]   4.00-5.00   sec  14.8 MBytes   124 Mbits/sec
[ 12]   4.00-5.00   sec  14.8 MBytes   124 Mbits/sec
[SUM]   4.00-5.00   sec  58.5 MBytes   490 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  15.9 MBytes   133 Mbits/sec
[  8]   5.00-6.00   sec  15.0 MBytes   126 Mbits/sec
[ 10]   5.00-6.00   sec  14.0 MBytes   118 Mbits/sec
[ 12]   5.00-6.00   sec  14.2 MBytes   120 Mbits/sec
[SUM]   5.00-6.00   sec  59.1 MBytes   497 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  14.2 MBytes   120 Mbits/sec
[  8]   6.00-7.00   sec  17.2 MBytes   145 Mbits/sec
[ 10]   6.00-7.00   sec  13.8 MBytes   116 Mbits/sec
[ 12]   6.00-7.00   sec  16.4 MBytes   138 Mbits/sec
[SUM]   6.00-7.00   sec  61.6 MBytes   518 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  18.1 MBytes   151 Mbits/sec
[  8]   7.00-8.00   sec  15.4 MBytes   128 Mbits/sec
[ 10]   7.00-8.00   sec  16.6 MBytes   139 Mbits/sec
[ 12]   7.00-8.00   sec  12.6 MBytes   105 Mbits/sec
[SUM]   7.00-8.00   sec  62.8 MBytes   524 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  17.8 MBytes   149 Mbits/sec
[  8]   8.00-9.00   sec  14.8 MBytes   124 Mbits/sec
[ 10]   8.00-9.00   sec  14.6 MBytes   123 Mbits/sec
[ 12]   8.00-9.00   sec  12.1 MBytes   102 Mbits/sec
[SUM]   8.00-9.00   sec  59.2 MBytes   498 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  17.0 MBytes   142 Mbits/sec
[  8]   9.00-10.00  sec  15.6 MBytes   131 Mbits/sec
[ 10]   9.00-10.00  sec  14.4 MBytes   120 Mbits/sec
[ 12]   9.00-10.00  sec  17.1 MBytes   143 Mbits/sec
[SUM]   9.00-10.00  sec  64.1 MBytes   537 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   161 MBytes   135 Mbits/sec                  sender
[  8]   0.00-10.01  sec   155 MBytes   130 Mbits/sec                  sender
[ 10]   0.00-10.01  sec   140 MBytes   117 Mbits/sec                  sender
[ 12]   0.00-10.01  sec   149 MBytes   125 Mbits/sec                  sender
[SUM]   0.00-10.01  sec   605 MBytes   507 Mbits/sec                  sender

htop usage:

Both iperf3 instances were launched in parallel with following bash script.

#!/bin/bash

for cmd in "$@"; do {
  $cmd & pid=$!
  PID_LIST+=" $pid";
} done

trap "kill $PID_LIST" SIGINT
wait $PID_LIST



Following is download test with different number of connections. This was done running single server.

1 connection

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   486 MBytes   407 Mbits/sec                  sender
[  5]   0.00-10.00  sec   485 MBytes   407 Mbits/sec                  receiver

2 connections

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   477 MBytes   400 Mbits/sec                  sender
[  5]   0.00-10.00  sec   476 MBytes   399 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   477 MBytes   400 Mbits/sec                  sender
[  7]   0.00-10.00  sec   476 MBytes   400 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec   955 MBytes   801 Mbits/sec                  sender
[SUM]   0.00-10.00  sec   952 MBytes   799 Mbits/sec                  receiver

3 connections

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   369 MBytes   310 Mbits/sec                  sender
[  5]   0.00-10.00  sec   368 MBytes   309 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   362 MBytes   303 Mbits/sec                  sender
[  7]   0.00-10.00  sec   360 MBytes   302 Mbits/sec                  receiver
[  9]   0.00-10.00  sec   391 MBytes   328 Mbits/sec                  sender
[  9]   0.00-10.00  sec   389 MBytes   326 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  sender
[SUM]   0.00-10.00  sec  1.09 GBytes   937 Mbits/sec                  receiver

4 connections

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   343 MBytes   288 Mbits/sec                  sender
[  5]   0.00-10.00  sec   341 MBytes   286 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   350 MBytes   294 Mbits/sec                  sender
[  7]   0.00-10.00  sec   349 MBytes   293 Mbits/sec                  receiver
[  9]   0.00-10.00  sec   337 MBytes   282 Mbits/sec                  sender
[  9]   0.00-10.00  sec   336 MBytes   282 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec   349 MBytes   292 Mbits/sec                  sender
[ 11]   0.00-10.00  sec   348 MBytes   292 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  1.35 GBytes  1.16 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  1.34 GBytes  1.15 Gbits/sec                  receiver

5 connections

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   292 MBytes   245 Mbits/sec                  sender
[  5]   0.00-10.00  sec   290 MBytes   244 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   292 MBytes   245 Mbits/sec                  sender
[  7]   0.00-10.00  sec   291 MBytes   244 Mbits/sec                  receiver
[  9]   0.00-10.00  sec   279 MBytes   234 Mbits/sec                  sender
[  9]   0.00-10.00  sec   278 MBytes   233 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec   280 MBytes   235 Mbits/sec                  sender
[ 11]   0.00-10.00  sec   279 MBytes   234 Mbits/sec                  receiver
[ 13]   0.00-10.00  sec   285 MBytes   239 Mbits/sec                  sender
[ 13]   0.00-10.00  sec   284 MBytes   239 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  1.39 GBytes  1.20 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  1.39 GBytes  1.19 Gbits/sec                  receiver

Hi @Ales !

Regarding your PCI NICs, do you have any news?

Did you consider testing another type of device to check the upstream x downstream speeds? For example some PCI storage devices?

Best regards,

Hi @henrique.tx ,

I have been trying all sorts of combinations on pci arguments but to no avail. Checked source code of driver and they do use DMA for memory operations. I was unable to get any better results for download.
Changing MTU size did help upload jump to around 6.5Gb/s but it didnt change for download.
Same behavior for CPU0 while testing.

Tried showing kernel processes in htop but there were no high usage processes. Sometimes soft_irq would jump to 5 or so % which is expected.

We dont have any other PCIe devices we could use to test speeds. The 2 we are using to test are because their drivers compile.

Hi @Ales ,

Sorry for the delay. We’re still trying to understand the behaviour you’re seeing.

Do you have any new information that would help us solve this issue?

Best Regards
Kevin

Hi @Ales,

I observed similar behavior on other CPUs as well. Most likely the NIC just generates too many interrupts which can’t be processed by it that fast, which adds some latency and therefore reduces the speed.

It could be that something like gro would help to get better performance if your NIC supports it:
https://docs.kernel.org/networking/segmentation-offloads.html

You can change settings with ethtool if they are supported (you have to check this first with the read command):

# Read it
ethtool -k eth0
# Set it
ethtool -K eth0 rx-gro-hw on

As an alternative, you might want to move the interrupt/task which generates a lot of load to a Cortex A72 CPU (CPU 4 and 5). First, you need to find out which task is responsible for the load. To do this you can use ps:

root@apalis-imx8qm-07021298:~# ps auxf
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           2  0.0  0.0      0     0 ?        S    00:10   0:00 [kthreadd]
root           3  0.0  0.0      0     0 ?        I<   00:10   0:00  \_ [rcu_gp]
root           4  0.0  0.0      0     0 ?        I<   00:10   0:00  \_ [rcu_par_gp]
root           5  0.0  0.0      0     0 ?        I<   00:10   0:00  \_ [slub_flushwq]

Which kthread is consuming a lot of TIME? Then you can move this thread to one of the A72 CPUs:

taskset -pc 5 <PID>

This might improve the performance because the A72 CPUs are much faster.

Regards,
Stefan

1 Like

Hi TX team,

I have run into a brick wall with this. Went over ps with and without kernel processes to see what is using up first CPU core on which the usage was shown on 0 processes.
Monitored soft_irq process, interrupts, and it was showing only few % of usage. By increasing MTU size the % went down and by decreasing it went up, this was to be expected.

Upload test reached the hardware maximum transfer speed of 6.8Gb/s. This speed is also the speed of a loopback test.

Didn’t get any improvements for download. CPU0 shots to 100%, process list does not show it, soft_irq process is as normal.

Marvell and Intel card does not support it.

root@apalis-imx8-07307404:~# ethtool -k eth1 | grep rx-gro-hw
rx-gro-hw: off [fixed]

Marvell upload:
CPU0 was 94%, moved iperf process to CPU4 which did improve speed and lower the core usage.
Speed was 5.02Gb/s.
Cleaned up the ps output a bit.

root@apalis-imx8-07307404:~# ps auxf
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           2  0.0  0.0      0     0 ?        S    10:52   0:00 [kthreadd]
root           9  1.7  0.0      0     0 ?        R    10:52   0:26  \_ [ksoftirqd/0]
root          15  0.1  0.0      0     0 ?        S    10:52   0:02  \_ [ksoftirqd/1]
root           1  0.2  0.1  91464  6924 ?        Ss   10:52   0:03 /sbin/init
root         376  0.0  0.1  29596  4744 ?        Ss   10:52   0:00 /lib/systemd/systemd-journald
root         623  0.0  0.1   8608  5248 ?        Ss   10:52   0:00 /usr/sbin/ofonod -n
root         721  0.0  0.1  11700  5368 ?        Ss   10:52   0:00 /usr/sbin/connmand -n
systemd+     722  0.0  0.1   7772  4696 ?        Ss   10:52   0:00 /lib/systemd/systemd-networkd
root         734  0.1  0.2  22264 10536 ?        Ss   10:52   0:01 /usr/sbin/snmpd -Ls0-6d -a -f
root         735  0.0  0.1  11248  6148 ?        Ss   10:52   0:00 /usr/sbin/snmptrapd -Lsd -f
root        1257 84.0  0.0   6952  3364 pts/1    R+   11:16   0:52      \_ iperf3 -p 8006 -B 172.25.1.12 -c 172.25.1.10
root        1256  2.0  0.0   4336  2980 pts/2    S+   11:16   0:01      \_ htop

Marvell download:
CPU0 was 97%, moved iperf to CPU4.
Speed was 1.14Gb/s.
Cleaned up the ps output a bit.

root@apalis-imx8-07307404:~# ps auxf
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           9  2.4  0.0      0     0 ?        S    10:52   0:46  \_ [ksoftirqd/0]
root          15  0.1  0.0      0     0 ?        S    10:52   0:02  \_ [ksoftirqd/1]
root           1  0.1  0.1  91464  6924 ?        Ss   10:52   0:03 /sbin/init
root         376  0.0  0.1  29596  4744 ?        Ss   10:52   0:00 /lib/systemd/systemd-journald
root         623  0.0  0.1   8608  5248 ?        Ss   10:52   0:00 /usr/sbin/ofonod -n
root         721  0.0  0.1  11700  5368 ?        Ss   10:52   0:00 /usr/sbin/connmand -n
systemd+     722  0.0  0.1   7772  4696 ?        Ss   10:52   0:00 /lib/systemd/systemd-networkd
root         734  0.1  0.2  22264 10536 ?        Ss   10:52   0:02 /usr/sbin/snmpd -Ls0-6d -a -f
root         735  0.0  0.1  11248  6148 ?        Ss   10:52   0:00 /usr/sbin/snmptrapd -Lsd -f
root        1268 42.2  0.0   6952  3088 pts/1    S+   11:23   0:10      \_ iperf3 -p 8006 -B 172.25.1.12 -c 172.25.1.10
root        1256  2.3  0.0   4336  2980 pts/2    S+   11:16   0:09      \_ htop

I am unable to find what uses up CPU0.

Hi @Ales,

What if you move PID 9 and 15 to cpu4/5? Also try to set the governor to performance for testing:

echo performance > /sys/bus/cpu/devices/cpu0/cpufreq/scaling_governor

Regards,
Stefan

Hi @stefan_e.tx ,

CPU governor was set to schedutil on all cores. Changed that to performance.

root@apalis-imx8-07307404:~# grep "" /sys/bus/cpu/devices/*/cpufreq/scaling_governor
/sys/bus/cpu/devices/cpu0/cpufreq/scaling_governor:schedutil
/sys/bus/cpu/devices/cpu1/cpufreq/scaling_governor:schedutil
/sys/bus/cpu/devices/cpu2/cpufreq/scaling_governor:schedutil
/sys/bus/cpu/devices/cpu3/cpufreq/scaling_governor:schedutil
/sys/bus/cpu/devices/cpu4/cpufreq/scaling_governor:schedutil
/sys/bus/cpu/devices/cpu5/cpufreq/scaling_governor:schedutil
root@apalis-imx8-07307404:~# grep "" /sys/bus/cpu/devices/*/cpufreq/scaling_governor
/sys/bus/cpu/devices/cpu0/cpufreq/scaling_governor:performance
/sys/bus/cpu/devices/cpu1/cpufreq/scaling_governor:performance
/sys/bus/cpu/devices/cpu2/cpufreq/scaling_governor:performance
/sys/bus/cpu/devices/cpu3/cpufreq/scaling_governor:performance
/sys/bus/cpu/devices/cpu4/cpufreq/scaling_governor:performance
/sys/bus/cpu/devices/cpu5/cpufreq/scaling_governor:performance

Governor didn’t change the speed of a test.

That didn’t work.

root@apalis-imx8-07307404:~# taskset -apc 5 9
pid 9's current affinity list: 0
taskset: failed to set pid 9's affinity: Invalid argument

Further investigation showed that each core already has a irq process attached to it.

root@apalis-imx8-07307404:~# ps -eL | grep ksoftirqd
      9       9 ?        00:00:00 ksoftirqd/0
     15      15 ?        00:00:00 ksoftirqd/1
     20      20 ?        00:00:00 ksoftirqd/2
     25      25 ?        00:00:00 ksoftirqd/3
     30      30 ?        00:00:00 ksoftirqd/4
     35      35 ?        00:00:00 ksoftirqd/5

Hi @Ales,

What about changing the interrupt affinity by using /proc/interrupts to see which one generates the interrupts and then changing the affinity via /proc/irq//smp_affinity?

root@colibri-imx8x-07021298:/proc/irq/2# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  9:          0          0          0          0     GICv3  25 Level     vgic
 11:    1268094    1231383    1292893    1298420     GICv3  30 Level     arch_timer
 12:          0          0          0          0     GICv3  27 Level     kvm guest vtimer
 14:          0          0          0          0     GICv3  23 Level     arm-pmu
 16:          0          0          0          0     GICv3 341 Level     58400000.jpegdec
 20:          0          0          0          0     GICv3 337 Level     58450000.jpegenc
 25:          0          0          0          0     GICv3 480 Level     imx_mu_chan[3-0]
 26:          0          0          0          0     GICv3 481 Level     imx_mu_chan[3-0]
 27:          0          0          0          0     GICv3 482 Level     imx_mu_chan[3-0]
 36:         30          0          0          0     GICv3 346 Level     sai
 46:       5064          0          0          0     GICv3 347 Level     edma0-chan13-tx, edma0-chan12-rx
 60:          0          0          0          0     GICv3 377 Level     fsl-lpuart
 61:          0          0          0          0     GICv3 379 Level     fsl-lpuart
 62:       8547          0          0          0     GICv3 380 Level     fsl-lpuart
 63:          0          0          0          0     GICv3 272 Level     5a880000.adc
 64:         16          0          0          0     GICv3 372 Level
 65:         92          0          0          0     GICv3 373 Level
 70:          4          0          0          0     GICv3 452 Level     edma2-chan4-rx
 71:          4          0          0          0     GICv3 453 Level     edma2-chan5-tx
 80:          0          0          0          0     GICv3 472 Level     edma2-chan14-rx
 81:          0          0          0          0     GICv3 473 Level     edma2-chan15-tx

Probably the correct one will be written as something with PCIe. Then

cd /proc/irq/465/
echo 10 > smp_affinity

Regards,
Stefan

This shows promise.

root@apalis-imx8-07307404:~# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5
565:        195          0          0          0          0          0     GICv3 105 Level     PCIe PME, eth1

Default smp_affinity value is 3f.
After doing echo 10 > /proc/irq/565/smp_affinity, CPU0 wasn’t used anymore and it shifted to CPU4.

root@apalis-imx8-07307404:/proc/irq/565# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5
565:      31183          0          0          0     480080          0     GICv3 105 Level     PCIe PME, eth1

Marvell card:
Upload:
iperf3 moved to CPU5 at 99%
irq was on CPU4 at 85%
speed was 6.1 Gb/s, around 1 Gb/s more than before.

Download:
iperf3 moved to CPU5 at 35%
irq was on CPU4 at 89%
speed was 1.75 Gb/s, around 0.6 Gb/s more than before.


Intel card:
Upload:
iperf3 moved to CPU5 at 38%
irq was on CPU4 at 92%
speed was 5.65 Gb/s, around 1.1 Gb/s more than before.

Download:
iperf3 moved to CPU5 at 33%
irq was on CPU4 at 100%
speed was 2.25 Gb/s, around 0.8 Gb/s more than before.


Changing MTU size to 3000 on intel I was able to achieve 6.3 Gb/s upload and 3 Gb/s on download.
These values are minimums, upload sometimes shot up to 6.5 and download even to 3.7.


Is there a way to make these changes persist after reboot?
Otherwise can have a setup bash script that executes on reboot.

sidenote:
Changing the smp_affinity on irq caused now to show the usage in ps/htop. It didn’t do that before.

Hi @Ales,

Great to hear this helped to get better results. I don’t think there is a way how you can set the affinity on reboot. The easiest way is to just create a bash script and a systemd service that sets the affinity correctly.

Regards,
Stefan