Hi,
We have a product uses Verdin iMX8M-Plus module (v1.0D) with Toradex BSP 5.6 (with Dunfell) and with the Linux Version:
uname -a
Linux verdin-imx8mp 5.4.193-0+git.ae0c2c1c7920 #1 SMP PREEMPT aarch64 aarch64 aarch64 GNU/Linux
On our custom board we have an FPGA connected to the SOM through the PCIe bus.
The structure is more less like that:
- The FPGA fills its FIFO with a measurement data
- Whenever FIFO is not empty, it pulls the GPIO1.7 pin high configured as a high level interrupt source on the SOM
- In the pin IRQ, the SOM disables the level interrupt and setup the DMA to read data in the FIFO through PCIe. (IRQ routine takes max 10us)
- Once DMA reads the FIFO, it generates an MSI interrupt, and inside MSI interrupt I push this data to a kfifo and enable GPIO1.7 level high interrupt back. (this interrupt takes about 3 to12us)
This structure works fine with one small problem. When the FPGA pulls the interrupt pin high, my interrupt routine takes the action within about 300uS which is too long for this processor. Our application requires it to be 50uS max.
This latency can be seen on the oscilloscope output above, the yellow is the FPGA pin which triggers pin interrupt on the SOM, and the green one is the first action taken in the pin IRQ.
Because I have two interrupts sequentially chained (first pin interrupt, and then DMA MSI interrupt), I observe similar latency between those two.
One interesting note that, according to the scope output above, only first 2 interrupts have this 300uS latency, the rest are executed almost immediately (3us). Any idea why it could be?
There is a similar discussion I found below:
I have followed the all suggestions there such as using IRQF_NO_THREAD flag or CONFIG_NO_HZ_FULL or CONFIG_HZ_1000 in the kernel config but I see no difference at each of those settings.
We didn’t try RT_PATCH because it causes USB Gadget issues from our previous experience, therefore I skip this option.
Since I stuck here and don’t know what to try next, your suggestions are highly appreciated.
Relevant driver code like below:
//this takes max 11us
static irqreturn_t fpga_pin_interrupt_handler(int irq, void *dev_id)
{
gpio_set_value(GPIO4_IO03, 1);
// disable IRQF_TRIGGER_HIGH interrupt
disable_irq_nosync(irq);
spin_lock_irqsave(&io_dev.streaming_spinlock, pin_irq_flags);
(void)setup_dma_to_read(); // triggers MSI interrupt at completion
spin_unlock_irqrestore(&io_dev.streaming_spinlock, pin_irq_flags);
return IRQ_HANDLED;
}
//executed by PCIe-MSI interrupt: this takes between 3 - 12us
static void streaming_dma_scan_done_cb(void *arg)
{
spin_lock_irqsave(&io_dev.streaming_spinlock, msi_irq_flags);
////////////////////////////////////////
// push data to kfifo
////////////////////////////////////////
...
////////////////////////////////////////
wake_up_interruptible(&streaming_dma_fifo_poll_wait);
spin_unlock_irqrestore(&io_dev.streaming_spinlock, msi_irq_flags);
gpio_set_value(GPIO4_IO03, 0);
// enable IRQF_TRIGGER_HIGH interrupt back
enable_irq(fpga_interrupt_pin_irq);
}
int register_ioctl(struct pci_dev *pci_dev, void __iomem **bar_addrs)
{
...
...
if (request_irq(fpga_interrupt_pin_irq, (void*)fpga_pin_interrupt_handler, IRQF_TRIGGER_HIGH | IRQF_NO_THREAD, "fpga_interrupt_pin", (void*)&io_dev))
{
dev_err(&pci_dev->dev, "%s: FPGA_INTERRUPT_PIN %d cannot register IRQ: %d\n", __func__, FPGA_INTERRUPT_PIN, fpga_interrupt_pin_irq);
goto err7;
}
}
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
1: 0 0 0 0 GICv3 25 Level vgic
3: 35694 30825 57117 25633 GICv3 30 Level arch_timer
4: 0 0 0 0 GICv3 27 Level kvm guest vtimer
6: 7759 5544 5140 2112 GICv3 79 Level timer@306a0000
7: 0 0 0 0 GICv3 130 Level imx8_ddr_perf_pmu
9: 0 0 0 0 GICv3 23 Level arm-pmu
20: 0 0 0 0 GICv3 110 Level 30280000.watchdog
21: 0 0 0 0 GICv3 52 Level caam-snvs
22: 0 0 0 0 GICv3 51 Level rtc alarm
23: 0 0 0 0 GICv3 36 Level 30370000.snvs:snvs-powerkey
26: 0 0 0 0 GICv3 63 Level 30820000.spi
27: 0 0 0 0 GICv3 58 Level 30860000.serial
28: 373 0 0 0 GICv3 60 Level 30880000.serial
29: 0 0 0 0 GICv3 59 Level 30890000.serial
32: 24436 0 0 0 GICv3 67 Level 30a20000.i2c
33: 372 0 0 0 GICv3 68 Level 30a30000.i2c
34: 5212 0 0 0 GICv3 70 Level 30a50000.i2c
36: 0 0 0 0 GICv3 108 Level 30ad0000.i2c
37: 0 0 0 0 GICv3 55 Level mmc1
38: 19233 0 0 0 GICv3 56 Level mmc2
39: 0 0 0 0 GICv3 139 Level 30bb0000.spi
40: 0 0 0 0 GICv3 34 Level sdma
41: 0 0 0 0 GICv3 166 Level eth0
42: 2978 0 0 0 GICv3 167 Level eth0
43: 0 0 0 0 GICv3 135 Level sdma
48: 0 0 0 0 GICv3 180 Level 32f10100.usb
49: 0 0 0 0 GICv3 181 Level 32f10108.usb
50: 0 0 0 0 GICv3 39 Level hantrodec
51: 0 0 0 0 GICv3 40 Level hantrodec
52: 0 0 0 0 GICv3 62 Level hx280enc
53: 4 0 0 0 GICv3 72 Level dwc3
54: 46194 0 0 0 GICv3 73 Level xhci-hcd:usb1
55: 351 0 0 0 GICv3 137 Level 30901000.jr
56: 4 0 0 0 GICv3 138 Level 30902000.jr
57: 0 0 0 0 GICv3 146 Level 30903000.jr
61: 0 0 0 0 gpio-mxc 3 Edge pca9450
64: 2595 0 0 0 gpio-mxc 6 Edge (null)
65: 3430 0 0 0 gpio-mxc 7 Level fpga_interrupt_pin
68: 3 0 0 0 gpio-mxc 10 Level stmmac-0:07
100: 0 0 0 0 gpio-mxc 10 Edge usb_1_id
102: 0 0 0 0 gpio-mxc 12 Edge 30b50000.mmc cd
154: 0 0 0 0 gpio-mxc 0 Edge Wake-Up
218: 0 0 0 0 pca9450 0 Level pca9450-pmic
220: 0 0 0 0 PCI-MSI 0 Edge PCIe PME, aerdrv
221: 4257 0 0 0 PCI-MSI 524288 Edge avalon_dma
IPI0: 9213 24924 25849 18176 Rescheduling interrupts
IPI1: 138 78 160 184 Function call interrupts
IPI2: 0 0 0 0 CPU stop interrupts
IPI3: 0 0 0 0 CPU stop (for crash dump) interrupts
IPI4: 540 142 193 2772 Timer broadcast interrupts
IPI5: 6560 4544 6300 941 IRQ work interrupts
IPI6: 0 0 0 0 CPU wake-up interrupts
Thanks
