[iMX8] SPI Real Time Performance

Hello,

I’m working on a project in which a cortex-A CPU in the imx8 sends data through SPI, waits a short while (~6µs) then recieves data again with an SPI transfer.

The Problem: As we tested the program under the realtime patched linux system,
we notice a big jitter between those two SPI transfers, where the 6µs is not held. instead a jitter up to 1.5 ms happen between the transfers.
we repeated the test on 2 different linux kernels and could reproduce the jitters.

The following pictures shows the chip select signal of the SPI (SPI1_CS).
the purple signal is a result of adding up the oscilloscope pictures over the test time.

first test environment: # Linux apalis-imx8 4.14.117-rt59-89455-g4ed9379-dirty #1 SMP PREEMPT RT
test time: ~2hours
[upload|SfNi7sj3zxiqGpaUdoz9+GvZLvY=]


second test environment:Linux apalis-imx8 4.14.159-rt70-3.0.3+gfff496c #1 SMP PREEMPT RT
test time: ~12 hours

[upload|FUs35RIJXeAW5Tmoii6opQOoWPE=]

The code used for this test is attached: spi_rt.c

Hi @majd.m

Thanks for writing to the Toradex Community!

Linux apalis-imx8 4.14.159-rt70-3.0.3

Bsp 3.0b3 is not supported anymore. Could you update to the newest Bsp 3.0b4 and check if you can reproduce this issue?

Thanks and best regards,
Jaski

Hi @jaski.tx,

I updated to BSP 3.0b4, but the SPI behaviour seem unfortunately just the same.
Kernel: Linux apalis-eval-imx8-man 4.14.170-rt74-3.0.4+gbaa6c24 #1 SMP PREEMPT RT

DVFS is deaktivated in all tests with:

cpufreq-set -f 1200MHz -c 0
cpufreq-set -f 1600MHz -c 4

The code is updated to add a GPIO pin which set to 1 during the spi transfers, 0 at the end of the both transfers.

spi_rt.c code

Legends:

Yellow signal: the GPIO signal

Purple signal: SPI_CS

alt text

Best regards, Majd

Hello,

To check the jitter of each part of the test program, I used a gpio for each part.
all gpios are switched to high at the beginning of the test program, then one gpio is switched to low after each part is executed.

In addition I configured the nanosleep function to so that it make the thread sleep at absolute time.

This modification makes it more clear that the big jitter comes from the SPI transfer.

The test program after the modification: spi_rt.c

The result after ~3hours of test looks like this

Yellow (gpiod_TEST_pin1): represents the delay of the nanosleep or the gpiod itself.

Cyan (gpiod_TEST_pin2): represents the delay to the first spi transfer

Blue (gpiod_TEST_pin2): represents the delay to the second spi transfer

Purple (SPI_CS): represents the chip select signal of the both spi transfers

alt text

Hi @majd.m

Being at Home-Office, I cannot do the measurement this week. I will try to reproduce the issue next week.

However you need to test your code without RT Kernel Patch and compare this to the kernel with Real-Time kernel Patch. We have not done any tests on iMX8QM, so I cannot provide any values. Using Linux OS there always will be a jitter even with Real-Time patch.

What is your application? What values for jitter do you expect?

Best regards,
Jaski

Dear @jaski.tx ,

our application is a real time critical measurement for motion control. I know we could do it on the Cortex-M side, but that would add additional development efforts for communication between Cortex-M and Cortex-A side.

I would expect to see a jitter that is roughly the same as the worst case latency from cyclictest (<<100µs). What we are seeing are jitters of > 400 µs, so I suspect we are not using the system correctly or the SPI driver isn’t RT-aware.

First tests with setting the “rt-flag” in spidev.c seem to show improvement, but we need to do long-term measurements to be sure.

Hi @m.sauer

I don’t know what you exactly mean with rt-flag.

Best regards,
Jaski

See Imx6/linux: Large and inconsistent latency when issuing an irq triggered spi read - #6 by Garyio - Technical Support - Toradex Community

The IMX8 seems to use spidev.c which includes spi.c. In this file you can find the same rt-flag, with minor naming differences:

if (ctlr->rt) {
        dev_info(&ctlr->dev,
            "will run message pump with realtime priority\n");
        sched_setscheduler(ctlr->kworker_task, SCHED_FIFO, &param);
    }

Hi

Thanks for the information.
I never tried this patch myself, but you could try this patch.

Best regards,
Jaski

Two more tips which can improve the latency seen on SPI:

Disable DMA. Setting up DMA, initiating transfer with it typically takes time and makes the worst case latency worse. The driver used on the i.MX 8QM is this: drivers/spi/spi-fsl-lpspi.c. If you recompile the kernel anyways you can just set fsl_lpspi->usedma to 0 always.

Second, the driver makes use of runtime power management. Runtime power management shuts down clocks when the peripheral is not in use. Clocks on i.MX 8QM are handled by a special controller, and therefor may take time to enable. Leaving clocks enabled always is probably the way to go when trying to achieve lowest latencies. As a hacky way, I think removing SET_RUNTIME_PM_OPS in the struct dev_pm_ops should do that.

Btw, you should be able to change the scheduling priority of the SPI message pump also by using user space tools. Kernel threads are in square brackets. You can use ps to find the pid of the message pump, and set its priority:

ps xfa | grep spi
chrt -f -p <pidof spi message pump kernel thread>

Hello @stefan.tx,

Thank you very much for your suggestions.

By commenting out the SET_RUNTIME_PM_OPS in the mentioned struct in spi driver, I get a kernel panic when i start the program with the spi communication.

I tried to comment out other parts where the clock suspend and resume is called, but i get the same kernel panic.

Is there another way to keep the clocks always enabled?

Thanks in advance,

Best regards, Majd

Hi @majd.m

Is the issue still present on your side?

Best regards,
Jaski