As long as swap and paging(more precise page out) not occurring then interrupt latency for the kernel and user thread will be the same but you need to set a really very high priority for the thread. As you know most of the embedded products a single application always run, so swap and page out wouldn’t occur if users not stressing the product. If you really want to write kernel driver and you can create subproject driver and add NK SHKM(Kernel and not page out) flag For example
YourDriver.dll $(_FLATRELEASEDIR)\YourDriver.dll NK SHKM
If you need to avoid all the operating system overhead of interrupt handling then you would think about bare metal firmware on M4 core. There is a dedicated interrupt vector, code or instructions will start to execute as soon as the interrupt is fired.
Please let me know if you have any other question on this topic.