Hi @Edge,
Moving all the code to OCRAM could again make your binary m4fwloader friendly.
You should make sure that GCC optimizes away unused code functions. It is not done by default. -ffunction-sections compiler switch makes function placed each to new section, then --gc-sections linker switch allows removing unused sections.
For ELF you should either stick to U-Boot’s bootaux command (the way suggested in Toradex Knowledge Base here) or try Linux remoteproc driver.
Remoteproc seems being not listed in KB. Perhaps this thread may help you. Seach on NXP iMX forums may help as well.