Unpredictable crashes of Cortex-M7 (i.MX8MP) since moving some code and data to OCRAM

Good evening everyone,

We recently enabled OCRAM on our Cortex-M7 (on a Verdin iMX8M Plus module) and moved some non-critical text and data into it. However, since we made this change, we have been getting some unpredictable crashes on our M7. What I mean is, while developing, depending on the changes we just added, it sometimes never crashes (uptime of >8h), it crashes after around 1h, and then sometimes even after 5min. We are currently analysing our code for UB and races, and have finally added serial print-outs to all FreeRTOS configAssertsand newlibs asserts, but we have not yet been able to pin down our mistake.
While we are still analysing our code, I wanted to double-check our OCRAM setup in the linker file and its MPU settings, because what I really don’t understand is why it never happened (~1.5 years!) when all the code was still in ITCM & DTCM. Also, we use OCRAM on a Cortex-M4 on Colibri iMX7 just fine. However, TBH, I still suspect some bad usage of FreeRTOS Kernel APIs, for which in slower code the configAsserts fire, but anyways I would be really thankful if you could glance over our setup :slight_smile: .

Our main questions would be:

  • Do you know of others that had issues with OCRAM on M7-Cortex
  • Do you note some unusual things with our setup below?
  • Might Linux use OCRAM at the same time as M7 does? I thought it was already declared no-map in imx8mp.dtsi#L213
    • We are building our own Linux using torizon minimal (still on the 7.4.0 tag with very minimal customisations)

I added the full linker, startup and MPU config file (board.c) as an attachment, but quickly cover the most important sections here:

We use remoteproc, and as far as I know, the driver splits OCRAM into three regions (c.f. imx_rproc.c#L189), so our linker file adds those three sections and adds the data & text accordingly:

MEMORY
{
  m_itcm_isr             (RX)  : ORIGIN = 0x00000000, LENGTH = 0x00000400
  m_itcm_text            (RX)  : ORIGIN = 0x00000400, LENGTH = 0x0001FC00
  m_dtcm_data            (RW)  : ORIGIN = 0x20000000, LENGTH = 0x00020000
  m_ocram_data           (RW)  : ORIGIN = 0x20200000, LENGTH = 0x00020000 /* 128 KB, fits ATT entry 1 */
  m_ocram_text_lo        (RWX) : ORIGIN = 0x20220000, LENGTH = 0x00020000 /* 128 KB, fits ATT entry 2 */
  m_ocram_text_hi        (RWX) : ORIGIN = 0x20240000, LENGTH = 0x00010000 /*  64 KB inside ATT entry 3, ends below the lowest BL31 base */
  /* 16MB DDR reserved in the Linux device-tree: https://git.toradex.com/cgit/device-tree-overlays.git/tree/overlays/verdin-imx8mp_hmp_overlay.dts?h=toradex_6.6-2.1.x-imx#n47 */
  m_ddr_data             (RW)  : ORIGIN = 0x80000000, LENGTH = 0x01000000
}

  /* Non-critical code pre-diverted to OCRAM before the ITCM wildcard below. */
  .text_ocram_lo :
  {
    . = ALIGN(4);
    */rtos_tasks/communication/*.c.obj(.text* .rodata*)
    */rtos_tasks/monitor/*.c.obj(.text* .rodata*)
    */rtos_tasks/worker/*.c.obj(.text* .rodata*)
    */rtos_tasks/fsm/*.c.obj(.text* .rodata*)
    . = ALIGN(4);
    __ocram_text_lo_end__ = .;
  } > m_ocram_text_lo

  .text_ocram_hi :
  {
    . = ALIGN(4);
    */rtos_tasks/input_output/*.c.obj(.text* .rodata*)
    */rtos_tasks/linear_motor/*.c.obj(.text* .rodata*)
    */rtos_tasks/pose_control/*.c.obj(.text* .rodata*)
    */drivers/linear_motor/*.c.obj(.text* .rodata*)
    . = ALIGN(4);
    __ocram_text_hi_end__ = .;
  } > m_ocram_text_hi

  /* Non-critical data pre-diverted to OCRAM before the DTCM wildcard below
   * .bss* is intentionally omitted because the startup zero loop in startup_MIMX8ML8_cm7.S
   * only covers __bss_start__/__bss_end__ (DTCM), so BSS from these modules fall through to 
   * .bss in DTCM and is zeroed there.
   * NOTE: we don't care anyway as we hardly use any BSS
   */
  .data_ocram :
  {
    . = ALIGN(4);
    */rtos_tasks/communication/*.c.obj(.data*)
    */rtos_tasks/monitor/*.c.obj(.data*)
    */rtos_tasks/worker/*.c.obj(.data*)
    */rtos_tasks/fsm/*.c.obj(.data*)
    */rtos_tasks/input_output/*.c.obj(.data*)
    */rtos_tasks/linear_motor/*.c.obj(.data*)
    */rtos_tasks/pose_control/*.c.obj(.data*)
    */drivers/linear_motor/*.c.obj(.data*)
    . = ALIGN(4);
    __ocram_data_end__ = .;
  } > m_ocram_data

In board.c we set the MPU like this (cachable):

    /* Region 7 OCRAM[0x2020_0000 - 0x2024_FFFF]: Normal, non-shareable, write-back/write-allocate.
     * Overrides Region 0 (Device/XN) to enable instruction fetch from OCRAM.
     *
     * 512KB base (smallest power-of-2 ≥ 320KB, aligned at 0x2020_0000). SRD=0xE0 disables
     * sub-regions 5-7 (64KB each), fencing off the OCRAM range 0x2025_0000-0x2027_FFFF that
     * holds ATF BL31 (and BL2, if NEED_BL2 is enabled). This is defensive across the full set
     * of TF-A configurations we might encounter on a Toradex BSP bump:
     *   NEED_BL2 unset (current, TF-A lf_v2.10_6.6.52_2.2.x): BL31_BASE = U(0x970000)
     *                                                         → M7-local 0x2027_0000-0x2028_FFFF (128KB)
     *   NEED_BL2 set:                                         BL31_BASE = U(0x950000), BL2 at U(0x970000)
     *                                                         → M7-local 0x2025_0000-0x2028_FFFF (256KB)
     *   https://github.com/nxp-imx/imx-atf/blob/lf_v2.10/plat/imx/imx8m/imx8mp/include/platform_def.h
     *
     * Older TF-A versions also placed BL31 inside sub-regions 5 or 6 (lf_v2.4: 0x960000,
     * lf_v2.6: 0x960000 regression). SRD=0xE0 covers all of these:
     *   https://www.mail-archive.com/barebox@lists.infradead.org/msg41959.html
     *
     * Sub-region map (disabled regions fall back to Region 0, Device/XN):
     *   enabled  0-4: 0x2020_0000-0x2024_FFFF (320KB: m_ocram_data + m_ocram_text_lo + m_ocram_text_hi)
     *   disabled 5-7: 0x2025_0000-0x2027_FFFF (BL2/BL31 across all known TF-A configurations) */
    MPU->RBAR = ARM_MPU_RBAR(7, 0x20200000U);
    MPU->RASR = ARM_MPU_RASR(0, ARM_MPU_AP_FULL, 1, 0, 1, 1, 0xE0, ARM_MPU_REGION_SIZE_512KB);

Thank you very much for your support, and let me know if you need any other info.

Sincerely,

Chris & Team

board-minimal.c (13.7 KB)

MIMX8ML8xxxLZ_ram-minimal.ld (9.8 KB)

startup_MIMX8ML8_cm7-minimal.S (66.0 KB)

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.


Good evening Chris & Team,

Thank you for the detailed description of your OCRAM setup and the linker configuration. I’ll address your questions based on the available documentation.

Memory Areas and OCRAM on Verdin iMX8M Plus

According to the Toradex documentation, the Verdin iMX8M Plus features a heterogeneous multicore architecture with the Cortex-M7 core alongside 4 Cortex-A53 primary CPU cores(1). The system includes Tightly Coupled Memory (TCM) for the M7 core, offering low-latency access(1).

The documented memory layout for Verdin iMX8M Plus shows(1):

  • DTCM for M7: 128KB at 0x2000_0000-0x2001_FFFF
  • ITCM for M7: 128KB at 0x0000_0000-0x0001_FFFF
  • DDR Address: 2GB accessible by M7 at 0x4000_0000-0xBFFF_FFFF

Observations About Your Setup

Looking at your configuration, I notice a few points worth verifying:

  1. OCRAM Address Range: Your linker file places OCRAM at 0x20200000-0x2024FFFF. While this appears to be a valid OCRAM range for iMX8MP, the official Toradex documentation for Verdin iMX8M Plus(1) does not explicitly list OCRAM regions in the memory map table (unlike the Colibri iMX7 documentation(2)(1), which does document OCRAM regions).

  2. Linux Reserved Memory: You mentioned the memory is declared no-map in the device tree overlay. The verdin-imx8mp_hmp_overlay.dts does reserve memory regions for M7 communication(3), but you should verify that your specific OCRAM regions are properly excluded from Linux access.

  3. Cache Coherency: Your MPU configuration sets OCRAM as cacheable (write-back/write-allocate). With RemoteProc and shared memory scenarios, cache coherency between the M7 and A53 cores could be a concern if any of this OCRAM is accessed by both cores.

Recommendations

Before we can provide more specific guidance, could you clarify:

  1. Which exact version of Torizon OS are you running? (You mentioned 7.4.0 tag, but please confirm the full version)
  2. Are you using RemoteProc to start the M7 firmware?(4)
  3. Is any of the OCRAM region shared with Linux, or is it exclusively used by the M7?
  4. Have you verified the device tree overlay is correctly applied and the memory regions are properly reserved?

You can use the tdx-info command to get detailed information about your system configuration. See: Getting Device Information with Tdx-Info | Toradex Developer Center

Additional Considerations

The TCM regions can be accessed with zero wait-states and provide massively better performance than DRAM, even if cached(1). The fact that your code ran stably in TCM for 1.5 years but crashes when moved to OCRAM suggests the issue might be related to:

  • Memory access timing differences
  • Cache configuration issues
  • Potential conflicts with other system components using OCRAM
  • Memory region alignment or MPU configuration

Once you provide the additional details about your exact OS version and setup, I can offer more targeted guidance.

Best regards,
Toradex AI Assistant.


Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

Additional notes:

  • Currently our application layer only uses OCRAM on the M7, we do not use it on linux
  • The crashes also happen when OCRAM was configures as non-cachable

Thank you for the additional information. The fact that crashes occur even with non-cacheable OCRAM configuration is particularly significant, as it rules out cache coherency issues.

Critical Setup Verification

Since you’re using RemoteProc with Torizon OS 7.4.0, let me highlight some important considerations:

RemoteProc and Kernel Command Line

For Verdin iMX8M Plus using RemoteProc, you need a special kernel command line argument for remoteproc to be functional(1). You must add clk-imx8mp.mcore_booted=1 to the kernel command line and reboot before the RemoteProc framework can properly manage the M7 core(1).

You can add this command-line argument using the fw_setenv command from the Linux user space(1):

fw_setenv tdxargs "clk-imx8mp.mcore_booted=1"

Have you verified this kernel parameter is set? Without it, there could be clock gating or power management conflicts between Linux and the M7 that could manifest as unpredictable crashes.

Device Tree Overlay Verification

To confirm your device tree overlay is correctly applied, you can check using dmesg(1):

dmesg | grep -E "remote|rproc"

You should see output similar to:

[ 1.629455] remoteproc remoteproc0: imx-rproc is available

Additionally, from Torizon OS 6.1.0-devel-202212 onwards, you can verify if overlays were applied successfully by checking /proc/device-tree/chosen/overlays(2). Each overlay that was attempted will have a file named <device_tree_overlay_name>.dtbo, and if the value is 0, it means the overlay was applied successfully(2).

OCRAM Memory Region Alignment

Looking at your linker configuration, your OCRAM regions are:

  • m_ocram_data: 0x20200000 (128 KB)
  • m_ocram_text_lo: 0x20220000 (128 KB)
  • m_ocram_text_hi: 0x20240000 (64 KB)

Your MPU configuration uses a 512KB base region with sub-region disable (SRD=0xE0) to protect the ATF BL31 area. This approach is sound, but please verify:

  1. Are these exact OCRAM addresses properly reserved in your device tree overlay? The overlay must match these regions precisely to prevent Linux from accessing them.

  2. Does your startup code properly initialize the OCRAM sections before use? Unlike TCM, OCRAM may require explicit initialization.

Debugging Suggestions

Since the crashes are unpredictable and timing-dependent, consider:

  1. Memory Access Patterns: OCRAM has different access characteristics than TCM. TCM provides zero wait-state access(1), while OCRAM may have different latency. This could expose race conditions that were masked by TCM’s faster access.

  2. Stack Placement: Verify that your stacks are still in TCM/DTCM and haven’t inadvertently been placed in OCRAM, which could cause stack-related issues.

  3. Interrupt Handlers: Ensure critical interrupt handlers remain in ITCM for deterministic timing.

Could you please confirm:

  • Whether the clk-imx8mp.mcore_booted=1 kernel parameter is set?
  • The output of dmesg | grep -E "remote|rproc"?
  • Whether you’ve verified the overlay application status in /proc/device-tree/chosen/overlays?

Best regards,
Toradex AI Assistant.


Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

Hi,
For curiosity purpose, I passed your interested case to the commercial Anthropic Claude Code with Opus 4.8 (and not the open-source alternative e.g. empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF · Hugging Face) to analyze it. Here the answer. Hope it helps, Daniel
—
Symptom recap. Rock‑solid for 1.5 years with all code in ITCM/DTCM; after moving non‑critical code+data to OCRAM, random crashes (sometimes >8 h, sometimes 5 min), and MTBF tracks which code change was added. Works fine on a Cortex‑M4 (Colibri iMX7).

Two filters every theory must pass: (A) variable MTBF → cause must be non‑deterministic; (B) TCM‑stable 1.5 y → cause must be something the move changed. The M4‑works clue is decisive: the M4F has no L1 cache and isn’t an r0p1 M7 → the cause is an M7‑specific feature: its cache or its core erratum.

Ranked causes

1 — Cacheable OCRAM (coherency + ECC). The move silently flipped the data from non‑cacheable (DTCM Region 3, board.c:113) to cacheable write‑back/write‑allocate (OCRAM Region 7, board.c:153), D‑cache on (board.c:190), with no cache maintenance anywhere. Two failure modes:

  • DMA/peripheral buffers in the moved modules are now incoherent — a textbook M7 footgun (STM32 SD/lwIP, STM32H7 DMA, Zephyr #36471, NXP AN12042 prescribes the clean/invalidate you’re missing).
  • ECC: i.MX8M Plus has OCRAM ECC on by default (confirmed on NXP’s forum). The unused OCRAM tails are never written → invalid ECC; a write‑allocate line fill or speculative prefetch near a region boundary reads them → uncorrectable ECC → BusFault. Both modes vanish in a valid non‑cacheable test (no line fills) — which is exactly why this is live for the cacheable build you ship, and why “non‑cacheable still crashed” doesn’t clear it unless that test was clean.

2 — Cortex‑M7 r0p1 erratum 837070 (check first). i.MX8MP M7 is r0p1; without the workaround/r0p1 port an IRQ can slip into a taskENTER_CRITICAL window and corrupt a DTCM kernel list. Because taskENTER_CRITICAL is inlined into the moved modules now running from wait‑stated OCRAM, the vulnerability window widens. Cache‑independent → survives the non‑cacheable test, and a vListInsert configASSERT is its fingerprint (“asserts fire in slow code”). Patched independently by FreeRTOS, Zephyr (#23930), RTIC (#53), QP/C (#162), ChibiOS (#675) — well‑corroborated.

3 — A latent race/UB in your code, unmasked by timing (your own hypothesis; can’t be excluded by config).
4 — ISR‑in‑OCRAM latency → missed motor/CAN deadlines (watchdog, not a fault).
5 — Another bus master writes 0x2020_0000–0x2024_FFFF (“no‑map in Linux” ≠ “no SoC master touches it”).

Conditional/low: mcore_booted=1 (Toradex AI’s pick — unlikely at 8 h uptimes, but free to test), -mfpu single‑vs‑double, MSP overflow.

Ruled out (verified in the files): uninitialized .data/.bss (remoteproc loads in place; DTCM bss zeroed, .S:225–310); MPU sub‑region math (correct, XN=0); FreeRTOS V11 configCHECK_HANDLER_INSTALLATION (vectors stay in ITCM); standalone temperature (fails filter B — same silicon ran 1.5 y; it can only modulate #2/#4/#5 via NoC/DVFS latency, and MTBF tracking code edits points away from ambient heat).

What the AI and the thread missed

ToradexAI led with its weakest idea (mcore_booted) and dropped its strongest (it raised “cache coherency,” then abandoned it without noticing the posted config is cacheable). It never mentioned the r0p1 erratum or the ECC‑default‑on detail — the two M7‑specific differentiators that the M4‑works clue and the corroborating reports both point to.

Do this, in order

  1. Run verify_m7_errata.c + grep the port/FreeRTOSConfig.h for the 837070 workaround. r0p1 + workaround OFF → enable it via freertos_errata_837070.h, retest. (Settles #2 — cheapest decisive test.)
  2. Add BOARD_ScrubOcramEcc() as the first line of main() before BOARD_InitMemory(). (Tests the ECC mechanism of #1.)
  3. On the next crash, dump CFSR/HFSR/BFAR/MMFAR + stacked PC/LR (faults already enabled, board.c:179,186): BusFault on 0x202x_xxxx → #1/#5/ECC; vListInsert assert → #2/#3; watchdog‑only → #4.
  4. Bisect (data‑in‑DTCM vs code‑in‑DTCM) and re‑run OCRAM non‑cacheable to split cache‑side from erratum‑side. Log the TMU in parallel for the thermal correlation.

README_OCRAM_M7_crash_findings.md (6.6 KB)

ocram_ecc_scrub.h (1.4 KB)

freertos_errata_837070.h (1.6 KB)

ocram_ecc_scrub.c (3.4 KB)

verify_m7_errata.h (539 Bytes)

verify_m7_errata.c (1.2 KB)

Hello @Fungiking

After some deep research on this topic, what I understand is the following:
In the case of M7, OCRAM is cacheable, while TCM never was. In your previous working/funcitonal case with ITCM/DTCM, it was non-cacheable, and TCM is dual-ported/tightly-coupled and the L1 D-cache was never in the path. So on Cortex-M7, cacheable OCRAM data can expose bugs that were invisible before.
In the case of Colibri iMX7, Cortex-M4 has no L1 data cache at all, so OCRAM there is inherently coherent.
From region 7 (OCRAM) in the board-minimal.c file, I see:
MPU->RASR = ARM_MPU_RASR(0, ARM_MPU_AP_FULL, 1, 0, 1, 1, 0xE0, ARM_MPU_REGION_SIZE_512KB); → which means it is cacheable (according to the table above on MPU configure, check line 85).
Now, if you check region 1 (ITCM) and 3 (DTCM), you will see they are normal, non-cacheable.

The behavior sounds like a stale cache line corruption. It could manifest as timing-dependent crashes that vary with code changes or assertion failures. With more code in OCRAM, more stack frames and local variables spill into cache, increasing the window for incoherence.

I would do a diagnostic to isolate the issue. Flip just the OCRAM region to non-cacheable. That iis, make it behave exactly like your DTCM did and leave everything else identical:

/* Region 7: match DTCM/ITCM attributes (Normal, non-cacheable) — diagnostic */
MPU->RASR = ARM_MPU_RASR(0, ARM_MPU_AP_FULL, 1, 0, 0, 0, 0xE0, ARM_MPU_REGION_SIZE_512KB);
/*                                            TEX=1 S=0 C=0 B=0                       */

If the crashes disappear, it’s a cache-coherency bug in the moved code (a missing clean/invalidate or false-sharing), and you can then decide whether to keep OCRAM non-cacheable (simple, slower) or reintroduce cacheability with proper maintenance / cache-line alignment + padding on the shared objects.
If the crashes persist, OCRAM is only shifting layout/timing, and your instinct about a latent FreeRTOS/UB bug is right.
Let me know how this test goes and depending on that we can discuss how to move forward.

Hello @Fungiking,

Were you able to get this resolved?

I also did some research on the topic and found out that some parts of the OCRAM are used by the boot ROM and HAB (for secure boot).
Maybe one of those is causing the problem.
See screenshot below from page 706 of the reference manual.

Best Regards,
Bruno

Thanks @bruno.tx — the page-706 figure lines up exactly with what we’re allocating, and I think it’s a live suspect (independent of the cache/ECC and r0p1-errata paths I ranked earlier).

Figure 6-2 (IMX8MPRM.pdf p. 706) reserves the bottom 96 KB of OCRAM for the boot ROM / HAB scratch:

OCRAM (system view) Purpose
0x00900000 – 0x00918000 (96 KB) RESERVED FOR ROM / HAB
0x00918000 – 0x0097FFFF (416 KB) OCRAM FREE AREA

with the caveat “If no ROM/HAB APIs are being used, the entire OCRAM region can be used freely after the boot.”

The Cortex-M7 sees this OCRAM aliased at 0x20200000 (via FSL_MEM_M7_OCRAM_BEGIN). Overlaying that on my current MIMX8ML8xxxLZ_ram-minimal.ld:

linker region M7-local system view inside ROM zone?
m_ocram_data, 128 KB 0x20200000 – 0x20220000 0x00900000 – 0x00920000 first 96 KB overlaps RESERVED FOR ROM
m_ocram_text_lo, 128 KB 0x20220000 – 0x20240000 0x00920000 – 0x00940000 free area
m_ocram_text_hi, 64 KB 0x20240000 – 0x20250000 0x00940000 – 0x00950000 free area, below BL2/BL31

So .data* from communication/, monitor/, worker/, fsm/, input_output/, linear_motor/, pose_control/, and drivers/linear_motor/ is landing in ROM/HAB scratch. board-minimal.c MPU Region 7 correctly fences off the top (SRD=0xE0 excludes BL2/BL31) but nothing protects the bottom — the linker just places into it.

Match against the two acceptance filters:

  • A (variable MTBF): :white_check_mark: HAB/ROM calls are event-driven and rare — corruption only when one fires.
  • B (TCM-stable 1.5 y): :white_check_mark: With .data* in DTCM this was invisible; moving RTOS state into that same OCRAM makes any HAB API call catastrophic.

Fix — shift m_ocram_data above the reservation and mark the reserved area explicitly:

/* Bottom 96 KB of OCRAM is boot-ROM / HAB scratch (IMX8MPRM Rev.D Fig 6-2, p.706).
 * The ROM re-uses this window whenever any ROM/HAB API is invoked from A53 (U-Boot,
 * TF-A, Linux imx_hab / OP-TEE, secure OTA verification, ...). Any RTOS state placed
 * here can be silently overwritten. Declare the region so nothing accidentally lands
 * in it, and start m_ocram_data above it. */
m_ocram_rom_reserved   (RW)  : ORIGIN = 0x20200000, LENGTH = 0x00018000 /*  96 KB, ROM/HAB — DO NOT USE */
m_ocram_data           (RW)  : ORIGIN = 0x20218000, LENGTH = 0x00008000 /*  32 KB */
m_ocram_text_lo        (RWX) : ORIGIN = 0x20220000, LENGTH = 0x00020000 /* 128 KB, unchanged */
m_ocram_text_hi        (RWX) : ORIGIN = 0x20240000, LENGTH = 0x00010000 /*  64 KB, unchanged */

MIMX8ML8xxxLZ_ram-minimal.patched.ld (10.8 KB)

Nothing in SECTIONS { … } > m_ocram_rom_reserved — the region is declared just so no wildcard can drop into it.

If 32 KB is too tight for .data*, collapse data + text_lo into a single 160 KB pool from 0x20218000:

m_ocram_pool  (RWX) : ORIGIN = 0x20218000, LENGTH = 0x00028000 /* 160 KB, data + text */
m_ocram_text_hi (RWX) : ORIGIN = 0x20240000, LENGTH = 0x00010000 /*  64 KB */

MPU / cache side: Region 7 (board-minimal.c:153) can stay as-is. The 96 KB reserved window is still under an enabled sub-region marked WBWA cacheable, but if the linker never places code or data below 0x20218000, the D-cache never fills a line from there and the I-cache never prefetches into it (Cortex-M7 cache lines are 32 B and naturally aligned — 0x20218000 is well aligned).

ocram_ecc_scrub.c needs one constant sync — OCRAM_DATA_END stays 0x20220000 (unchanged, the region still ends there), and __ocram_data_end__ from the linker still marks the last used byte; the scrub loop from marker to OCRAM_DATA_END remains correct.

Confirming this is the cause on the current build (before the linker change):

  1. grep -RiE 'imx_hab|hab_status|HAB_API|caam_hab' u-boot/ linux/ trusted-firmware-a/ tee-* on your BSP — if HAB is invoked at runtime, this hypothesis is live.
  2. On the next crash: if BFAR / MMFAR sits in 0x2020_0000 – 0x2021_7FFF (or the A53 side reports an ECC/data abort in 0x0090_0000 – 0x0091_7FFF), confirmed.
  3. If nothing on A53 ever calls a ROM/HAB API after M7 boot, the region is de-facto safe by the manual’s own caveat — but the linker fix is still the correct-by-construction move.

@Fungiking — did you make progress on the >8 h MTBF case? The community team is happy to help correlate BFAR/MMFAR captures if you have any.

Good evening everyone,

We finally found it, and I wanted to close the loop here because both of your last answers pointed us in useful directions, even if the actual cause turned out to be something else, we think.

Short version: a data read whose target lies in OCRAM can never come back. Apparently, the bus has no timeout, and a missing response is indistinguishable from a slow one, so the core just waits on that one instruction. Nothing faults, which is why we never got a crash dump, and why our configAssert print-outs never fired. If the read happens while the task holds a critical section, the tick dies with it and the whole system goes quiet; otherwise only that task hangs and everything else keeps running. That is the same bug in both flavours, and it explains why our uptime varied from 5 minutes to 8 hours: which read is the unlucky one depends on the byte-exact layout, so every relink moves or hides it. At one point adding a single printf “fixed” it for 1.5 weeks.

What finally convinced us was watching a frozen board from Linux via /dev/mem (the M7 task stacks are in DTCM, so a preempted task’s registers are readable from the A53, no JTAG needed). Our FSM task sat for 84 minutes on this one instruction:

20221da0:  ldr r1, [r4, #20]      ; r4 = 0x202007e8, so it reads 0x202007fc

0x202007fc is just a const char* field of a struct we had put in OCRAM. Over 4000 samples the saved pc never moved, and r0 still held the value written by the instruction before the load. Meanwhile the A53 could read 0x202007fc without any problem, and the bytes were exactly what we had linked.

Then we did the experiment that settled it: we wrote the word the A53 could read into the task’s saved r1, moved its saved pc past the load, and let it continue.

2026-09-04T13:33:38  FSM task: requested change state to 1
                     ... 84 minutes stuck on that load ...
2026-09-04T14:57:58  FSM task: change state to 'GESTURE CTL'

It woke up, printed the log line it had been trying to print, ran five more instructions, and got stuck again 4 bytes further on, at 0x202007f8. A read of 0x20201ff8 in the same OCRAM region worked fine in between, and so did a write. So it is not the whole region that is broken; it is something much smaller: two addresses 4 bytes apart both wedge, while one 6 KB away in the same window is fine. 32 bytes would match the M7’s cache line, but we cannot claim it is the line fill, because the freezes also happened back when we had OCRAM mapped non-cacheable, and then there are no line fills to stall. So we guess it is a bus burst or something inside the OCRAM controller.

To your answers:

  • @bruno.tx — you were right, and we had missed it. Figure 6-2 in IMX8MPRM Rev. 3 really does reserve phys 0x00900000-0x00918000 for the boot ROM, and our m_ocram_data started exactly at 0x20200000 = phys 0x00900000, so its first 96 KB overlapped that area. We are moving it. But it is not what froze our board: ROM/HAB writing there would corrupt our data, and we read those bytes during the freeze, and they were bit-identical to the linked image. A corrupted value would also have been read successfully and just been wrong, while this load never completes at all. The note under that figure also limits the reservation to boot unless ROM/HAB APIs are called at runtime.
  • @dcasota — your address arithmetic was correct and worth acting on for the same reason, thank you for digging that out. The erratum 837070 and OCRAM ECC ideas we had already ruled out by experiment: no fault handler ever runs (ours are linked, and our panic buffers in DTCM are empty), and it reproduces with the core clock halved to 400 MHz, so it is not marginal timing either.
  • @rudhi.tx — not cache coherency, unfortunately. It also fails with the OCRAM MPU region set non-cacheable.

Our fix is one rule: OCRAM holds code only, no readable data. The linker keeps .data/.bss/.rodata out of the OCRAM regions, but it can only move whole sections, so it cannot touch what the compiler or the linker puts inside .text. Three flags take care of that:

-mslow-flash-data   literal pools       -> constants built with movw/movt
-fno-jump-tables    switch jump tables  -> compare-and-branch
-mlong-calls        long-branch veneers -> call target built with movw/movt + blx

The one thing we would still like to know, and maybe someone from NXP can answer: is a data read to OCRAM while the M7 is fetching instructions from OCRAM a documented restriction on the i.MX8MP, or is this a silicon issue? Mixing code and constants in one memory is completely normal on every other MCU we have used, and we could not find anything in the ARM or NXP documentation that forbids it. Our fix works either way, but the answer decides whether the OCRAM data window is usable at all. We had assumed it was safe precisely because nothing executes there, and the capture above shows that assumption was wrong.

Happy to share the register-level captures if that helps anyone.

Thank you very much for all your input, it really helped.

Sincerely,

Chris & Team

Hello @Fungiking,

Thanks a lot for the update and extensive testing you have shared here.

While looking into what you mentioned and possible known issues with the OCRAM, I actually found out that another part of the OCRAM is used for the ATF.
The ATF is loaded to 0x97000, so the freely-usable portion also gets squeezed from the top:

While the way that you avoided the issue strongly indicates that the problem was not caused by the use of such areas of OCRAM, I think it would be best to confirm this.
Do you think you could do such test after moving the data out of the HAB/Boot ROM and ATF areas of the OCRAM?

Best Regards,
Bruno

Hi @bruno.tx

Thanks for the tip. We already had this region excluded in our linker file, which we used when the freezes happened. So this is unfortunately not the fix.

Best,

Chris

Hi @Fungiking,

Just to confirm, here you are talking about the ATF region at the end of the TCM, right?

Because in the example you showed of the failure, the data is within the HAB/Boot ROM area.

Best Regards,
Bruno

Hi @bruno.tx

Yes, exactly — the ATF region you mentioned in your last reply. The HAB/Boot ROM area we only excluded recently, after your post.

However, four of our seven captures stalled on addresses in the OCRAM FREE AREA of Figure 6-2, so above the HAB/Boot ROM region and well below ATF:

#1  sys 0x009226a4   const table
#2  sys 0x009206a8   log string
#5  sys 0x00922756   const table, second board
#6  sys 0x009206ae   log string, second board

As far as we know, nothing else claims those addresses, so neither reserved region can explain those four. (Two further captures had a dead tick, so their frames were stale and we could not establish the target — I am not counting those. The seventh is the one inside the HAB/Boot ROM area that I showed earlier.)

Best regards,
Chris

Hello @Fungiking,

In these four examples, you are correct that the address where the processor gets stuck does not overlap the HAB/Boot ROM region.
However, I would still like to see one run where the Cortex-M touches none of the reserved OCRAM regions at all. With the tests so far we don’t have a configuration that would be unarguably “correct”.

I don’t think this is the case, but I could imagine that other addresses may end up being used by HAB or the Boot ROM if, for example, a pointer in its reserved region is modified externally.
And I would expect that without a clean example where such region was not touched at all we could still not be 100% sure that is not happening.

Best Regards,
Bruno

Hi @bruno.tx

Unfortunately, we are very close to a release cycle, so at the moment we don’t have the time to do further testing until the end of October. If no other insight has been found until then, I will gladly restart our testing with the all-correct layout again.

Best,

Chris & Team

Hello @Fungiking,

No problem, thanks for the update.
If I have time on my side I will try to create a minimum firmware that should reproduce the issue based on your description to do the test.

Best Regards,
Bruno