I’ve just found a strange issue with Verdin iMX8M-Plus running Torizon OS on A53-cores and a custom firmware on M7-core.
M7 firmware is loaded and started bu u-boot, and everything is ok.
As soon I plug an ethernet cable into the eth0 (both on Yavia board and on my custom carrier board), M7 crashes immediately wi no info on M7 side.
Has someone seens a similar issue?
Which kind of log should I take to help the investigation?
This is a major blocking issue, becasue in this way I cannot use my system.
Hi, I’m the Toradex AI Assistant. I’m in early beta, and I would appreciate your feedback.
The Toradex Support team will follow up with you shortly!
Did you like my answer?
Handshake,
The issue you’re encountering with the Verdin iMX8M Plus, where plugging in an Ethernet cable causes the M7 core to crash, is indeed unusual and suggests a potential conflict or resource contention between the M7 firmware and the A53 core’s handling of Ethernet connectivity.
To begin investigating this issue, consider the following steps:
Review M7 Firmware: Ensure that the M7 firmware does not inadvertently configure or use resources that are also used by the Ethernet driver running on the A53 cores. This includes GPIOs, clocks, or peripheral configurations that might be shared between the cores.
Check Power Management: Since the Ethernet PHY’s power is controlled via a GPIO expander, as noted in your device tree snippet, ensure that there is no power management code in the M7 firmware that could interfere with the power state of the Ethernet PHY or related components.
Isolate the Issue: Try to isolate whether the crash is directly related to the act of Ethernet cable insertion or if it’s related to network activity that begins once the connection is established. This could help narrow down the cause to either electrical (e.g., a surge or grounding issue) or software (e.g., interrupt handling or resource contention).
Logging and Debugging:
M7 Core Debugging: Use a debugger connected to the M7 core to catch the crash in real-time. This could provide insights into the crash location and the state of the M7 core at the time of the crash.
Kernel Logs: Check the kernel log (dmesg) for any messages that occur at the time of the Ethernet cable insertion. This might reveal errors or warnings from the Ethernet driver or related subsystems.
Network Activity Logs: Use network debugging tools (e.g., tcpdump) to monitor any network activity that occurs upon Ethernet connection. This might help identify if specific network traffic is triggering the issue.
Review Electrical Design: If the issue persists across different carrier boards, review the electrical design around the Ethernet PHY, especially the power control and signal integrity aspects. Ensure that the design adheres to the recommendations in the Verdin iMX8M Plus datasheet and the reference design.
Firmware and OS Updates: Ensure that both the Torizon OS and the M7 firmware are up to date. It’s possible that this issue has been encountered and addressed in a newer software release.
Given the complexity of the issue, a systematic approach to
Please, also check if these links can help you solve your problem:
Cortex-M7 firmware uses a lot of peripherals, but I ask the opposite.
Which peripherals use eth0 in Torizon OS?
In this way I can check that M7 firmware doesn’t use any conflicting peripherals.
How do you reach the conclusion that the Cortex-M7 firmware has stopped?
Does it stop performing an action or does the serial output stop working?
I think this may actually be related to GPIO banks, as you may be using a GPIO bank on the Cortex-M which is needed by the Ethernet Phy.
Can you reproduce this problem with a simple Hello World example?
I blink some leds from M7 and I see they don’t blink anymore.
I have rpmsg communication in place and I see that Torizon OS doesn’t receive any answer anymore.
The issue happens only on ETH0 (the one with the PHY on the SoM). ETH1 (with PHY on my carrier board) works.
I’m going to do other tests with the Yavia and a “standard” Torizon OS build to see what happens.
The GPIO bank is a hot topic and I’ve already had an answer from @rudhi.txhere.
I know that Toradex did a great job with the SoMs and Torizon OS and I appreciate it really much. But it’s clear that the interaction between the cores has been under-estimated both from the hardware and the software point of view.
I need a set of GPIO pins on M7 (and I think that if a user involves M7 it has real-time needs, and probably GPIO pins is one of the most common real-time needs) and so I need a clear answer on which GPIO pins are used by Torizon OS. In this way I can refactor my carrier board and use other pins on M7, but I need one final answer.
Thanks for the explanation, it seems likely that Linux is taking ownership over some resource which is in use by the Cortex-M7 core.
That is understandable, but unfortunately there is no straight-forward answer.
Most peripherals are configured to be used on Linux.
In general I would say the best approach would be to assume that the peripheral that you want to use on the Cortex-M core is somehow used on Linux and check how you could disable it.
For this process, I usually use the find_in_dt tool, which is not a tool from Toradex, but was developed by one of our FAEs and makes it much easier to navigate device trees.
If you want, I can check your current configuration and see what may be conflicting on the Linux side.
For that, I would need you to send me the pins and interfaces which are currently in use in your Cortex-M program and your device tree customization/overlays.
If you don’t want to send this in a public thread, please feel free to send a private message to the moderators here on the Community or an email via support.eu@toradex.com.
@bruno.tx I’ve just found that this error is the same already reported by @aditya.yantra in this topic.
I’m going to do the checks suggested by @hfranco.tx in that topic and I’ll post here the results.
I boot my Yavia with ethernet cable unplugged and I see that rpmsg works as expected. Importan note:
My M7 firmware doesn’t riun from ITCM memory, but from DDR (because it’s big).
Note sure if this has an impact, or not.
The first lines are the normal logging of rpmsg, then the first log is [ 170.717948] imx-rproc imx8mp-cm7: imx_rproc_kick: failed (1, err:-62)
just before the ethernet stuff
Jul 22 10:01:10 verdin-imx8mp-15128316 systemd-networkd[678]: ethernet0: Gained carrier
Jul 22 10:01:10 verdin-imx8mp-15128316 NetworkManager[701]: <info> [1721642470.9470] device (ethernet0): carrier: link connected
Jul 22 10:01:10 verdin-imx8mp-15128316 NetworkManager[701]: <info> [1721642470.9481] device (ethernet0): state change: unavailable -> disconnected (reason 'carrier-changed', sys-iface-state: 'managed')
Jul 22 10:01:10 verdin-imx8mp-15128316 kernel: imx-dwmac 30bf0000.ethernet ethernet0: Link is Up - 100Mbps/Full - flow control rx/tx
Jul 22 10:01:10 verdin-imx8mp-15128316 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ethernet0: link becomes ready
...
For this reason I suspect some hardware or device tree conflict (some clock, the DDR, …?)
If I try to run # strace modprobe imx_rpmsg_tty
I get the error sh: strace: command not found
The imx_rproc_kick: failed error indicates that the Cortex-M7 is not responding to the signals from remoteproc.
So it may be reset or halted in some way do to a conflict in resource usage.
I will have a look at the information you sent regarding the used pins and device tree to find out more about what may be the conflict.
After looking at the used GPIO pins and the device tree, there are two possible pins which are used as GPIO and are related to the on-board ethernet phy:
GPIO1_IO10 - Ethernet Interrupt (input)
GPIO2_IO20 - Used to enable the Ethernet power (output)
Considering the problem does not happen on startup, but when the Ethernet cable is plugged in, I believe this problem is related to GPIO1_IO10, the interrupt signal.
To confirm this, can you test this with a Cortex-M7 firmware which does not initialize any pins on GPIO1?
If the problem is still present, we can continue looking for the cause.
Otherwise, the solution may be to delete the interrupt and interrupt-parent properties from the ethphy0 node in the device tree.
These are not required fields, but needed to support interrupts from the Ethernet phy.
Hello @bruno.tx
I’ve double checked my M7 source code and I found a mistake: interrupt for GPIO1 pins were partially configured.
I can configure GPIO1 pins as either inputs or outputs , but I cannot configure interrupt on these pins.
I removed this partial configuration of interrupts and everything seems ok now.
Basically, on M7 core I can enabled GPIO interrupts for GPIO3 and GPIO5 only.
The only compromise is that I must disable I2C recovery feature too for I2C peripherals shared with GPIO5 pins (but I don’t need this feature).
