iMX6ULL_WB_IT kernel panic

I have a design that uses the iMX6ULL_WB_IT V1.1A SOM.

I’m using the procedure outlined on the Toradex website, titled, “Load Toradex Easy Installer From USB OTG” to perform initial programming of the SOMs.

After running “recovery-windows.bat”, I often get a kernel panic and the process stops.

Sometimes, if I get no kernel panic after running “recovery-windows.bat”, I will get it after typing in “ifconfig”. And, again, the process stops.

The failure seems related to Ethernet, but I’m not 100% sure. It’s my understanding that the V1.1A SOM has a switch that switches 3.3V to the Ethernet PHY. I think I’ve located the switch

on the SOM. If I jumper around the switch, the kernel panics no longer occur. I wonder if the switch is causing the issue I’m seeing.

This is a persistent problem for me. I’ve gone to the trouble of isolating the SOM from my main board - the only SODIMM pins connected to my main board are 3.3V, GND, Ethernet data lines,

USB_C, and UART_A. Even in this isolated condition, I still get kernel panics. Any ideas as to why the kernel is panicing?

I would like to get this update procedure working over Ethernet. Ethernet(not SD card) is the preferred initial programming method for our manufacturing process. A sample kernel panic dump is attached.

Karllink text

Hi @kkracht

Could you provide the Software version of your module? Which carrier board are you using?
Please share the dmesg log in a File?

Thanks and best regards, Jaski

We’re using the baseline Linux 2.8b4 image for the Colibri iMX6ULL. This problem is happening inside of the latest stable release of the Toradex Easy Installer for the iMX6ULL.

Text file with dmesg log, followed by kernel panic is attached: link text

The carrier board is our own design.

Karl

I used an oscilloscope to monitor the 3.3V input to the SOM and the switched 3.3V rail to the SOM’s ethernet PHY. I see the ethernet PHY 3.3V rail go up and down as the SOM is booting up and making its way toward the TEZI prompt. see attached oscilloscope shot. The yellow trace is the SOM’s main 3.3V rail. The blue trace is the 3.3V rail to the ethernet PHY.

alt text

Karl

Hi @kkracht

Thanks for your Input.

Is the module working on a Toradex Carrier Board?
Could you try to install using a Toradex Carrier Board and install the Bsp 2.8?
Is the usual Linux Booting?

Could you provide a schematic of the connections between your carrier board and the SODIMM Connector?

The yellow trace is the SOM’s main 3.3V rail. The blue trace is the 3.3V rail to the ethernet PHY.

Which SODIMM Pins are you measuring?

Could you provide the serial number of your module?

Thanks.

The TEZI installer does work when the SOM is mounted on a Toradex dev board.
I’ll inquire about installing BSP 2.8.
I can’t share schematic pieces in a community forum. I could email it to you directly.
In the oscilloscope picture, below, both traces are measured on the SOM. The yellow trace is 3.3V, as provided to the SOM by my baseboard. The blue trace is the switched 3.3V to the SOM’s Ethernet PHY. There is a switch on the SOM that turns the Ethernet PHY’s 3.3V rail on and off.
The serial number of the SOM is 06398997. The issues I’m seeing aren’t specific to one SOM. Almost all of our SOMs exhibit this behavior. Some worse than others. They work when programmed on the Toradex dev board. To get the SOMs to program while plugged into our custom baseboard, I had to jumper around the switch that switches the Ethernet PHY’s 3.3V rail. For some reason, this makes the problem go away.

This issue is also being looked into by another Toradex person. I emailed tech support directly, inquiring about the Ethernet power switch, mentioned above. I just thought you should know.

Karl

We were able to install BSP 2.8 successfully on the SOM using the Toradex dev kit. The dmsg file is attached.

link text

There seems to be no errors. Which version of Toradex Easy Installer are you using?

we use Tezi version 1.8

Karl

So this means the issue is only with Tezi on your custom carrier board. Did you send the schematic of your carrier board to @alex.tx?

Yes, Alex has the schematic.

Karl

I did a quick review of your schematic. The Ethernet PHY connection looks OK. This issue could be related to a problem with insufficient decoupling and/or power traces layout. So when you enable Ethernet the power consumption spikes causing glitches. Please refer to our Carrier Board Layout Guide and Colibry Carrier Board Design Guide https://developer.toradex.com/carrier-board-design/carrier-board-design-guides .

Always enabled power for Etehrnet (jumper fix) should not create any problem except extra power consumption.

Issue solved.

I solved the kernel panic issue. It may require a fix by Toradex.

SODIMM pins 183(Ethernet Link/Activity Status) and 185(Ethernet Speed Status) must be tied to valid logic levels. They cannot be allowed to float.
Allowing these pins to float can cause kernel panics during the TEZI install process. Even if the kernel doesn’t panic, the SOM’s Ethernet PHY(eth0) will not show up as a recognized device when entering the “ifconfig” command at the TEZI prompt.

This behavior has been proven by running the TEZI installer(v 1.8) with two Toradex iMX6ULL_WB_IT 1.1A SOMs on our custom carrier board. It has also been proven by running the TEZI installer with these SOMs on the Toradex Colibri Evaluation Board. In the latter case, R24 and R25, series resistors on Ethernet speed and activity LEDs, were removed. When the TEZI installer was executed, one SOM did not show “eth0” as a recognized device when the “ifconfig” command was entered. The second SOM experienced a kernel panic before the “ifconfig” command was entered.

There is no mention in the iMX6ULL datasheet about strapping these pins to valid logic levels. The iMX6ULL hardware design guide says all unused Ethernet pins can be left floating. The datasheet for the SOM’s Ethernet PHY(KSZ8041NL) says there are internal pullups on these pins during powerup and reset. So no external pullup or pulldown resistors should be necessary. At some point in the TEZI process, these pins aren’t being sampled properly.

This issue may be a consequence of switching the SOM’s Ethernet PHY power rails on and off. In our case, we found that jumpering around the switch on the PHY’s 3.3V rail made the kernel panics go away.

There are at least two solutions to this problem:
1) pull SODIMM pins 183 and 185 to valid logic levels at all times. For most applications, this likely means pulling both of them to logic high.
2) overwrite the latched values for these pins whenever the PHY is reset, or its power is switched on.

Our application for the iMX6ULL is low power, so we don’t populate any LEDs for production. That’s how the issue revealed itself to us. By not populating the Ethernet link status and speed LEDs, the PHY’s LED0 and LED1 pins are floating. LED0 enables or disables auto negotiate. LED1 sets the advertised link speed during the auto negotiate process.

This issue could also exist on your other your SOMs if their Ethernet power rails are switched on and off, like the iMX6ULL.

I’ll leave it to a Toradex person to provide the final “answer” response to this thread, in case they want to include specific verbiage.

Karl

Thanks for update. We will investigate this issue.