Enable 2nd eth in device-tree for Apalis i.MX8QM with Micrel KSZ9031RNX as phy

What do we want to do?

We want to enable the second ethernet port and get it working properly.

Hardware/Software setup:

  • Baseboard: custom, but based on the Ixora reference design
  • Second PHY on Baseboard: Micrel KSZ9031RNX
  • SoM: Apalis i.MX8QM V1.1B
  • BSP: 5.5.0

Relevant part of the schematics:

Not seen here:

  • RGMI_(MDC|MDIO) go through a level changer before connecting to the phy to bringing them down to 1.8V
  • Some RGMII* pins that are int the range of pins for MMC1 and UART2
  • ETH_RESET connects to Apalis: pin 26, function RESET_MOCI#
  • ETH_INT connects to Apalis: pin 159, function TS_6 (MMC1)

What did we try?

We tried to create a device-tree overlay as follows (Note: ECU refers to our custom base board):

#include <dt-bindings/pinctrl/pads-imx8qm.h>
#include <dt-bindings/gpio/gpio.h>
#include <dt-bindings/interrupt-controller/irq.h>
#include <dt-bindings/firmware/imx/rsrc.h>

/dts-v1/;
/plugin/;

/* ECU: eth1 */
/* we "just" try to mimic the pin config we see on fec1 */

/*
* in general, you need to look up the pin config field for fsl,pins
* in the "imx8 quad max applications processor manual" under "IOMUXD"
* (the "source part" of the mux setting ususally matches pretty well)
* 
* you may also check http://git.toradex.com/cgit/linux-toradex.git/tree/Documentation/devicetree/bindings/pinctrl/fsl,imx8qm-pinctrl.txt?h=toradex_4.9-2.3.x-imx
* for the device tree binding doc, what they call _hw_pad_iomux_28fdsoi is what I believe to be the 90+% case
* it matches up with the stuff I describe below
*
* bits we usually care about (0-indexed):
* 26-25 sw_config -> 00 default, 01 open drain, 10 open drain input, 11 inout
* 6-5   PULL      -> 00 illegal, 01 pull-up, 10 pull-down,  11 pull-disabled
* 0     PDRV      -> 0 high drive strength, 1 low drive strength
*
* common values:
* 0x2P -> pull up, P = PDRV, 0/1
* 0x4P -> pull down, P = PDRV, 0/1
*/

&iomuxc {
	pinctrl-names = "default";
	/* NOTE: most of these labels come from imx8-apalis-v1.1.dtsi */
	pinctrl-0 = <&pinctrl_cam1_gpios>, <&pinctrl_dap1_gpios>,
		<&pinctrl_esai0_gpios>, <&pinctrl_gpio3>, <&pinctrl_gpio4>, <&pinctrl_gpio_keys>,
		<&pinctrl_gpio_usbh_oc_n>, <&pinctrl_lpuart1ctrl>,
		<&pinctrl_lvds0_i2c0_gpio>, <&pinctrl_lvds1_i2c0_gpios>,
		<&pinctrl_mipi_dsi_0_1_en>, <&pinctrl_mipi_dsi1_gpios>,
		<&pinctrl_mlb_gpios>, <&pinctrl_qspi1a_gpios>,
		<&pinctrl_sata1_act>, <&pinctrl_sim0_gpios>,
		<&pinctrl_usdhc1_gpios>, <&pinctrl_pwm2>;

	ecu {
		pinctrl_fec2: fec2grp {
			fsl,pins = <
			/* pin conf: special case, see IMOUXD_COMP_CTL_GPIO_1V8_3V3_SIM in manual
			* bits set, rest is whatever 0 means:
			* RASRCP = reset value, RASRCN = reset value
			* corresponds to _hw_pad_iomux_28fdsoi_comp in dt binding doc

			* Apalis comment for ENET0:  Use pads in 3.3V mode
			* Not sure where they get that info from...
			*/
			/* NOTE: weirdly, fec1 is ENETB_PAD, and fec2 is (by exclusion) ENETA_PAD */
			IMX8QM_COMP_CTL_GPIO_1V8_3V3_ENET_ENETA_PAD	0x000014a0

			/* Apalis: pin 253, function LCD1_R1 */
			/* IMX8QM_ENET1_MDC_LSIO_GPIO4_IO18		0x00000021 */
			/* pin conf: sw_config inout, pull-up, high drive */
			IMX8QM_ENET1_MDC_CONN_ENET1_MDC                 0x06000020

			/* Apalis: pin 251, function LCD1_R0 */
			/* IMX8QM_ENET1_MDIO_LSIO_GPIO4_IO17		0x00000021 */
			/* pin conf: sw_config inout, pull-up, high drive */
			IMX8QM_ENET1_MDIO_CONN_ENET1_MDIO		0x06000020

			/* Apalis: pin 263, function LCD1_R6 */
			/* IMX8QM_ENET1_RGMII_TX_CTL_LSIO_GPIO6_IO11	0x00000021 */
			/* pin conf: sw_config inout, pull-up, high drive */
			IMX8QM_ENET1_RGMII_TX_CTL_CONN_ENET1_RGMII_TX_CTL	0x06000020

			/* Apalis: pin 261, function LCD1_R5 */
			/* IMX8QM_ENET1_RGMII_TXC_LSIO_GPIO6_IO10		0x00000021 */
			/* pin conf: sw_config inout, pull-up, high drive */
			IMX8QM_ENET1_RGMII_TXC_CONN_ENET1_RGMII_TXC	0x06000020

			/* Apalis: pin 259, function LCD1_R4 */
			/* IMX8QM_ENET1_RGMII_TXD0_LSIO_GPIO6_IO12		0x00000021 */
			/* pin conf: sw_config inout, pull-up, high drive */
			IMX8QM_ENET1_RGMII_TXD0_CONN_ENET1_RGMII_TXD0	0x06000020

			/* Apalis: pin 257, function LCD1_R3 */
			/* IMX8QM_ENET1_RGMII_TXD1_LSIO_GPIO6_IO13		0x00000021 */
			/* pin conf: sw_config inout, pull-up, high drive */
			IMX8QM_ENET1_RGMII_TXD1_CONN_ENET1_RGMII_TXD1	0x06000020

			/* Apalis: pin 255, function LCD1_R2 */
			/* IMX8QM_ENET1_RGMII_TXD2_LSIO_GPIO6_IO14		0x00000021 */
			/* pin conf: sw_config inout, pull-up, high drive */
			IMX8QM_ENET1_RGMII_TXD2_CONN_ENET1_RGMII_TXD2	0x06000020

			/* moved from lpuart3 */
			/* Apalis: pin 128, function LCD1_R2 */
			/* pin conf: sw_config inout, pull-up, high drive */
			/* IMX8QM_ENET1_RGMII_TXD3_DMA_UART3_RTS_B		0x06000020 */
			IMX8QM_ENET1_RGMII_TXD3_CONN_ENET1_RGMII_TXD3	0x06000020

			/* Apalis: pin 130, function LCD1_R2 */
			/* pin conf: sw_config inout, pull-up, high drive */
			/* IMX8QM_ENET1_RGMII_RXC_DMA_UART3_CTS_B		0x06000020 */
			IMX8QM_ENET1_RGMII_RXC_CONN_ENET1_RGMII_RXC	0x06000020

			/* Apalis: pin 265, function LCD1_R7 */
			/* IMX8QM_ENET1_RGMII_RX_CTL_LSIO_GPIO6_IO17	0x00000021 */
			/* pin conf: sw_config inout, pull-up, high drive */
			IMX8QM_ENET1_RGMII_RX_CTL_CONN_ENET1_RGMII_RX_CTL	0x06000020

			/* Apalis: pin 249, function LCD1_DE */
			/* IMX8QM_ENET1_RGMII_RXD0_LSIO_GPIO6_IO18		0x00000021 */
			/* pin conf: sw_config inout, pull-up, high drive */
			IMX8QM_ENET1_RGMII_RXD0_CONN_ENET1_RGMII_RXD0		0x06000020

			/* Apalis: pin 247, function LCD1_HSYNC */
			/* IMX8QM_ENET1_RGMII_RXD1_LSIO_GPIO6_IO19		0x00000021 */
			/* pin conf: sw_config inout, pull-up, high drive */
			IMX8QM_ENET1_RGMII_RXD1_CONN_ENET1_RGMII_RXD1		0x06000020

			/* Apalis: pin 245, function LCD1_VSYNC */
			/* IMX8QM_ENET1_RGMII_RXD2_LSIO_GPIO6_IO20		0x00000021 */
			/* pin conf: sw_config inout, pull-up, high drive */
			IMX8QM_ENET1_RGMII_RXD2_CONN_ENET1_RGMII_RXD2		0x06000020

			/* Apalis: pin 243, function LCD1_PCLK */
			/* IMX8QM_ENET1_RGMII_RXD3_LSIO_GPIO6_IO21		0x00000021 */
			/* pin conf: sw_config inout, pull-up, high drive */
			IMX8QM_ENET1_RGMII_RXD3_CONN_ENET1_RGMII_RXD3		0x06000020

			/* Apalis: pin 269, function LCD1_G0 */
			/* IMX8QM_ENET1_REFCLK_125M_25M_LSIO_GPIO4_IO16	0x00000021 */
			/* pin conf: sw_config inout, pull-up, high drive */
			IMX8QM_ENET1_REFCLK_125M_25M_CONN_ENET1_REFCLK_125M_25M	0x06000020


			/* For ENET0: Apalis: pin "on module", function ETH_RESET# */
			/* FOR ENET1: ECU is wired directly to (Apalis: pin 26, function RESET_MOCI#) */
			/* so we don't mux anything for that pin*/
			/* ENET0 mux */
			/* IMX8QM_LVDS1_GPIO01_LSIO_GPIO1_IO11		0x06000020 */

			/* For ENET0: Apalis: pin "on module", function ETH_INT# */
			/* For ENET1: Apalis: pin 159, function TS_6 (MMC1) */
			/* ENET0 mux */
			/* IMX8QM_MIPI_CSI1_MCLK_OUT_LSIO_GPIO1_IO29	0x04000060 */
			/* ENET1 mux */
			/* pin conf: sw_config open drain input, pull-disabled, high drive */
			IMX8QM_USDHC1_DATA6_LSIO_GPIO5_IO21 0x04000060
			>;
		};

		pinctrl_fec2_sleep: fec2-sleepgrp {
			fsl,pins = <
			IMX8QM_COMP_CTL_GPIO_1V8_3V3_ENET_ENETA_PAD	0x000014a0

			IMX8QM_ENET1_MDC_LSIO_GPIO4_IO18		0x04000040
			IMX8QM_ENET1_MDIO_LSIO_GPIO4_IO17		0x04000040
			IMX8QM_ENET1_RGMII_TX_CTL_LSIO_GPIO6_IO11	0x04000040
			IMX8QM_ENET1_RGMII_TXC_LSIO_GPIO6_IO10		0x04000040
			IMX8QM_ENET1_RGMII_TXD0_LSIO_GPIO6_IO12		0x04000040
			IMX8QM_ENET1_RGMII_TXD1_LSIO_GPIO6_IO13		0x04000040
			IMX8QM_ENET1_RGMII_TXD2_LSIO_GPIO6_IO14		0x04000040
			IMX8QM_ENET1_RGMII_TXD3_LSIO_GPIO6_IO15		0x04000040
			IMX8QM_ENET1_RGMII_RXC_LSIO_GPIO6_IO16		0x04000040
			IMX8QM_ENET1_RGMII_RX_CTL_LSIO_GPIO6_IO17	0x04000040
			IMX8QM_ENET1_RGMII_RXD0_LSIO_GPIO6_IO18		0x04000040
			IMX8QM_ENET1_RGMII_RXD1_LSIO_GPIO6_IO19		0x04000040
			IMX8QM_ENET1_RGMII_RXD2_LSIO_GPIO6_IO20		0x04000040
			IMX8QM_ENET1_RGMII_RXD3_LSIO_GPIO6_IO21		0x04000040
			/* IMX8QM_ENET1_REFCLK_125M_25M_LSIO_GPIO4_IO16	0x04000040 */
			/* not wired on ECU */
			/* IMX8QM_LVDS1_GPIO01_LSIO_GPIO1_IO11		0x06000020 */
			IMX8QM_USDHC1_DATA6_LSIO_GPIO5_IO21 0x04000040
			>;
		};
	};
};

/* Disable Apalis fec2 pinctrl, just in case */
/* However, it looks like this is not necessary, because we don't reference that node anymore */
/* and we do not get error messages about pin conflicts at runtime */
/* NOTE: overlays cannot delete nodes, they can only grow the device trees shape, not shrink it */ 
&pinctrl_fec2_gpios {
	status = "disabled";
};


/* Disable Apalis UART2, which is UART3 on the CPU */
/* some pins overlap with ENET1, so we just disable it and hope for the best */
&pinctrl_lpuart3 {
	status = "disabled";
};

/* Disable Apalis MMC1, which is USDHC2 on the CPU */
/* some pins overlap with ENET1, so we just disable it and hope for the best */
&pinctrl_usdhc2 {
	status = "disabled";
};

&lpuart3 {
	status = "disabled";
};

&usdhc2 {
	status = "disabled";
};

&reg_ext_rgmii {
	regulator-min-microvolt = <1800000>;
	regulator-max-microvolt = <1800000>;
	power-domains = <&pd IMX_SC_R_BOARD_R2>;

	regulator-state-mem {
		regulator-off-in-suspend;
	};
};

&fec2 {
	pinctrl-names = "default", "sleep";
	pinctrl-0 = <&pinctrl_fec2>;
	pinctrl-1 = <&pinctrl_fec2_sleep>;

	fsl,magic-packet;
	fsl,mii-exclusive;

	phy-supply = <&reg_ext_rgmii>;
	phy-handle = <&ethphy1>;

	phy-mode = "rgmii-id";
	/* TODO: we don't control reset, as it's wired directly to SoM reset */
	/* deprecated and optional, should be defined in phy? */
	/* phy-reset-duration = <10>; */
	/* phy-reset-gpios = <&lsio_gpio0 8 GPIO_ACTIVE_LOW>; */
	status = "okay";

	mdio {
		#address-cells = <1>;
		#size-cells = <0>;

		/* we just name it @8 to avoid a conflict with ethphy0 which is @7 */
		/* should we decide to move the ethphy1 under the same mdio node as ethphy0 */
		ethphy1: ethernet-phy@8 {
			compatible = "ethernet-phy-ieee802.3-c22";
			interrupt-parent = <&lsio_gpio5>;
			interrupts = <21 IRQ_TYPE_LEVEL_LOW>;
			micrel,led-mode = <0>;

			/* don't set a reg, because we don't know it yet, kernel will scan all phy ids */
			/* once we know it, we can hardcode it */
			/* reg = <7>; */
		};
	};
};

Notes

  • to avoid pin conflicts at run-time we had to overwrite &iomuxc rather than just overriding and using pinctrl_fec2_gpios

What works, what doesn’t

We get no pin conflict messages from the iomux driver, the kernel picks up the second phy and tries to initialize it. Since we do not set a reg property, it tries to probe all 32 possible phy addresses. However in contrast to fec1, it never finds one.
We drilled down into the kernel and the first 2 reads on the mdio bus return garbage (mostly 1 bits).
The kernel then rejects that address an tries the next one, until it finally hits the last one and gives up.

We also measured the connection with an oscilloscope where we could verify that Apalis: pin 251 (which we mux to ENET1_MDIO) indeed never changes levels.
The kernel thinks it’s reading on the bus, but it really isn’t.

To verify that the physical connection is OK, we muxed the pin to GPIO and toggled it every second from the cmdline. We could verify this with the oscilloscope.

kernel output for fec1

[    0.893973] libphy: Fixed MDIO Bus: probed
....
[    0.896024] fec 5b040000.ethernet: Adding to iommu group 0
[    0.911861] fec 5b050000.ethernet: Adding to iommu group 0
[    1.401109] libphy: fec_enet_mii_bus: probed
[    1.542103] mdio_bus 5b040000.ethernet-1: registered phy ethernet-phy at address 7
[    1.550625] fec 5b040000.ethernet eth0: registered PHC device 0
... after link set up
[    7.287440] Microchip KSZ9131 Gigabit PHY 5b040000.ethernet-1:07: attached PHY driver [Microchip KSZ9131 Gigabit PHY] (mii_bus:phy_addr=5b040000.ethernet-1:07, irq=305)
[   10.807163] fec 5b040000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx

kernel output for fec2 on intit

[    1.601043] mdio_bus 5b050000.ethernet-2: ethernet-phy@8 has invalid PHY address
[    1.626762] mdio_bus 5b050000.ethernet-2: scan phy ethernet-phy at address 0
... trying all addrs 0..31
[    2.077779] mdio_bus 5b050000.ethernet-2: scan phy ethernet-phy at address 31
[    2.093169] fec 5b050000.ethernet eth1: registered PHC device 1
...

kernel output on setting up eth1 (fec2)

root@UNKNOWN-UNKNOWN:~# ip link set up dev eth1
[   35.555849] fec 5b050000.ethernet eth1: Unable to connect to phy
ip: SIOCSIFFLAGS: No such device

What do we expect?

We expect so see at least some activity on the ENET1_MDIO pin. We should be able to verify it on the oscilloscope. Furthermore, the kernel should successfully find and initialize the phy for fec2 at some phy address. Once we know it, we can hardcode it in the device-tree.

Questions

1. What’s the correct placement of ethphy1 in the device-tree?

We saw a lot of device-trees trying to do something similar that defined the ethphy1 node to be a sibling of ethphy0. We think that would mean that they are physically on the same mdio bus.
We do think this is wrong in our case as we connect our phy to ENET1_(MDC|MDIO) rather than ENET0_(MDC|MDIO).

Is this assessment correct? If not, why and what should we do diffrently?

2. What device-tree changes are needed to get the correct voltage levels?

The Apalis iMX8 Datasheet states:

The secondary RGMII/RMII Ethernet interface needs special attention regarding the supply voltage
level. The RGMII/RMII voltage is switchable through LDO1OUT of the second PMIC PF8100. The
voltage level must be defined by software configuration. During the power-up sequence, the IO
voltage is set to 3.3V by default. All the RGMII/RMII signals are configured as GPIO inputs with
enabled pull-down resistors. If the interface is used with 1.8V IO voltage, it is crucial to configure
first the LDO1OUT voltage and then change the alternate function to RGMII/RMII. This ensures full
compatibility with 1.8V RGMII signal levels.

We took that to mean that we should set the regulator of fec2 to something that maps to a 1.8V power domain. Like this:

&reg_ext_rgmii {
	regulator-min-microvolt = <1800000>;
	regulator-max-microvolt = <1800000>;
	power-domains = <&pd IMX_SC_R_BOARD_R2>;

	regulator-state-mem {
		regulator-off-in-suspend;
	};
};
....
&fec2 {
	...
	phy-supply = <&reg_ext_rgmii>;

Note: According to a comment in imx8-apalis-v1.1.dtsi the value IMX_SC_R_BOARD_R2 should be for 1.8V.

Is this correct and complete?

3. Are the pin mux settings correct?

4. Is not having a kernel controlled ETH_RESET a problem?

Hello Lars1,

we are looking into it.

Best Regards,

Matthias

The Ethernet MAC Interface is only supporting 1.8 to 2.5 volts as specified from now on. All the IMX.8 modules that will final production ones will only be specified for 1.8 or 2.5.

So your interface on the Ethernet side needs to be changed, we recommend 1.8 volt.

The secondary Ethernet on the Apalis IMX8 (MAC) is can be switch to 1.8volt or 2.5volt via device tree. The port of the IMX8 will be switched to 1.8volt.

The MDC and MDIO signal remain at 3.3 volt on the iMX8 side.

So you need level shifting from 3.3 to 1.8 of the switch since you need to operate the switch at 1.8volt.

The reset should also be shifted to 1.8 volts.

The 22 ohms resistor R27 in the clock line RGMII_REF_CLK should be left away or best up there a 0 ohm resistor.

The RGMI1 Interface need also 1.8 Volt strapping resistors for the configuration.

The fec2 should look like this.
&fec2 {
pinctrl-names = “default”, “sleep”;
pinctrl-0 = <&pinctrl_enet2>;
pinctrl-1 = <&pinctrl_enet2_sleep>;
assigned-clocks = <&clks IMX7D_ENET2_TIME_ROOT_SRC>,
<&clks IMX7D_ENET2_TIME_ROOT_CLK>;
assigned-clock-parents = <&clks IMX7D_PLL_ENET_MAIN_100M_CLK>;
assigned-clock-rates = <0>, <100000000>;
phy-mode = “rmii”;
phy-supply = <&reg_LDO1>;
phy-reset-gpios = <&gpio2 18 GPIO_ACTIVE_LOW>;
phy-reset-duration = <100>;
fsl,magic-packet;
phy-handle = <&ethphy1>;
status = “okay”;
fsl,mii-exclusive;
mdio {
#address-cells = <1>;
#size-cells = <0>;

    ethphy1: ethernet-phy@1 {
        compatible = "ethernet-phy-ieee802.3-c22";
        max-speed = <100>;
        micrel,led-mode = <0>;
        reg = <1>;
    };
};

};

@matthias.tx: can you post the ref. schematic with an Ethernet PHY here too? Thanks

Hello Andi,

I did send the Schematic to the customers email.

Best Regards,

Matthias Gohlke

We did get it working.

There are multiple things that went wrong.
NOTE: we upgraded to BSP 5.6.0 in the mean time

Wrong voltage levels

we configure 1.8V when our board actually expects 2.5V

...
&reg_ext_rgmii {
	regulator-min-microvolt = <2500000>;
	regulator-max-microvolt = <2500000>;
	power-domains = <&pd IMX_SC_R_BOARD_R3>;

	regulator-state-mem {
		regulator-off-in-suspend;
	};
};
...

We did not mux the reset pin

Even if we do not control it via a dedicated GPIO, we still need to mux it.

...
&iomuxc {
	pinctrl-names = "default";
	/* NOTE: most of these labels come from imx8-apalis-v1.1.dtsi, the rest is defined here */
	pinctrl-0 = <&pinctrl_reset_moci>, ...
...

Reset timings need to be configured in nested phy node

&fec2 {
	pinctrl-names = "default", "sleep";
	pinctrl-0 = <&pinctrl_fec2>;
	pinctrl-1 = <&pinctrl_fec2_sleep>;

	fsl,magic-packet;
	fsl,mii-exclusive;

	phy-supply = <&reg_ext_rgmii>;
	phy-handle = <&ethphy1>;
	phy-mode = "rgmii-id";

	status = "okay";

	mdio {
		#address-cells = <1>;
		#size-cells = <0>;

		ethphy1: ethernet-phy@7 {
			compatible = "ethernet-phy-ieee802.3-c22";
			interrupt-parent = <&lsio_gpio0>;
			interrupts = <23 IRQ_TYPE_LEVEL_LOW>;
			micrel,led-mode = <0>;

			/* these seem to be popular for KSZ9031*/
			reset-assert-us = <10000>;
			reset-deassert-us = <10000>;
			reset-gpios = <&lsio_gpio0 30 GPIO_ACTIVE_LOW>;
			reset-names = "phy-reset";

			reg = <7>;
		};
	};
};

Notes

  • After some discussion with toradex, they suggested using a dedicated GPIO to trigger the reset like they do with eth0
  • We pull ETH_INT to the wrong voltage on our board. It just happens to work and we should fix it, perhaps with a level changer.

Tips

  • Carefully measuring the interaction between our reset and the assertion of the phys MODE pins on was really helpful in verifiying the correct reset timings etc.
  • Sample with a high enough frequency when trying to capture MDIO traffic

Thank you @Lars1 for sharing your solution with us!

Best regards,
Andi