Setting up second ethernet as RGMII Gigabit on Colibri i.MX7 eMMC 1GB

isaac · January 24, 2018, 5:23am

Goal

I am trying configure a RGMII PHY as second ethernet on Colibri iMX7 eMMC 1GB.

We have wired a Micrel KSZ9031RNX chip on RGMII signals as the second ethernet based module datasheet:

We want to use this PHY as 1 Gigabit second ethernet along with the original 10/100 ethernet.

What I have tried

I have made the following changes in the device-tree to configure second fec with RGMII signals:

Adding configs for fec2 based on imx7d-cl-som-imx7.dts.
Adding a mdio sub-node in fec1 and setting phy-handle property in fec1 and fec2 accordingly.
Removing related pins from pinctrl_gpioX to avoid registering them as GPIO.
Adding pinctrl_enet2 pinctrl group based on imx7d-cl-som-imx7.dts

Here is the changes in the device-tree:

diff --git a/arch/arm/boot/dts/imx7-colibri.dtsi b/arch/arm/boot/dts/imx7-colibri.dtsi
index 28e7e97..8c53f56 100644
--- a/arch/arm/boot/dts/imx7-colibri.dtsi
+++ b/arch/arm/boot/dts/imx7-colibri.dtsi
@@ -127,6 +127,34 @@
 	phy-mode = "rmii";
 	phy-supply = <&reg_LDO1>;
 	fsl,magic-packet;
+
+	phy-handle = <&ethphy0>;
+
+	mdio {
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		ethphy0: ethernet-phy@0 {
+			reg = <0>;
+		};
+
+		ethphy1: ethernet-phy@1 {
+			reg = <1>;
+		};
+	};
+};
+
+&fec2 {
+	pinctrl-names = "default";
+	pinctrl-0 = <&pinctrl_enet2>;
+	assigned-clocks = <&clks IMX7D_ENET2_TIME_ROOT_SRC>,
+			  <&clks IMX7D_ENET2_TIME_ROOT_CLK>;
+	assigned-clock-parents = <&clks IMX7D_PLL_ENET_MAIN_100M_CLK>;
+	assigned-clock-rates = <0>, <100000000>;
+	phy-mode = "rgmii";
+	phy-handle = <&ethphy1>;
+	fsl,magic-packet;
+	status = "okay";
 };
 
 &gpmi {
@@ -342,7 +370,7 @@
 
 &iomuxc {
 	pinctrl-names = "default";
-	pinctrl-0 = <&pinctrl_gpio1 &pinctrl_gpio2 &pinctrl_gpio3 &pinctrl_gpio4 &pinctrl_gpio5>;
+	pinctrl-0 = <&pinctrl_gpio1 &pinctrl_gpio2 &pinctrl_gpio3 &pinctrl_gpio5>;
 
 	pinctrl_gpio1: gpio1-grp {
 		fsl,pins = <
@@ -363,27 +391,15 @@
 			MX7D_PAD_EPDC_DATA05__GPIO2_IO5		0x14 /* SODIMM 121 */
 			MX7D_PAD_EPDC_DATA06__GPIO2_IO6		0x14 /* SODIMM 123 */
 			MX7D_PAD_EPDC_DATA07__GPIO2_IO7		0x14 /* SODIMM 125 */
-			MX7D_PAD_EPDC_SDCE2__GPIO2_IO22		0x14 /* SODIMM 127 */
 			MX7D_PAD_UART3_RTS_B__GPIO4_IO6		0x14 /* SODIMM 131 */
-			MX7D_PAD_EPDC_GDRL__GPIO2_IO26		0x14 /* SODIMM 133 */
 			MX7D_PAD_SAI1_RX_DATA__GPIO6_IO12	0x14 /* SODIMM 169 */
 			MX7D_PAD_SAI1_RX_BCLK__GPIO6_IO17	0x14 /* SODIMM 24 */
 			MX7D_PAD_SD2_DATA2__GPIO5_IO16		0x14 /* SODIMM 100 */
 			MX7D_PAD_SD2_DATA3__GPIO5_IO17		0x14 /* SODIMM 102 */
-			MX7D_PAD_EPDC_GDSP__GPIO2_IO27		0x14 /* SODIMM 104 */
 			MX7D_PAD_EPDC_BDR1__GPIO2_IO29		0x14 /* SODIMM 110 */
 			MX7D_PAD_EPDC_PWR_COM__GPIO2_IO30	0x14 /* SODIMM 112 */
-			MX7D_PAD_EPDC_SDCLK__GPIO2_IO16		0x14 /* SODIMM 114 */
-			MX7D_PAD_EPDC_SDLE__GPIO2_IO17		0x14 /* SODIMM 116 */
-			MX7D_PAD_EPDC_SDOE__GPIO2_IO18		0x14 /* SODIMM 118 */
-			MX7D_PAD_EPDC_SDSHR__GPIO2_IO19		0x14 /* SODIMM 120 */
-			MX7D_PAD_EPDC_SDCE0__GPIO2_IO20		0x14 /* SODIMM 122 */
-			MX7D_PAD_EPDC_SDCE1__GPIO2_IO21		0x14 /* SODIMM 124 */
 			MX7D_PAD_EPDC_DATA14__GPIO2_IO14	0x14 /* SODIMM 126 */
 			MX7D_PAD_EPDC_PWR_STAT__GPIO2_IO31	0x14 /* SODIMM 128 */
-			MX7D_PAD_EPDC_SDCE3__GPIO2_IO23		0x14 /* SODIMM 130 */
-			MX7D_PAD_EPDC_GDCLK__GPIO2_IO24		0x14 /* SODIMM 132 */
-			MX7D_PAD_EPDC_GDOE__GPIO2_IO25		0x14 /* SODIMM 134 */
 			MX7D_PAD_EPDC_DATA12__GPIO2_IO12	0x14 /* SODIMM 150 */
 			MX7D_PAD_EPDC_DATA11__GPIO2_IO11	0x14 /* SODIMM 152 */
 			MX7D_PAD_SD2_CLK__GPIO5_IO12		0x14 /* SODIMM 184 */
@@ -467,6 +483,23 @@
 		>;
 	};
 
+	pinctrl_enet2: enet2grp {
+		fsl,pins = <
+			MX7D_PAD_EPDC_GDSP__ENET2_RGMII_TXC			0x1
+			MX7D_PAD_EPDC_SDCE2__ENET2_RGMII_TD0		0x1
+			MX7D_PAD_EPDC_SDCE3__ENET2_RGMII_TD1		0x1
+			MX7D_PAD_EPDC_GDCLK__ENET2_RGMII_TD2		0x1
+			MX7D_PAD_EPDC_GDOE__ENET2_RGMII_TD3			0x1
+			MX7D_PAD_EPDC_GDRL__ENET2_RGMII_TX_CTL		0x1
+			MX7D_PAD_EPDC_SDCE1__ENET2_RGMII_RXC		0x1
+			MX7D_PAD_EPDC_SDCLK__ENET2_RGMII_RD0		0x1
+			MX7D_PAD_EPDC_SDLE__ENET2_RGMII_RD1			0x1
+			MX7D_PAD_EPDC_SDOE__ENET2_RGMII_RD2			0x1
+			MX7D_PAD_EPDC_SDSHR__ENET2_RGMII_RD3		0x1
+			MX7D_PAD_EPDC_SDCE0__ENET2_RGMII_RX_CTL		0x1
+		>;
+	};
+
 	pinctrl_enet1_sleep: enet1sleepgrp {
 		fsl,pins = <
 			MX7D_PAD_ENET1_RGMII_RX_CTL__GPIO7_IO4		0x0

Problem

During boot, the second PHY isn’t correctly recognized. Here is the relevant lines (full boot log can be found at very below):

libphy: fec_enet_mii_bus: probed
fec 30be0000.ethernet eth0: registered PHC device 0
30bf0000.ethernet supply phy not found, using dummy regulator
pps pps1: new PPS source ptp1
fec 30bf0000.ethernet (unnamed net_device) (uninitialized): Invalid MAC address: 00:00:00:00:00:00
fec 30bf0000.ethernet (unnamed net_device) (uninitialized): Using random MAC address: e2:31:b0:bf:41:a5
fec 30bf0000.ethernet eth1: registered PHC device 1

Also strangely, after the boot two ethernet adapters are registered, both referencing the same fec (30bf0000):

Micrel KSZ8041 30be0000.ethernet-1:00: attached PHY driver [Micrel KSZ8041] (mii_bus:phy_addr=30be0000.ethernet-1:00, irq=-1)
IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Micrel KSZ8041 30be0000.ethernet-1:01: attached PHY driver [Micrel KSZ8041] (mii_bus:phy_addr=30be0000.ethernet-1:01, irq=-1)
IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
fec 30bf0000.ethernet eth1: Link is Up - 100Mbps/Full - flow control rx/tx
IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
fec 30be0000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Questions

Is my changes in device-tree sensible for configuring a RGMII as second PHY on Colibri iMX7 1GB?
Specifically, I am not sure about the clocking configurations for fec2. I have noticed the Colibri clocking for fec1 is a little bit different than mainline clocking for imx7d. Maybe that should be done for fec2 too?
Is the pin bitmasks for pinctrl_enet2 correct (0x1)? Or I should use a different bitmask (possibly a value of 0x73 like pinctrl_enet1 pins)?
Is adding mdio subnode really necessary?
Why two ethernet adapters are registered after boot? I expected no eth1 to be registered due to failure in registering second PHY. I can confirm both are on fec0 as they both are referencing 30bf0000 and a ifconfig eth1 down puts eth0 down too.
What other configurations should we try if the DTS seems correct?

Here is the full boot log: Bootlog with fec2

Here is the full boot log before above changes in DTS: Bootlog without fec2

stefan.tx · January 24, 2018, 8:58am

I am assuming you have the second Ethernet PHY connected according to Table 5-5 RGMII signals.

The Sabre reference design as well as many other designs share the one MDIO bus for both PHY’s but this is not true for the Colibri. If you connected the MDC/MDIO signals according to the table, they are connected to the FEC2 instance, hence the second PHY needs to be part of the fec2 device tree node. Also you are missing pin mux information for the MDC/MDIO pins.
The clocking seems correct. I also would expect that the PHY appears even if clocks are not properly set up (since the MDIO bus allows to talk to the PHY indpendent of the main data signals)
0x73 enables pull-ups and uses a higher drive strength. Either way should work.
I think not strictly. Afaik, the FEC driver defaults to scan the local bus for a PHY (FEC2->FEC2 MDIO bus), which would be what you want in your case. If you leave the mdio subnode away, then you also need to remove the reference in phy-handle. You still have to mux the MDC/MDIO pins according to you design…
Ethernet controller can live without PHY. But in your case, the second Ethernet uses the PHY of the first Ethernet controller. Hence the mixup…

isaac · February 19, 2018, 10:21am

Hi,

We finally got the second RGMII ethernet working. The below is the changes needed in the kernel and device tree.

One final issue is the MAC address of second ethernet. It is set correctly in U-Boot environment, (with the offset of 0x100000 with the first MAC address), but the kernel sees an MAC address with all zeros and choose a random MAC address. I have added aliases for both fec1 and fec2 ( ethernet0=&fec1 and ethernet1=&fec2) to device tree as you have suggested here, but that didn’t help in this case.
Is there any other changes needed for Colibri iMX7 to pass the MAC address to kernel?

Here is the bootlog as reference.

Nevertheless, the second ethernet is working correctly. The key changes are:

Specifying fsl,mii-exclusive in fec2 device tree: As you have already mentioned, Colibri iMX7 has two separate MDIO buses. However, the FEC driver (in drivers/net/ethernet/freescale/fec_main.c) will use the first MDIO buses for other FECs, even if a separate MDIO bus is defined for them. This flag prevents this behavior.
Changing IMX7D_GPR1_ENET_TX_CLK_SEL_MASK to IMX7D_GPR1_ENET1_TX_CLK_SEL_MASK mach-imx7d.c: in Apparently, the first ethernet in Colibri iMX7 uses an external oscillator for the transmit clock, hence the lack of enet_out in its clocks device tree definition. This causes the code in imx7d_enet_clk_sel ( mach-imx7d.c) to disable internal clock generation for both ethernets. As a workaround I modified the code to just disable the internal clock for the first ethernet when enet_out is missing. The ideal solution would be to check for existence of enet_out for each ethernet and enable/disable internal clock per ethernet.

Here is the diff of all changes needed for enabling second ethernet for RGMII (our PHY address is 3):

diff --git a/arch/arm/boot/dts/imx7-colibri.dtsi b/arch/arm/boot/dts/imx7-colibri.dtsi
index 28e7e97..7c60e55 100644
--- a/arch/arm/boot/dts/imx7-colibri.dtsi
+++ b/arch/arm/boot/dts/imx7-colibri.dtsi
@@ -129,6 +129,35 @@
 	fsl,magic-packet;
 };
 
+&fec2 {
+	pinctrl-names = "default";
+	pinctrl-0 = <&pinctrl_enet2>;
+	assigned-clocks = <&clks IMX7D_ENET2_TIME_ROOT_SRC>,
+			  <&clks IMX7D_ENET2_TIME_ROOT_CLK>;
+	assigned-clock-parents = <&clks IMX7D_PLL_ENET_MAIN_100M_CLK>;
+	assigned-clock-rates = <0>, <100000000>;
+
+	phy-mode = "rgmii";
+	phy-handle = <&ethphy1>;
+	fsl,magic-packet;
+	status = "okay";
+	fsl,mii-exclusive;
+
+	mdio {
+	 	#address-cells = <1>;
+	 	#size-cells = <0>;
+
+		ethphy1: ethernet-phy@3 {
+			compatible = "ethernet-phy-ieee802.3-c22";
+			reg = <3>;
+		};
+	};
+};
+
 &gpmi {
 	pinctrl-names = "default";
 	pinctrl-0 = <&pinctrl_gpmi_nand>;
@@ -342,7 +371,7 @@
 
 &iomuxc {
 	pinctrl-names = "default";
-	pinctrl-0 = <&pinctrl_gpio1 &pinctrl_gpio2 &pinctrl_gpio3 &pinctrl_gpio4 &pinctrl_gpio5>;
+	pinctrl-0 = <&pinctrl_gpio1 &pinctrl_gpio2 &pinctrl_gpio3 &pinctrl_gpio5>;
 
 	pinctrl_gpio1: gpio1-grp {
 		fsl,pins = <
@@ -363,27 +392,15 @@
 			MX7D_PAD_EPDC_DATA05__GPIO2_IO5		0x14 /* SODIMM 121 */
 			MX7D_PAD_EPDC_DATA06__GPIO2_IO6		0x14 /* SODIMM 123 */
 			MX7D_PAD_EPDC_DATA07__GPIO2_IO7		0x14 /* SODIMM 125 */
-			MX7D_PAD_EPDC_SDCE2__GPIO2_IO22		0x14 /* SODIMM 127 */
 			MX7D_PAD_UART3_RTS_B__GPIO4_IO6		0x14 /* SODIMM 131 */
-			MX7D_PAD_EPDC_GDRL__GPIO2_IO26		0x14 /* SODIMM 133 */
 			MX7D_PAD_SAI1_RX_DATA__GPIO6_IO12	0x14 /* SODIMM 169 */
 			MX7D_PAD_SAI1_RX_BCLK__GPIO6_IO17	0x14 /* SODIMM 24 */
 			MX7D_PAD_SD2_DATA2__GPIO5_IO16		0x14 /* SODIMM 100 */
 			MX7D_PAD_SD2_DATA3__GPIO5_IO17		0x14 /* SODIMM 102 */
-			MX7D_PAD_EPDC_GDSP__GPIO2_IO27		0x14 /* SODIMM 104 */
 			MX7D_PAD_EPDC_BDR1__GPIO2_IO29		0x14 /* SODIMM 110 */
 			MX7D_PAD_EPDC_PWR_COM__GPIO2_IO30	0x14 /* SODIMM 112 */
-			MX7D_PAD_EPDC_SDCLK__GPIO2_IO16		0x14 /* SODIMM 114 */
-			MX7D_PAD_EPDC_SDLE__GPIO2_IO17		0x14 /* SODIMM 116 */
-			MX7D_PAD_EPDC_SDOE__GPIO2_IO18		0x14 /* SODIMM 118 */
-			MX7D_PAD_EPDC_SDSHR__GPIO2_IO19		0x14 /* SODIMM 120 */
-			MX7D_PAD_EPDC_SDCE0__GPIO2_IO20		0x14 /* SODIMM 122 */
-			MX7D_PAD_EPDC_SDCE1__GPIO2_IO21		0x14 /* SODIMM 124 */
 			MX7D_PAD_EPDC_DATA14__GPIO2_IO14	0x14 /* SODIMM 126 */
 			MX7D_PAD_EPDC_PWR_STAT__GPIO2_IO31	0x14 /* SODIMM 128 */
-			MX7D_PAD_EPDC_SDCE3__GPIO2_IO23		0x14 /* SODIMM 130 */
-			MX7D_PAD_EPDC_GDCLK__GPIO2_IO24		0x14 /* SODIMM 132 */
-			MX7D_PAD_EPDC_GDOE__GPIO2_IO25		0x14 /* SODIMM 134 */
 			MX7D_PAD_EPDC_DATA12__GPIO2_IO12	0x14 /* SODIMM 150 */
 			MX7D_PAD_EPDC_DATA11__GPIO2_IO11	0x14 /* SODIMM 152 */
 			MX7D_PAD_SD2_CLK__GPIO5_IO12		0x14 /* SODIMM 184 */
@@ -467,6 +484,25 @@
 		>;
 	};
 
+	pinctrl_enet2: enet2grp {
+		fsl,pins = <
+			MX7D_PAD_GPIO1_IO14__ENET2_MDIO				0x37
+			MX7D_PAD_GPIO1_IO15__ENET2_MDC				0x37
+			MX7D_PAD_EPDC_GDSP__ENET2_RGMII_TXC			0x1
+			MX7D_PAD_EPDC_SDCE2__ENET2_RGMII_TD0		0x1
+			MX7D_PAD_EPDC_SDCE3__ENET2_RGMII_TD1		0x1
+			MX7D_PAD_EPDC_GDCLK__ENET2_RGMII_TD2		0x1
+			MX7D_PAD_EPDC_GDOE__ENET2_RGMII_TD3			0x1
+			MX7D_PAD_EPDC_GDRL__ENET2_RGMII_TX_CTL		0x1
+			MX7D_PAD_EPDC_SDCE1__ENET2_RGMII_RXC		0x1
+			MX7D_PAD_EPDC_SDCLK__ENET2_RGMII_RD0		0x1
+			MX7D_PAD_EPDC_SDLE__ENET2_RGMII_RD1			0x1
+			MX7D_PAD_EPDC_SDOE__ENET2_RGMII_RD2			0x1
+			MX7D_PAD_EPDC_SDSHR__ENET2_RGMII_RD3		0x1
+			MX7D_PAD_EPDC_SDCE0__ENET2_RGMII_RX_CTL		0x1
+		>;
+	};
+
 	pinctrl_enet1_sleep: enet1sleepgrp {
 		fsl,pins = <
 			MX7D_PAD_ENET1_RGMII_RX_CTL__GPIO7_IO4		0x0
diff --git a/arch/arm/mach-imx/mach-imx7d.c b/arch/arm/mach-imx/mach-imx7d.c
index 36d5954..f9e107d 100644
--- a/arch/arm/mach-imx/mach-imx7d.c
+++ b/arch/arm/mach-imx/mach-imx7d.c
@@ -112,7 +112,7 @@ static void __init imx7d_enet_clk_sel(void)
 		if (IS_ERR(enet_out_clk)) {
 			pr_info("%s: failed to get enet_out clock, assuming ext. clock source\n", __func__);
 			/* use external clock for PHY */
-			regmap_update_bits(gpr, IOMUXC_GPR1, IMX7D_GPR1_ENET_TX_CLK_SEL_MASK, IMX7D_GPR1_ENET_TX_CLK_SEL_MASK);
+			regmap_update_bits(gpr, IOMUXC_GPR1, IMX7D_GPR1_ENET_TX_CLK_SEL_MASK, IMX7D_GPR1_ENET1_TX_CLK_SEL_MASK);
 			regmap_update_bits(gpr, IOMUXC_GPR1, IMX7D_GPR1_ENET_CLK_DIR_MASK, 0);
 		} else {
 			pr_info("%s: found enet_out clock, assuming internal clock source\n", __func__);

isaac · February 19, 2018, 10:26am

Hi Stefan.

Thanks for the explanations. This helped me to get the second ethernet working. I have posted the details in the below.
There is a minor issue with MAC address that doesn’t get set. I have explained this problem in the below too. Let me know if I should submit it as a separate question.

stefan.tx · February 19, 2018, 1:14pm

Thanks for your feedback! I am glad it works now!

Oh I see, fsl,mii-exclusive is a property NXP added in the downstream kernel, it is not necessary for vanilla/upstream kernels.
Oh yes this is really not nice in the current code. It is even different whether external or internal clock is selected (also in the external case only the clock for ENET1 is enabled). I just pushed a change to our toradex_4.9-1.0.x-imx-next branch which should handle your case properly. It also adds the aliases to the device tree. With the aliases set, U-Boots fdt_fixup_ethernet should handle MAC addresses fine.

Sidenote: The Toradex design initially planned to use the internal clock source. Unfortunately the clock pad GPIO1_IO12 we choose also has the NMI input for the Cortex-M4 on the same pin. Despite different pinmux alternative it seems that the clock triggers NMI on the M4 side…

isaac · February 19, 2018, 2:40pm

Wow, that was fast!

After applying your patches, I can confirm the kernel is now picking up both MAC addresses correctly and the eth0 and eth1 are working as expected on desired speed.

Thanks for the throughout explanation and prompt response.

isaac · February 19, 2018, 3:28pm

Just as a note for future readers, I still had to keep fsl,mii-exclusive in fec2 device tree node until fec_main.c stop ignoring the specified mdio node for fec2.

stefan.tx · February 20, 2018, 8:27am

Thanks for noting. Yes, for the Toradex provided (NXP based) kernels the fsl,mii-exclusive property is required.