What do we want to do?
We want to enable the second ethernet port and get it working properly.
Hardware/Software setup:
- Baseboard: custom, but based on the Ixora reference design
- Second PHY on Baseboard: Micrel KSZ9031RNX
- SoM: Apalis i.MX8QM V1.1B
- BSP: 5.5.0
Relevant part of the schematics:
Not seen here:
-
RGMI_(MDC|MDIO)
go through a level changer before connecting to the phy to bringing them down to 1.8V - Some
RGMII*
pins that are int the range of pins forMMC1
andUART2
-
ETH_RESET
connects toApalis: pin 26, function RESET_MOCI#
-
ETH_INT
connects toApalis: pin 159, function TS_6 (MMC1)
What did we try?
We tried to create a device-tree overlay as follows (Note: ECU refers to our custom base board):
#include <dt-bindings/pinctrl/pads-imx8qm.h>
#include <dt-bindings/gpio/gpio.h>
#include <dt-bindings/interrupt-controller/irq.h>
#include <dt-bindings/firmware/imx/rsrc.h>
/dts-v1/;
/plugin/;
/* ECU: eth1 */
/* we "just" try to mimic the pin config we see on fec1 */
/*
* in general, you need to look up the pin config field for fsl,pins
* in the "imx8 quad max applications processor manual" under "IOMUXD"
* (the "source part" of the mux setting ususally matches pretty well)
*
* you may also check http://git.toradex.com/cgit/linux-toradex.git/tree/Documentation/devicetree/bindings/pinctrl/fsl,imx8qm-pinctrl.txt?h=toradex_4.9-2.3.x-imx
* for the device tree binding doc, what they call _hw_pad_iomux_28fdsoi is what I believe to be the 90+% case
* it matches up with the stuff I describe below
*
* bits we usually care about (0-indexed):
* 26-25 sw_config -> 00 default, 01 open drain, 10 open drain input, 11 inout
* 6-5 PULL -> 00 illegal, 01 pull-up, 10 pull-down, 11 pull-disabled
* 0 PDRV -> 0 high drive strength, 1 low drive strength
*
* common values:
* 0x2P -> pull up, P = PDRV, 0/1
* 0x4P -> pull down, P = PDRV, 0/1
*/
&iomuxc {
pinctrl-names = "default";
/* NOTE: most of these labels come from imx8-apalis-v1.1.dtsi */
pinctrl-0 = <&pinctrl_cam1_gpios>, <&pinctrl_dap1_gpios>,
<&pinctrl_esai0_gpios>, <&pinctrl_gpio3>, <&pinctrl_gpio4>, <&pinctrl_gpio_keys>,
<&pinctrl_gpio_usbh_oc_n>, <&pinctrl_lpuart1ctrl>,
<&pinctrl_lvds0_i2c0_gpio>, <&pinctrl_lvds1_i2c0_gpios>,
<&pinctrl_mipi_dsi_0_1_en>, <&pinctrl_mipi_dsi1_gpios>,
<&pinctrl_mlb_gpios>, <&pinctrl_qspi1a_gpios>,
<&pinctrl_sata1_act>, <&pinctrl_sim0_gpios>,
<&pinctrl_usdhc1_gpios>, <&pinctrl_pwm2>;
ecu {
pinctrl_fec2: fec2grp {
fsl,pins = <
/* pin conf: special case, see IMOUXD_COMP_CTL_GPIO_1V8_3V3_SIM in manual
* bits set, rest is whatever 0 means:
* RASRCP = reset value, RASRCN = reset value
* corresponds to _hw_pad_iomux_28fdsoi_comp in dt binding doc
* Apalis comment for ENET0: Use pads in 3.3V mode
* Not sure where they get that info from...
*/
/* NOTE: weirdly, fec1 is ENETB_PAD, and fec2 is (by exclusion) ENETA_PAD */
IMX8QM_COMP_CTL_GPIO_1V8_3V3_ENET_ENETA_PAD 0x000014a0
/* Apalis: pin 253, function LCD1_R1 */
/* IMX8QM_ENET1_MDC_LSIO_GPIO4_IO18 0x00000021 */
/* pin conf: sw_config inout, pull-up, high drive */
IMX8QM_ENET1_MDC_CONN_ENET1_MDC 0x06000020
/* Apalis: pin 251, function LCD1_R0 */
/* IMX8QM_ENET1_MDIO_LSIO_GPIO4_IO17 0x00000021 */
/* pin conf: sw_config inout, pull-up, high drive */
IMX8QM_ENET1_MDIO_CONN_ENET1_MDIO 0x06000020
/* Apalis: pin 263, function LCD1_R6 */
/* IMX8QM_ENET1_RGMII_TX_CTL_LSIO_GPIO6_IO11 0x00000021 */
/* pin conf: sw_config inout, pull-up, high drive */
IMX8QM_ENET1_RGMII_TX_CTL_CONN_ENET1_RGMII_TX_CTL 0x06000020
/* Apalis: pin 261, function LCD1_R5 */
/* IMX8QM_ENET1_RGMII_TXC_LSIO_GPIO6_IO10 0x00000021 */
/* pin conf: sw_config inout, pull-up, high drive */
IMX8QM_ENET1_RGMII_TXC_CONN_ENET1_RGMII_TXC 0x06000020
/* Apalis: pin 259, function LCD1_R4 */
/* IMX8QM_ENET1_RGMII_TXD0_LSIO_GPIO6_IO12 0x00000021 */
/* pin conf: sw_config inout, pull-up, high drive */
IMX8QM_ENET1_RGMII_TXD0_CONN_ENET1_RGMII_TXD0 0x06000020
/* Apalis: pin 257, function LCD1_R3 */
/* IMX8QM_ENET1_RGMII_TXD1_LSIO_GPIO6_IO13 0x00000021 */
/* pin conf: sw_config inout, pull-up, high drive */
IMX8QM_ENET1_RGMII_TXD1_CONN_ENET1_RGMII_TXD1 0x06000020
/* Apalis: pin 255, function LCD1_R2 */
/* IMX8QM_ENET1_RGMII_TXD2_LSIO_GPIO6_IO14 0x00000021 */
/* pin conf: sw_config inout, pull-up, high drive */
IMX8QM_ENET1_RGMII_TXD2_CONN_ENET1_RGMII_TXD2 0x06000020
/* moved from lpuart3 */
/* Apalis: pin 128, function LCD1_R2 */
/* pin conf: sw_config inout, pull-up, high drive */
/* IMX8QM_ENET1_RGMII_TXD3_DMA_UART3_RTS_B 0x06000020 */
IMX8QM_ENET1_RGMII_TXD3_CONN_ENET1_RGMII_TXD3 0x06000020
/* Apalis: pin 130, function LCD1_R2 */
/* pin conf: sw_config inout, pull-up, high drive */
/* IMX8QM_ENET1_RGMII_RXC_DMA_UART3_CTS_B 0x06000020 */
IMX8QM_ENET1_RGMII_RXC_CONN_ENET1_RGMII_RXC 0x06000020
/* Apalis: pin 265, function LCD1_R7 */
/* IMX8QM_ENET1_RGMII_RX_CTL_LSIO_GPIO6_IO17 0x00000021 */
/* pin conf: sw_config inout, pull-up, high drive */
IMX8QM_ENET1_RGMII_RX_CTL_CONN_ENET1_RGMII_RX_CTL 0x06000020
/* Apalis: pin 249, function LCD1_DE */
/* IMX8QM_ENET1_RGMII_RXD0_LSIO_GPIO6_IO18 0x00000021 */
/* pin conf: sw_config inout, pull-up, high drive */
IMX8QM_ENET1_RGMII_RXD0_CONN_ENET1_RGMII_RXD0 0x06000020
/* Apalis: pin 247, function LCD1_HSYNC */
/* IMX8QM_ENET1_RGMII_RXD1_LSIO_GPIO6_IO19 0x00000021 */
/* pin conf: sw_config inout, pull-up, high drive */
IMX8QM_ENET1_RGMII_RXD1_CONN_ENET1_RGMII_RXD1 0x06000020
/* Apalis: pin 245, function LCD1_VSYNC */
/* IMX8QM_ENET1_RGMII_RXD2_LSIO_GPIO6_IO20 0x00000021 */
/* pin conf: sw_config inout, pull-up, high drive */
IMX8QM_ENET1_RGMII_RXD2_CONN_ENET1_RGMII_RXD2 0x06000020
/* Apalis: pin 243, function LCD1_PCLK */
/* IMX8QM_ENET1_RGMII_RXD3_LSIO_GPIO6_IO21 0x00000021 */
/* pin conf: sw_config inout, pull-up, high drive */
IMX8QM_ENET1_RGMII_RXD3_CONN_ENET1_RGMII_RXD3 0x06000020
/* Apalis: pin 269, function LCD1_G0 */
/* IMX8QM_ENET1_REFCLK_125M_25M_LSIO_GPIO4_IO16 0x00000021 */
/* pin conf: sw_config inout, pull-up, high drive */
IMX8QM_ENET1_REFCLK_125M_25M_CONN_ENET1_REFCLK_125M_25M 0x06000020
/* For ENET0: Apalis: pin "on module", function ETH_RESET# */
/* FOR ENET1: ECU is wired directly to (Apalis: pin 26, function RESET_MOCI#) */
/* so we don't mux anything for that pin*/
/* ENET0 mux */
/* IMX8QM_LVDS1_GPIO01_LSIO_GPIO1_IO11 0x06000020 */
/* For ENET0: Apalis: pin "on module", function ETH_INT# */
/* For ENET1: Apalis: pin 159, function TS_6 (MMC1) */
/* ENET0 mux */
/* IMX8QM_MIPI_CSI1_MCLK_OUT_LSIO_GPIO1_IO29 0x04000060 */
/* ENET1 mux */
/* pin conf: sw_config open drain input, pull-disabled, high drive */
IMX8QM_USDHC1_DATA6_LSIO_GPIO5_IO21 0x04000060
>;
};
pinctrl_fec2_sleep: fec2-sleepgrp {
fsl,pins = <
IMX8QM_COMP_CTL_GPIO_1V8_3V3_ENET_ENETA_PAD 0x000014a0
IMX8QM_ENET1_MDC_LSIO_GPIO4_IO18 0x04000040
IMX8QM_ENET1_MDIO_LSIO_GPIO4_IO17 0x04000040
IMX8QM_ENET1_RGMII_TX_CTL_LSIO_GPIO6_IO11 0x04000040
IMX8QM_ENET1_RGMII_TXC_LSIO_GPIO6_IO10 0x04000040
IMX8QM_ENET1_RGMII_TXD0_LSIO_GPIO6_IO12 0x04000040
IMX8QM_ENET1_RGMII_TXD1_LSIO_GPIO6_IO13 0x04000040
IMX8QM_ENET1_RGMII_TXD2_LSIO_GPIO6_IO14 0x04000040
IMX8QM_ENET1_RGMII_TXD3_LSIO_GPIO6_IO15 0x04000040
IMX8QM_ENET1_RGMII_RXC_LSIO_GPIO6_IO16 0x04000040
IMX8QM_ENET1_RGMII_RX_CTL_LSIO_GPIO6_IO17 0x04000040
IMX8QM_ENET1_RGMII_RXD0_LSIO_GPIO6_IO18 0x04000040
IMX8QM_ENET1_RGMII_RXD1_LSIO_GPIO6_IO19 0x04000040
IMX8QM_ENET1_RGMII_RXD2_LSIO_GPIO6_IO20 0x04000040
IMX8QM_ENET1_RGMII_RXD3_LSIO_GPIO6_IO21 0x04000040
/* IMX8QM_ENET1_REFCLK_125M_25M_LSIO_GPIO4_IO16 0x04000040 */
/* not wired on ECU */
/* IMX8QM_LVDS1_GPIO01_LSIO_GPIO1_IO11 0x06000020 */
IMX8QM_USDHC1_DATA6_LSIO_GPIO5_IO21 0x04000040
>;
};
};
};
/* Disable Apalis fec2 pinctrl, just in case */
/* However, it looks like this is not necessary, because we don't reference that node anymore */
/* and we do not get error messages about pin conflicts at runtime */
/* NOTE: overlays cannot delete nodes, they can only grow the device trees shape, not shrink it */
&pinctrl_fec2_gpios {
status = "disabled";
};
/* Disable Apalis UART2, which is UART3 on the CPU */
/* some pins overlap with ENET1, so we just disable it and hope for the best */
&pinctrl_lpuart3 {
status = "disabled";
};
/* Disable Apalis MMC1, which is USDHC2 on the CPU */
/* some pins overlap with ENET1, so we just disable it and hope for the best */
&pinctrl_usdhc2 {
status = "disabled";
};
&lpuart3 {
status = "disabled";
};
&usdhc2 {
status = "disabled";
};
®_ext_rgmii {
regulator-min-microvolt = <1800000>;
regulator-max-microvolt = <1800000>;
power-domains = <&pd IMX_SC_R_BOARD_R2>;
regulator-state-mem {
regulator-off-in-suspend;
};
};
&fec2 {
pinctrl-names = "default", "sleep";
pinctrl-0 = <&pinctrl_fec2>;
pinctrl-1 = <&pinctrl_fec2_sleep>;
fsl,magic-packet;
fsl,mii-exclusive;
phy-supply = <®_ext_rgmii>;
phy-handle = <ðphy1>;
phy-mode = "rgmii-id";
/* TODO: we don't control reset, as it's wired directly to SoM reset */
/* deprecated and optional, should be defined in phy? */
/* phy-reset-duration = <10>; */
/* phy-reset-gpios = <&lsio_gpio0 8 GPIO_ACTIVE_LOW>; */
status = "okay";
mdio {
#address-cells = <1>;
#size-cells = <0>;
/* we just name it @8 to avoid a conflict with ethphy0 which is @7 */
/* should we decide to move the ethphy1 under the same mdio node as ethphy0 */
ethphy1: ethernet-phy@8 {
compatible = "ethernet-phy-ieee802.3-c22";
interrupt-parent = <&lsio_gpio5>;
interrupts = <21 IRQ_TYPE_LEVEL_LOW>;
micrel,led-mode = <0>;
/* don't set a reg, because we don't know it yet, kernel will scan all phy ids */
/* once we know it, we can hardcode it */
/* reg = <7>; */
};
};
};
Notes
- to avoid pin conflicts at run-time we had to overwrite
&iomuxc
rather than just overriding and usingpinctrl_fec2_gpios
What works, what doesn’t
We get no pin conflict messages from the iomux driver, the kernel picks up the second phy and tries to initialize it. Since we do not set a reg
property, it tries to probe all 32 possible phy addresses. However in contrast to fec1
, it never finds one.
We drilled down into the kernel and the first 2 reads on the mdio bus return garbage (mostly 1 bits).
The kernel then rejects that address an tries the next one, until it finally hits the last one and gives up.
We also measured the connection with an oscilloscope where we could verify that Apalis: pin 251
(which we mux to ENET1_MDIO
) indeed never changes levels.
The kernel thinks it’s reading on the bus, but it really isn’t.
To verify that the physical connection is OK, we muxed the pin to GPIO and toggled it every second from the cmdline. We could verify this with the oscilloscope.
kernel output for fec1
[ 0.893973] libphy: Fixed MDIO Bus: probed
....
[ 0.896024] fec 5b040000.ethernet: Adding to iommu group 0
[ 0.911861] fec 5b050000.ethernet: Adding to iommu group 0
[ 1.401109] libphy: fec_enet_mii_bus: probed
[ 1.542103] mdio_bus 5b040000.ethernet-1: registered phy ethernet-phy at address 7
[ 1.550625] fec 5b040000.ethernet eth0: registered PHC device 0
... after link set up
[ 7.287440] Microchip KSZ9131 Gigabit PHY 5b040000.ethernet-1:07: attached PHY driver [Microchip KSZ9131 Gigabit PHY] (mii_bus:phy_addr=5b040000.ethernet-1:07, irq=305)
[ 10.807163] fec 5b040000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
kernel output for fec2 on intit
[ 1.601043] mdio_bus 5b050000.ethernet-2: ethernet-phy@8 has invalid PHY address
[ 1.626762] mdio_bus 5b050000.ethernet-2: scan phy ethernet-phy at address 0
... trying all addrs 0..31
[ 2.077779] mdio_bus 5b050000.ethernet-2: scan phy ethernet-phy at address 31
[ 2.093169] fec 5b050000.ethernet eth1: registered PHC device 1
...
kernel output on setting up eth1 (fec2)
root@UNKNOWN-UNKNOWN:~# ip link set up dev eth1
[ 35.555849] fec 5b050000.ethernet eth1: Unable to connect to phy
ip: SIOCSIFFLAGS: No such device
What do we expect?
We expect so see at least some activity on the ENET1_MDIO
pin. We should be able to verify it on the oscilloscope. Furthermore, the kernel should successfully find and initialize the phy for fec2 at some phy address. Once we know it, we can hardcode it in the device-tree.
Questions
1. What’s the correct placement of ethphy1
in the device-tree?
We saw a lot of device-trees trying to do something similar that defined the ethphy1
node to be a sibling of ethphy0
. We think that would mean that they are physically on the same mdio
bus.
We do think this is wrong in our case as we connect our phy to ENET1_(MDC|MDIO)
rather than ENET0_(MDC|MDIO)
.
Is this assessment correct? If not, why and what should we do diffrently?
2. What device-tree changes are needed to get the correct voltage levels?
The Apalis iMX8 Datasheet states:
The secondary RGMII/RMII Ethernet interface needs special attention regarding the supply voltage
level. The RGMII/RMII voltage is switchable through LDO1OUT of the second PMIC PF8100. The
voltage level must be defined by software configuration. During the power-up sequence, the IO
voltage is set to 3.3V by default. All the RGMII/RMII signals are configured as GPIO inputs with
enabled pull-down resistors. If the interface is used with 1.8V IO voltage, it is crucial to configure
first the LDO1OUT voltage and then change the alternate function to RGMII/RMII. This ensures full
compatibility with 1.8V RGMII signal levels.
We took that to mean that we should set the regulator of fec2
to something that maps to a 1.8V power domain. Like this:
®_ext_rgmii {
regulator-min-microvolt = <1800000>;
regulator-max-microvolt = <1800000>;
power-domains = <&pd IMX_SC_R_BOARD_R2>;
regulator-state-mem {
regulator-off-in-suspend;
};
};
....
&fec2 {
...
phy-supply = <®_ext_rgmii>;
Note: According to a comment in imx8-apalis-v1.1.dtsi
the value IMX_SC_R_BOARD_R2
should be for 1.8V.
Is this correct and complete?