I found an issue when disabling some pins from the device tree to dedicate them to the M4 Core.
Commenting the SODIMM_29 in
&gpio1 {
gpio-line-names =
is not enough to prevent the Linux from messing with the pin.
I had to comment the line:
&uart1 {
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_uart1 &pinctrl_uart1_ctrl1 /*&pinctrl_uart1_ctrl2*/>;
I am using the UART1, however not with the control pins.