I found the solution:
The pins were not enabled in U-Boot (I assumed, they were because I could set them)
I added the needed pin configurations to arch/arm/include/asm/arch-vf610/iomux-vf610.h (I took the values from the kernel devicetree vf610-pinfunc.h, so I avoided reading the processor manual)
and appended the pin configs in board/toradex/colibri_vf/colibri_vf.c to setup_iomux_gpio().
@sst Yes, you are correct one need to configure the IO mux of respective pin for using it as GPIO properly. We added IO mux config for few free IO’s as an example in this commit.