Colibri VFxx USB full speed on Linux

Here High performance, low power Embedded Computing Systems | Toradex Developer Center it is mentioned that USB can be switched to 12Mbit/s under WinCE.

Would it be hard to modify the Linux driver to do the same?

USB speed can be selected using the maximum-speed property (see Documentation/devicetree/bindings/usb/generic.txt). Please refer to the Device Tree Customization article how to alter the device tree. In practice you should set that property in your Carrier Board level device tree with something like this:

&usbh1 {
    maximum-speed = "full-speed";
}