Would you mind explaining this in more detail?
Yes, of course. After you have loaded the binary into the cortex-m, you have to boot up your module and stop at the u-boot terminal, pressing the space bar multiple times after powering up the module. After that, you have to run the debugger on your host PC. This is also explained here: High performance, low power Embedded Computing Systems | Toradex Developer Center
If this does not work, I would ask you to change the USB port/ USB hub that you are using. If you are using a USB hub, I would also recommend trying connecting the USB from the debugger directly to your PC.