You’re trying something unusual.
Maybe you could try to ping the server before starting the TFTP transaction, however there is also a seconds long timeout in which U-Boot waits for the answer to ping.
E.g. with echo as a surrogate to tftpboot:
Colibri iMX6 # ping 192.168.10.100 && echo up
Using FEC device
ping failed; host 192.168.10.100 is not alive
Colibri iMX6 # ping 192.168.10.1 && echo up
Using FEC device
host 192.168.10.1 is alive
up