Apalis update script fails when trying to load second stage flash_eth.img

When I try to update the TK1 with run setethupdate on my local developer machine the second stage flash image file apalis-tk1/flash_eth.img is not loaded for some reason. (I use the same tftp server setup as described in the wiki.)

Apalis TK1 # setenv 10.0.1.19                                                                                                                                                               
Apalis TK1 # setenv serverip 10.0.1.19                                                                                                                                                      
Apalis TK1 # setenv autoload false; if env exists ethaddr; then; else setenv ethaddr 00:14:2d:00:00:00; fi; pci enum; dhcp;                                                                 
BOOTP broadcast 1                                                                                                                                                                           
DHCP client bound to address 10.0.2.40 (1 ms)                                                                                                                                               
Apalis TK1 # run setethupdate                                                                                                                                                               
Using e1000#0 device                                                                                                                                                                        
TFTP from server 10.0.1.19; our IP address is 10.0.2.40                                                                                                                                     
Filename 'flash_eth.img'.                                                                                                                                                                   
Load address: 0x81000000                                                                                                                                                                    
Loading: ##################################################  444 Bytes                                                                                                                      
         26.4 KiB/s                                                                                                                                                                         
done                                                                                                                                                                                        
Bytes transferred = 444 (1bc hex)                                                                                                                                                           
## Executing script at 81000000                                                                                                                                                             
Using e1000#0 device                                                                                                                                                                        
TFTP from server 10.0.1.19; our IP address is 10.0.2.40                                                                                                                                     
Filename 'apalis-tk1/flash_eth.img'.                                                                                                                                                        
Load address: 0x81000000                                                                                                                                                                    
Loading: T T                                                                                                                                                                                
Abort

All files are present in the tftp directory and other single files like e.g. the MBR from the <tftp-root>/apalis-tk1 directory can be loaded (command from update_1 in flash_eth.scr):

Apalis TK1 # tftpboot ${loadaddr} apalis-tk1/mbr.bin && mmc dev 0 0 && mmc write ${loadaddr} 0x0 0x1                                                                                        
Using e1000#0 device                                                                                                                                                                        
TFTP from server 10.0.1.19; our IP address is 10.0.2.40                                                                                                                                     
Filename 'apalis-tk1/mbr.bin'.                                                                                                                                                              
Load address: 0x81000000                                                                                                                                                                    
Loading: ##################################################  512 Bytes                                                                                                                      
         44.9 KiB/s                                                                                                                                                                         
done                                                                                                                                                                                        
Bytes transferred = 512 (200 hex)                                                                                                                                                           
switch to partitions #0, OK                                                                                                                                                                 
mmc0(part 0) is current device                                                                                                                                                              
                                                                                                                                                                                            
MMC write: dev # 0, block # 0, count 1 ... 1 blocks written: OK                                                                                                                             
Apalis TK1 #

If I use a tftp client ( sudo apt-get install tftp) on the same local machine I can get both flash_eth.img files without any problems (means the tftp server should not be the problem):

tftp> get 10.0.1.19:flash_eth.img
Received 450 bytes in 0.0 seconds
tftp> get 10.0.1.19:apalis-tk1/flash_eth.img
Received 2507 bytes in 0.1 seconds
tftp> get 10.0.1.19:apalis-tk1/mbr.bin
Received 512 bytes in 0.3 seconds

What could be the reason for that?

What could be the reason for that?

I suspect either some misconfiguration or otherwise network issue. Does the link LED light up during that second TFT transfer?

Here how my preparation looks like:

[user@host Apalis-TK1_LXDE-Image_2.7.3]$ ./update.sh -o /var/lib/tftpboot/
Apalis TK1 rootfs detected
Image file apalis-tk1.img has been successfully generated!

Creating MBR file and do the partitioning
0+0 records in
0+0 records out
0 bytes copied, 8.9586e-05 s, 0.0 kB/s
Model:  (file)
Disk /home/zim/Toradex/Linux/Releases/2.7b3/Apalis-TK1_LXDE-Image_2.7.3/apalis-tk1_bin/mbr.bin: 30760960s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End        Size       Type     File system  Flags
 1      8192s   40959s     32768s     primary               lba
 2      40960s  30719999s  30679040s  primary


Creating VFAT partition image with the kernel
mkfs.fat 4.1 (2017-01-24)
mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows

Creating rootfs partition image
735+0 records in
735+0 records out
770703360 bytes (771 MB, 735 MiB) copied, 0.278934 s, 2.8 GB/s
mke2fs 1.43.3 (04-Sep-2016)
Discarding device blocks: done                            
Creating filesystem with 188160 4k blocks and 47040 inodes
Filesystem UUID: 34466114-6be6-4f9e-b0b9-0d5f7cc9b630
Superblock backups stored on blocks: 
	32768, 98304, 163840

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

tune2fs 1.43.3 (04-Sep-2016)
Setting maximal mount count to -1
Setting interval between checks to 0 seconds
Successfully copied data to target folder.

To flash the Apalis/Colibri T20/T30/TK1 module a running U-Boot is required. Boot
the module to the U-Boot prompt and

insert the SD card, USB flash drive or when using TFTP connect Ethernet only
and enter:
'run setupdate'

then to update all components enter:
'run update'

Alternatively, to update U-Boot enter:
'run update_uboot'
to update a component stored in UBI enter:
'run prepare_ubi' (for Colibri T20)
followed by one of:
'run update_kernel'
'run update_fdt' (for device tree enabled kernels)
'run update_rootfs'

If you don't have a working U-Boot anymore, connect your PC to the module's USB
client port, bring the module into the recovery mode and start the update.sh
script with the -d option. This will copy U-Boot into the module's RAM and
execute it.

And on the target:
U-Boot 2016.11-2.7.3+gf0e4149 (Jun 30 2017 - 23:28:22 +0200)

TEGRA124
DRAM:  2 GiB
MMC:   Tegra SD/MMC: 0, Tegra SD/MMC: 1, Tegra SD/MMC: 2
In:    serial
Out:   serial
Err:   serial
Model: Toradex Apalis TK1 2GB V1.1A, Serial# 02864654
Net:   No ethernet found.
Hit any key to stop autoboot:  1  0 
Apalis TK1 # run setupdate
MMC: no card present
** Bad device mmc 1 **
MMC: no card present
** Bad device mmc 2 **
starting USB...
USB0:   USB EHCI 1.10
USB1:   USB EHCI 1.10
USB2:   USB EHCI 1.10
scanning bus 0 for devices... 1 USB Device(s) found
scanning bus 1 for devices... 3 USB Device(s) found
scanning bus 2 for devices... 1 USB Device(s) found
** Bad device usb 0 **
e1000: no NVM
Using e1000#0 device
TFTP from server 192.168.10.1; our IP address is 192.168.10.2
Filename 'flash_eth.img'.
Load address: 0x81000000
Loading: *##################################################  444 Bytes
	 0 Bytes/s
done
Bytes transferred = 444 (1bc hex)
## Executing script at 81000000
Using e1000#0 device
TFTP from server 192.168.10.1; our IP address is 192.168.10.2
Filename 'apalis-tk1/flash_eth.img'.
Load address: 0x81000000
Loading: *##################################################  2.4 KiB
	 0 Bytes/s
done
Bytes transferred = 2479 (9af hex)
## Executing script at 81000000
enter "run update" to update the entire module

I suspect either some misconfiguration or otherwise network issue.

I have the Apalis Eval Board and the host machine both connected to a switch…

Does the link LED light up during that second TFT transfer?

Yes it does light up during the attempt for the second TFT transfer.

The RESET MOCI# led on the Apalis Eval board is lighting up all the time but I think that should not have an impact. (The u-boot version on the SOM seems to be a patched one already.)

The permissions should be ok as well in the tftp root

ll /srv/tftp/
total 24
drwxrwxr-x 3 tftp nogroup 4096 Sep 20 16:05 ./
drwxr-xr-x 3 root root    4096 Sep 20 16:04 ../
drwxrwxr-x 2 tftp nogroup 4096 Sep 20 16:05 apalis-tk1/
-rwxrwxr-x 1 tftp nogroup  710 Sep 20 16:08 flash_blk.img*
-rwxrwxr-x 1 tftp nogroup  444 Sep 20 16:08 flash_eth.img*
-rwxrwxr-x 1 tftp nogroup  307 Sep 20 16:08 flash_mmc.img*

and sub directory

ll /srv/tftp/apalis-tk1/
total 912228
drwxrwxr-x 2 tftp nogroup     4096 Sep 20 16:05 ./
drwxrwxr-x 3 tftp nogroup     4096 Sep 20 16:05 ../
-rwxrwxr-x 1 tftp nogroup   594944 Sep 20 16:08 apalis-tk1.img*
-rwxrwxr-x 1 tftp nogroup 16777216 Sep 20 16:08 boot.vfat*
-rwxrwxr-x 1 tftp nogroup     2723 Sep 20 16:08 flash_blk.img*
-rwxrwxr-x 1 tftp nogroup     2479 Sep 20 16:08 flash_eth.img*
-rwxrwxr-x 1 tftp nogroup      512 Sep 20 16:08 mbr.bin*
-rwxrwxr-x 1 tftp nogroup 67108864 Sep 20 16:08 root.ext3-100*
-rwxrwxr-x 1 tftp nogroup 67108864 Sep 20 16:08 root.ext3-101*
-rwxrwxr-x 1 tftp nogroup 67108864 Sep 20 16:08 root.ext3-102*
-rwxrwxr-x 1 tftp nogroup 67108864 Sep 20 16:08 root.ext3-103*
-rwxrwxr-x 1 tftp nogroup 67108864 Sep 20 16:08 root.ext3-104*
-rwxrwxr-x 1 tftp nogroup 67108864 Sep 20 16:08 root.ext3-105*
-rwxrwxr-x 1 tftp nogroup 67108864 Sep 20 16:08 root.ext3-106*
-rwxrwxr-x 1 tftp nogroup 67108864 Sep 20 16:08 root.ext3-107*
-rwxrwxr-x 1 tftp nogroup 67108864 Sep 20 16:08 root.ext3-108*
-rwxrwxr-x 1 tftp nogroup 67108864 Sep 20 16:08 root.ext3-109*
-rwxrwxr-x 1 tftp nogroup 67108864 Sep 20 16:08 root.ext3-110*
-rwxrwxr-x 1 tftp nogroup 67108864 Sep 20 16:08 root.ext3-111*
-rwxrwxr-x 1 tftp nogroup 67108864 Sep 20 16:08 root.ext3-112*
-rwxrwxr-x 1 tftp nogroup 38797312 Sep 20 16:08 root.ext3-113*
-rwxrwxr-x 1 tftp nogroup    49826 Sep 20 16:08 tegra124-apalis-eval.dtb*
-rwxrwxr-x 1 tftp nogroup  5450280 Sep 20 16:08 uImage*
-rwxrwxr-x 1 tftp nogroup      218 Sep 20 16:08 versions.txt*

I have the Apalis Eval Board and the host machine both connected to a switch…

Me, too. A HP 1820-8G one using a separate VLAN for target to workstation communication.

Yes it does light up during the attempt for the second TFT transfer.

Strange that it does not transfer the file then. Have you looked at what is going on with a network analyser e.g. wireshark or the like?

The RESET MOCI# led on the Apalis Eval board is lighting up all the time but I think that should not have an impact.

Yeah, right. Then you are already using -next stuff I guess as this was a recent change we did also in conjunction with the new V1.2A hardware revision in order to avoid multiple transitions on RESET_MOCI# during system boot. The only thing it would affect on our regular evaluation board is the USB hub. It should not have any effect on gigabit Ethernet operation at all.

I changed the network setup (host machine which runs a dhcp server and is directly connected to Apalis Eval Board) and tried it again with the same result:

Apalis TK1 # run setethupdate                                                                                                                                            
e1000: no NVM                                                                                                                                                            
Using e1000#0 device                                                                                                                                                     
TFTP from server 192.168.10.1; our IP address is 192.168.10.2                                                                                                            
Filename 'flash_eth.img'.                                                                                                                                                
Load address: 0x81000000                                                                                                                                                 
Loading: ##################################################  444 Bytes                                                                                                   
         0 Bytes/s                                                                                                                                                       
done                                                                                                                                                                     
Bytes transferred = 444 (1bc hex)                                                                                                                                        
## Executing script at 81000000                                                                                                                                          
Using e1000#0 device                                                                                                                                                     
TFTP from server 192.168.10.1; our IP address is 192.168.10.2                                                                                                            
Filename 'apalis-tk1/flash_eth.img'.                                                                                                                                     
Load address: 0x81000000                                                                                                                                                 
Loading: T T T T T T T T T T                                                                                                                                             
Retry count exceeded; starting again                                                                                                                                     
Apalis TK1 #

Strange that it does not transfer the file then. Have you looked at what is going on with a network analyser e.g. wireshark or the like?

I tried to update another old V1.0B TK1 SOM with old u-boot on it. With again the same result. It must be some kind of tftp server issue. However I check the traffic with wireshark as well now.

The capture shows TFTP Read Requests from the SOM and the host sends the TFTP packet over and over again to the SOM run_setethupdate.pcap

How about if you manually directly try to download resp. file?

U-Boot 2016.11-2.7.3+gf0e4149 (Jun 30 2017 - 23:28:22 +0200)

TEGRA124
DRAM:  2 GiB
MMC:   Tegra SD/MMC: 0, Tegra SD/MMC: 1, Tegra SD/MMC: 2
In:    serial
Out:   serial
Err:   serial
Model: Toradex Apalis TK1 2GB V1.1A, Serial# 02864654
Net:   No ethernet found.
Hit any key to stop autoboot:  0 
Apalis TK1 # setenv ipaddr 192.168.10.2
Apalis TK1 # setenv serverip 192.168.10.1
Apalis TK1 # pci enum
Apalis TK1 # tftpboot ${loadaddr} ${board_name}/flash_eth.img
e1000: no NVM
Using e1000#0 device
TFTP from server 192.168.10.1; our IP address is 192.168.10.2
Filename 'apalis-tk1/flash_eth.img'.
Load address: 0x81000000
Loading: ##################################################  2.4 KiB
         0 Bytes/s
done
Bytes transferred = 2479 (9af hex)
Apalis TK1 #

Does not work either:

Apalis TK1 # tftpboot ${loadaddr} "${board_name}/flash_eth.img" && source ${loadaddr}                                                  
Using e1000#0 device                                                                                                                   
TFTP from server 10.0.1.19; our IP address is 10.0.2.84                                                                                
Filename 'apalis-tk1/flash_eth.img'.                                                                                                   
Load address: 0x81000000                                                                                                               
Loading: T T T T T T T T T T                                                                                                           
Retry count exceeded; starting again

The file resides in the correct tftp output directory

ll /srv/tftp/apalis-tk1/ | grep flash_eth.img
-rwxrwxr-x 1 tftp nogroup     2479 Sep 20 16:08 flash_eth.img*

and the tftp server is serving from this directory:

ps ax | grep tftp
 1339 ?        Ss     0:00 /usr/sbin/in.tftpd --listen --user tftp --address :69 --secure /srv/tftp
24424 ?        S      0:00 /usr/sbin/in.tftpd --listen --user tftp --address :69 --secure /srv/tftp
24428 pts/0    S+     0:00 grep --color=auto tftp

I could not figure out what is misconfigured yet… even if I restart the tftp server (to get rid of the doubled entries in the ps ax | grep tftp output) and if I use another root tftp output directory it does not work.

Looking at your IP addresses it seems you are not using a class C network which is probably the default in U-Boot:

Apalis TK1 # echo $netmask
255.255.255.0

Have you updated your netmask accordingly?

I updated it. Makes no difference. (Shouldn’t the load of the first file fail then as well?)

Yeah, you are right. So it rather seems to have to do with the file being served from the root TFTP folder vs. from an additional sub folder, not?

Right, I try to get it flashed somehow… I let you know about any changes in status.
(I can flas another file from the subfolder without any problems. That’s also not the root cause.)

Because the only file from the sub directory which cannot be loaded is apalis-tk1/flash_eth.img I rebuild it from its source file with sudo ./mk-u-boot-scripts.sh before deploying the image files into the tftp output directory with ./update.sh -m 2 -o /srv/tftp (non-Mainliine version of v2.7b3). However the file cannot be loaded.

I uninstalled the TFTP server, its settings, the output directory and made a clean new install and setup. However the file apalis-tk1/flash_eth.img can still not be loaded. For further root cause analysis I copy pasted the commands from fwd_eth.scr and from flash_eth.scr into u-boot to figure out which of the other files can be loaded with the commands and which not.

  • tftpboot ${loadaddr} "flash_blk.img" && source ${loadaddr} load passed
  • tftpboot ${loadaddr} "flash_mmc.img" && source ${loadaddr} load passed
  • tftpboot ${loadaddr} "flash_eth.img" && source ${loadaddr} load passed (just the first file, not the second one)
  • tftpboot ${loadaddr} "${board_name}/flash_eth.img" && source ${loadaddr} (apalis-tk1/flash_eth.img) load failed
  • run update_fdt (apalis-tk1/tegra124-apalis-eval.dtb) load failed
  • run update_kernel (apalis-tk1/uImage) load failed
  • run update_uboot (apalis-tk1/apalis-tk1.img) load failed
  • run update_1 (apalis-tk1/mbr.bin) load passed
  • run update_2 (apalis-tk1/boot.vfat) load failed

The file permissions of the files are the same, the one thing which is different is the size of the files:

ll /srv/tftp/
drwxrwxr-x 2 root nogroup 4096 Sep 28 11:15 apalis-tk1/
-rwxrwxr-x 1 root nogroup  710 Sep 28 11:22 flash_blk.img*
-rwxrwxr-x 1 root nogroup  444 Sep 28 11:22 flash_eth.img*
-rwxrwxr-x 1 root nogroup  307 Sep 28 11:22 flash_mmc.img*

ll /srv/tftp/apalis-tk1/
-rwxrwxr-x 1 root nogroup   594944 Sep 28 11:22 apalis-tk1.img*
-rwxrwxr-x 1 root nogroup 16777216 Sep 28 11:22 boot.vfat*
-rwxrwxr-x 1 root nogroup     2723 Sep 28 11:22 flash_blk.img*
-rwxrwxr-x 1 root nogroup     2479 Sep 28 11:22 flash_eth.img*
-rwxrwxr-x 1 root nogroup      512 Sep 28 11:22 mbr.bin*
-rwxrwxr-x 1 root nogroup 67108864 Sep 28 11:22 root.ext3-100*
-rwxrwxr-x 1 root nogroup 67108864 Sep 28 11:22 root.ext3-101*
-rwxrwxr-x 1 root nogroup 67108864 Sep 28 11:22 root.ext3-102*
-rwxrwxr-x 1 root nogroup 67108864 Sep 28 11:22 root.ext3-103*
-rwxrwxr-x 1 root nogroup 67108864 Sep 28 11:22 root.ext3-104*
-rwxrwxr-x 1 root nogroup 67108864 Sep 28 11:22 root.ext3-105*
-rwxrwxr-x 1 root nogroup 67108864 Sep 28 11:22 root.ext3-106*
-rwxrwxr-x 1 root nogroup 67108864 Sep 28 11:22 root.ext3-107*
-rwxrwxr-x 1 root nogroup 67108864 Sep 28 11:22 root.ext3-108*
-rwxrwxr-x 1 root nogroup 67108864 Sep 28 11:22 root.ext3-109*
-rwxrwxr-x 1 root nogroup 67108864 Sep 28 11:22 root.ext3-110*
-rwxrwxr-x 1 root nogroup 67108864 Sep 28 11:22 root.ext3-111*
-rwxrwxr-x 1 root nogroup 67108864 Sep 28 11:22 root.ext3-112*
-rwxrwxr-x 1 root nogroup 38797312 Sep 28 11:22 root.ext3-113*
-rwxrwxr-x 1 root nogroup    49826 Sep 28 11:22 tegra124-apalis-eval.dtb*
-rwxrwxr-x 1 root nogroup  5450280 Sep 28 11:22 uImage*
-rwxrwxr-x 1 root nogroup      218 Sep 28 11:22 versions.txt*

When I flashed flash_blk.img and flash_mmc.img I got some warnings due to some invalid load command. However I am just interested in if the can be flashed at all:

Apalis TK1 # tftpboot ${loadaddr} "flash_blk.img" && source ${loadaddr}                                                                                                                                     
Using e1000#0 device                                                                                                                                                                                        
TFTP from server 10.0.1.19; our IP address is 10.0.2.84                                                                                                                                                     
Filename 'flash_blk.img'.                                                                                                                                                                                   
Load address: 0x81000000                                                                                                                                                                                    
Loading: ##################################################  710 Bytes                                                                                                                                      
         62.5 KiB/s                                                                                                                                                                                         
done                                                                                                                                                                                                        
Bytes transferred = 710 (2c6 hex)                                                                                                                                                                           
## Executing script at 81000000                                                                                                                                                                             
Card did not respond to voltage select!                                                                                                                                                                     
** Bad device mmc 1 **                                                                                                                                                                                      
Apalis TK1 # tftpboot ${loadaddr} "flash_mmc.img" && source ${loadaddr}                                                                                                                                     
Using e1000#0 device                                                                                                                                                                                        
TFTP from server 10.0.1.19; our IP address is 10.0.2.84                                                                                                                                                     
Filename 'flash_mmc.img'.                                                                                                                                                                                   
Load address: 0x81000000                                                                                                                                                                                    
Loading: ##################################################  307 Bytes                                                                                                                                      
         1000 Bytes/s                                                                                                                                                                                       
done                                                                                                                                                                                                        
Bytes transferred = 307 (133 hex)                                                                                                                                                                           
## Executing script at 81000000                                                                                                                                                                             
Card did not respond to voltage select!                                                                                                                                                                     
** Bad device mmc 1 **

The root cause of this issue was that the “maximum transmission unit” (MTU) of the hosts network interface was set to a high value. Decreasing the value to 1500 solves the problem (e.g. sudo ifconfig eth0 mtu 1500).

Yes, I just wanted to suggest you checking that. Wireshark should also have indicated that I assume. Unfortunately I am not quite sure about the exact status of IP fragmentation support and such in U-Boot.

When I flashed flash_blk.img and flash_mmc.img I got some warnings due to some invalid load command.

Yes, they are only intended to be run from a block device or SD/MMC card resp.