The ticket mentioned got the solution. The order of the arguments was the trick. The correct setup for me is:
setenv setup 'setupargs console=null console=${console},${baudrate}n8 ${memargs} consoleblank=0; setenv bootm_boot_mode sec'
setenv vidargs 'video=mxsfb:800x480M-16@60,pixclockpol=1'
setenv defargs ' vt.global_cursor_default=0 consoleblank=0 quiet console=${console}'