I end up downgrade the host back to 4.1 and afterwards the prepare went smoothly. VMware support suggested following steps if I want to use ESXi 5.1, which is not tested by me:
- Took an ESXi 5.1 host and downloaded the vcloud director agent from VCD cell from /opt/vmware/vcloud-director/agent/vcloudagent-esx51-5.1.0-799577.vib
- Manually installed it using the following command on the esxi host: esxcli software vib install -v=/vmfs/volumes/10.131.0.200-Local/vcloudagent-esx51-5.1.0-799577.vib
- Post install, I stopped the VCD service and made the following changes in the db: Update managed_server set is_prepared=1 where display_name like '%<hostname>%';
- Restarted the service and the Host should show up as prepared.
VMware Support gave following reason for why it failed on ESXi 5.1
"The reason this issue is not seen in 4.1 and might be observed only in 5.1 with slower SD cards is that the newer bootbank takes a while to update the vibs and hence could possibly timeout with slower responding SD cards. Though you replaced the SD cards but the throughput/performance seems to be exactly as the old SD card."
The reason they gave does not make sense to me. As why something was working perfectly fine will break after upgrade? Isn't this some kind of bug? I have asked if they have any patches or hotfix for this issue. Haven't got an response yet.