Ah I see,
The DVS hell sounds scary if for some reason my VCenter VM which is not linked each other got issue or corrupted DB (I got 4 of them managing 4 different AD domain), that is real PITA 
So I guess in this case during the VCenter downtime, the fastest and easiest way is just to restore from Image Backup.
My goal is that when I have rebuild the ESX 4.1 into ESXi 5.1, the networking part can be automatically applied to the VMHost without any issue at all.