Cannot remove unused Uplink Ports on Distributed Swtich vSphere 5.1

I have recently removed a host with 6 Nics from a Distributed Switch .  The remaining hosts only have 2 Nics so I would like to remove the unused uplink ports on the vDS.  I have tried this by dialling down the number of uplink ports on the Switch properties but it throws an error as it tries to remove uplinks with NICs assigned and are in use instead of the unused uplinks. 

Is there a way to do this?

Regards

Tony