Dan, what you are talking about sounds a lot like the "memory balloon driver" in VMware ESX. This comes into play when the memory requirements of all the VMs exceeds the available physical memory. ESX will inflate the balloon on the lower priority VMs forcing the guest OS in those VM to decide what pages to page out to their swap file.
I don't know if this driver exists in the other VMware products.