Memory De-duplication in VMware

One of the companies I worked for got a Netapp filer and I loved the fact that it would dedupe the data that was sitting on disk.  I got over 40% more storage just by having that sweet little feature on.  I was thinking, “How awesome would it be to dedupe my memory?”  Getting more memory out of my servers would be a nice thing.  Well as it turns out, VMware does this already, but they call it “Transparent Page Sharing.”

VMware uses a technique called Transparent Page Sharing, which looks at each of the virtual machines on a host and creates a hash on each of the memory pages.  If the pages have an identical hash value as another page, from any virtual machine, the pages are de-duplicated by pointing the virtual machine pages at the same memory location on the physical host.

From the picture above you can see that VMs have memory pages in the Host’s memory, as you would imagine, but the pages in blue share the same hash value and can then be shared.  This is one of the mechanisms that VMware uses to allow you to over commit the memory.  For instance it’s possible that the host in the above example has 4GB of RAM and each VM has 2GB of RAM.  If 2GB of the RAM have identical hash values, it’s possible that the host’s memory could be overcommited by 2GB and still not swap.  This is a pretty extreme example though.  I think it would be quite difficult to reach the equivalent of the example that I provided.

In case you’re wondering what happens when one of the VMs needs to change the page that it’s sharing with other VMs; VMware uses a “Copy on Write” procedure where the memory location would be copied to a private memory location for the VM to write to.