vROps Memory Metrics - Prior to vROps 6.7+

Memory metrics in vROps have caused many customers confusion and I hope this post will explain the different metrics available and why the values don't necessarily reflect what you might think. Let's start with a few definitions to help set the context.

Virtual Machine / Guest: The virtual operating system object where the guest OS such as Windows Server is installed.
Host: The physical blade/rack mount server where ESXi is installed. The host consists of RAM, CPU, network etc.
Cluster: A group of hosts.

Machine memory: The RAM that’s physically installed in the hardware that comprises the ESXi server. This applies at the host layer.
Guest “physical” memory: The amount of physically installed memory presented to the virtual machine and made available when the virtual machine is running.
Overhead Memory: The amount of machine memory allocated to a virtual machine beyond its reserved amount.
At the virtual machine layer, this is the amount of machine memory used by the VMkernel to run the virtual machine
At the host layer, this is the total of all overhead metrics for powered-on virtual machines, plus the overhead of running vSphere services.
Granted Memory: At the virtual machine layer, the guest physical memory that is mapped to machine memory.
Includes shared memory. Does not include overhead.
At the host layer, it is the sum of all granted metrics for all powered-on virtual machines, plus machine memory for vSphere services on the host.
Consumed Memory: At the virtual machine layer, this is the amount of guest physical memory used.
VM consumed memory = memory granted – memory saved due to memory sharing.
It includes shared memory and memory that is reserved (but may not be used). Overhead memory is not included.
At the cluster layer, this metric shows the amount of host machine memory used by all powered on virtual machines in the cluster.
A clusters consumed memory consists of virtual machine consumed memory and overhead memory.
Shared Memory: Memory sharing is a proprietary ESXi technique that can help achieve greater memory density on a host.
When virtual machines that have the same applications or components loaded, or contain common data, a host uses a proprietary Transparent Page Sharing (TPS) technique to eliminate redundant copies of memory pages.
The amount of memory saved by memory sharing depends on whether the workload consists of nearly identical machines which might free up more memory.
Note: Due to security concerns, inter-virtual machine transparent page sharing is disabled by default.
Page sharing is being restricted to intra-virtual machine memory sharing.
This means page sharing does not occur across virtual machines and only occurs inside of a virtual machine.

The VMkernel maps guest physical memory to machine memory. However, because of the VMkernel’s unique memory-management techniques, such as ballooning, memory-sharing, swapping, and other techniques, guest physical memory-to-machine memory mapping does not have a 1:1 correspondence. Specifically, the VMkernel can:

• Map multiple regions of guest physical memory to a single region of machine memory (which the VMkernel does for memory sharing among virtual machines).
• Leave specific regions of guest physical memory and machine memory unmapped (which the VMkernel does during swapping and ballooning).

In the table below, the metrics used in vROps are explained:

Capacity: Machine memory / the amount of physical memory installed per host in a vCenter Cluster.
Provisioned: The amount of memory available to a guest OS / virtual machine.
Used: This metric is pulled from vROPs, specifically the “Memory|Consumed (GB)” metric.

That is certainly a lot to get your head around, but should provide some understanding. I was recently at a customer site where they completed a rightsizing exercise because this environment was run extremely hot (over-provisioning memory etc.). I was not involved in the rightsizing activity but was asked to comment why they had removed 300+GB from the VMs, but they didn't have any capacity to provision new VMs still. The part of the story that wasn't communicated well was that the reclaim activity will target the unused memory.

The rightsizing exercise completed, reclaimed memory which was not being used, hence the provisioned memory figure has fallen, but the used figure has not changed. In an environment not over-provisioned, organisations will typically ensure that the provisioned memory metric closely matches the peak used metrics. Rightsizing will not decrease the used figure and will not free up used resources. The used figure if of course dependent on what is happening in the virtual machine (i.e. the applications running).

Rightsizing is an important best practise activity which should be performed to ensure that your virtual machine provisioned resources closely matches the used resources, when evaluating the minimum, maximum and average values. This is a feature that vROps performs extremely well when determining if a virtual machine is oversized based on these metrics. The metrics being used in the version of vROps prior to 6.7+ are the correct metrics to be using as recommended by VMware. The fluctuation which is seen in the used memory values will be related to the workload on the virtual machines themselves, due to the workload of the applications installed etc.

VMware has reworked the memory metrics in both vROps 6.7 and 7.0 and this has been well documented here by VMware. For vROps 6.7+, vROps will utilize VMware Tools to gather accurate Guest Memory utilization and this is reflected in the Memory|Usage (%) metric.