VMware ESX and Enhanced VMotion Compatibility

The classic VMotion problem of recent times is the customer who buys tin from a vendor, only to find out that the processor number e.g. 53xx and 54xx, is actually quite significant. The principle difference which will prevent VMotion is the addition of SSE4.1 to the 54xx range of Intel processors (if you previously only had 53xx Intel processors).

An application using the SSE4.1 feature (or is aware of this feature) when VMotioned to a non-SSE4.1 host would likely blue screen trying to make use of this feature on the older host.

People then recalled the CPU mask feature in VMware - we’ll just mask it out they thought. Unfortunately VMware declared this was an unsupported mask with KB1993 and KB1991.

Note that for production environments, VMware neither supports nor recommends modifying VMotion masks for SSE3, SSE3, or SSE4.1 because of the risk of failure of the application or guest operating system after migration.

The reason soon became clear:

* SSE features can be used by user-level code (applications).
* Mask does not work for user-level code (i.e. applications).
* In user-level code, CPUID is executed directly on hardware and is not intercepted by VMware.
* Thus, VM cannot reliably hide SSE from an application

The best that customers could do was buy compatible hardware, and keep themselves right with vendor compatibility charts such as these ones from Dell and HP.

ESX 3.5 U2 brought a new feature: Enhanced VMotion Compatibility (EVC for short) which I’ll explain…

New CPUs are coming out with a facility to ‘turn off’ (mask) features that would make them VMotion incompatible with other hosts running older (compatible) CPUs.

For Intel this is called Flex Migration. It is clear that for processors that are in different families but support Flex Migration will negotiate the common feature set amongst the CPUs.

Where it is not absolutely clear, is where you have an existing cluster and you want to add in hosts with the new Flex Migration facility. You should be able to do this if you satisfy the following requirements (paste from Basic Administration Guide):

All hosts in the cluster must either have hardware live migration support(Intel FlexMigration or AMD‐V Extended Migration) or have the CPU whose baseline feature set you intend to enable for the cluster. For specific host processors supported, see Table 15‐1.

Table 15-1. Processors Supported in EVC Clusters

Vendor Baseline Processor Processors Supported
Intel Intel Core 2 (Merom) Intel Core 2 (Merom)
45nm Core 2 (Penryn)
AMD AMD Second Generation Opteron (Rev‐E/F) AMD Second Generation Opteron (Rev‐E/F)
AMD Third Generation Opteron (Barcelona)

It is therefore advised that customers buying new tin, if they cannot acquire a processor that is exactly the same or in the same family – purchase a Flex Migration enabled CPU.

It should also be clear that only masks the features on the processors, it does not turn them off. Programs which are written to guidelines will behave, those which are coded badly will see these features regardless – and more importantly will encounter issues if VMotioned to a host that used that feature. Pasted from VMware:

EVC utilizes hardware support to modify the semantics of the CPUID instruction only. It does not disable the feature itself. For example, if an attempt to disable SSE4.1 is made by applying the appropriate masks to a CPU that has these features, this feature bit indicates SSE4.1 is not available to the guest or the application, but the feature and the SSE4.1 instructions themselves (such as PTESE and PMULLD) are still available for use. This implies applications that do not use the CPUID instruction to determine the list of supported features, but use try‐catch undefined instructions (#UD) instead, can still detect the existence of this feature.

Therefore, for EVC to be useful, application developers must adhere to recommended guidelines on feature detection. CPU vendors recommend that software programmers query CPUID prior to using special instructions and features available on their CPUs. If this guideline is followed by programmers, EVC is a reliable mechanism for live migration of x86 virtual machines across varied hardware. Thus, you can use EVC to enable an entire cluster to use the same set of basic features, allowing migration with VMotion across any two nodes in the cluster. VirtualCenter can also set up new hardware add‐ons to the cluster and apply these masks.

Intel provide a comparison tool to check beforehand if a CPU possesses Flex Migration capability.

Further Information on this subject can be located in the VMware VMotion Guide and the ESX 3.5U2 Basic Admin Guide Page 238 Onwards.

Bron: http://www.novosco.com/articles/2008/08/19/vmware-esx-and-enhanced-vmotion-compatibility/