Seems like the industry wants to treat virtualization like a “high performance computing” workload. There is a growing obsession with hardware vendors (including AMD) to tout top VMmark benchmark scores. The truth is any analysis of virtualization performance needs to consider more than just “raw performance.”  So let’s go “beyond the score” and take a closer look at the systems posting some of the top VMmark scores.

First, a short bit on VMmark . VMmark is a consolidation benchmark that generates an aggregate score of individual VMs for a given number of tiles. A tile is six VMs running common load-generation tools that represent typical workloads: web server, file server, mail server, database, java server as well as an idle VM. One client computer is used to generate the load for one tile.

Looking at top VMmark scores you find systems that can run over 100 VMs per server. And there is a lot of chatter about how a 2 Socket Intel Xeon “Gainestown” processor-based server can run 16 tiles (or 96 VMs). However, there is no reference to the cost of the systems posting scores. VMmark documents the system configuration for the benchmark so you can take a stab at pricing these configurations on-line at the hardware vendor sites. (See slides 3 and 4 of presentation) In doing so we found that some of the top VMmark Intel Xeon “Gainestown” processor-based server configurations price out at about 175% and in some cases even higher than the top performing AMD Opteron™ processor (“Shanghai”) configurations (based on April 16, 2009 prices). Even in the performance-oriented high performance computing world this would turn heads. Cost does matter.

Going a step further, we now have the information to evaluate the price/performance of some of these systems by taking the estimated system cost and dividing it by number of VMs achieved during the VMmark run. While large number of VMs might be impressive – most IT professionals in today’s economy are focused on balancing performance and price—looking at the cost per VM helps to better understand the cost of putting the system in action. What you find is the system with the top VMmark score is not the system that gives you the best cost per VM.

Now that we have looked at the VMmark systems configurations, what type of virtualization configurations are customers really running? When looking at customer case studies posted on hardware and software vendors’ sites we find servers configurations ranging from 16GB to 64G of memory as more of the norm. We also don’t find many data centers pushing 100 VMs on a system. Responses to SearchDataCenter.com’s 2008 Purchasing Intentions Survey reveals that only 5% of respondents are running more than 25 VMs on a server – 61% are running less than 10 VMs per server and 33% are running 10 to 25 VMs per server. And since many customers are implementing virtualization as a cost saving strategy – we don’t see many of these customers buying the top bin “performance” processor models, which by design tend to consume the most power.

We did the pricing on systems configurations using energy efficient processors and more typical memory configurations for virtualization (again based on April 16, 2009 pricing), comparing both system pricing and cost per VMs (see slide 5 of presentation). Take a look for yourself. We think you will agree considering performance and price can give you a better view of its overall value.

So the question remains: how do *you* define “simply spectacular” virtualization? Is it in terms of raw performance or is it price/performance? Hopefully after reading this post, you have a different answer than when you started.

Margaret Lewis (@margaretjlewis) is a Product Marketing Director at AMD. Her postings are her own opinions and may not represent AMD’s positions, strategies or opinions. Links to third party sites are provided for convenience and unless explicitly stated, AMD is not responsible for the contents of such linked sites and no endorsement is implied.