I’d like to start off my blog today by extending my congratulations to my AMD colleagues on capturing the VMmark benchmark crown.  Our CMO Nigel Dessau  covers this achievement today  in his blog here. 

While the best way in my opinion for enterprises to compare hardware is to load their own data in their own datacenters, it is encouraging to see that "benchmarkship" is already entering the world of virtualization.  Because doing real-world testing is difficult with virtualization, this is a good first step.  And it’s interesting to see how benchmark developers are begining to develop guidelines around how to properly conduct these virtualization benchmarking activities.  These developers are essentially telling the industry that they are going to insist that we standardize our testing methodology, so that we are comparing “apples to apples”.  Personally, I think this is a great idea and I hope it will help ensure that meaningfull, truthful data is available to those who need it to make their virtualization purchase decisions.

Of course guidelines are meaningless if they aren’t enforced.  That isn’t the case here.  We’re seeing more examples of how benchmark developers are willing to enforce these guidelines. Adhering to guidelines is something that AMD learned the importance of firsthand in 2007.  And we’re not the only ones. Recently the benchmarks for one of our competitor’s solutions had to be withdrawn from the VMmark results page due to “noncompliance”.

I also want to point out that it is possible to follow the guidelines and still not necessarily tell the whole story.  For example, a customer recently pointed out to me that our competitor posted some system benchmarks, but somehow omitted any reference to the AMD-based Dell R905, even though benchmarks for this AMD system had been posted on the VMWare’s site for more than a month.   Perhaps it was an oversight?

While real-world datacenter comparisons using real business data is the best way to evaluate systems, I beleive that as vendors we need to take the benchmark high road - we need to use benchmarks as one tool to help customers evaluate technology - so they can make wiser choices. We need to fairly and honestly benchmark systems and report all the pertinent data.   Trust and credibility go hand-in-hand, and if we do not keep the process fair and open, then the hard work of many people within my company, at our customers and partners and even within our competitor is at risk.  As a demonstration of telling the whole story, let me point you to the “complete” results of the VMWare benchmarks. 

https://twitter.com/PatrickMoorhead

Pat Moorhead is Vice President of Advanced Marketing at AMD.  His postings are his own opinions and may not represent AMD’s positions, strategies or opinions. Links to third party sites are provided for convenience and unless explicitly stated, AMD is not responsible for the contents of such linked sites and no endorsement is implied.