I’m a fan of benchmarks. I think they are very helpful in allowing consumers to make informed purchase decisions about products. But they generally have some flexibility built into them so you can focus on those elements you want. And this means you can use a benchmark to tell a number of stories – which means you can choose to tell the story you want.

For example, take a recent review by AnandTech entitled “Sixteen Cores, Four Sockets” published on June 17, 2008. This article featured Quad-Core AMD Opteron™ processor-based systems. One of the performance evaluations in this article was a SPECjbb2005 benchmark estimate. What is particularly interesting about this article is that the published estimates list the 4 socket server running AMD Opteron processors model 8356 as 25% faster than the competition while running at similar frequencies and 7% faster than the fastest competitive solution. These results vary widely from the official scores posted on the SPEC site. Now you might ask – how can that be? How can you run what is considered to be an industry standard benchmark and get a different set of numbers? That can’t be right!

Taking a closer look at the SPECjbb2005 benchmark helps to unravel this mystery. SPECjbb2005 is a memory-intensive benchmark that is intended to evaluate the performance of servers running typical Java business applications. Its results evaluate the interaction of the CPU, caches, memory hierarchy, JVM (Java Virtual Machine), and JIT (Just-In-Time) compiler. SPECjbb2005 can be configured to run in a variety of ways, resulting in different performance outcomes. Different configuration = different story. For example, you can get different results based on the operating system used, the version of JVM used, the level of optimization of the JVM and JIT, JVM tuning options, and thread allocations.

The SPECjbb2005 scores published by SPEC tend to be achieved using very aggressive software tuning and processor settings. These settings help to achieve a “best possible score” but do not necessarily reflect how a system would be configured in a data center environment to provide the most stable and efficient performance. The scores published in the AnandTech article, according to the author, are more likely to reflect real world configurations with optimizations used consistent over the different processor architectures.

Indeed – if you do a survey around the internet you can find reference to other SPECjbb2005 scores and estimates that reflect a variety of configuration options and the resulting differences in the benchmark scores:

http://blogs.sun.com/bmseer/entry/sun_fire_x4440_best_opteron

Blog featuring SPECjbb2005 results with the 4 socket Sun Fire x4440 running quad-core AMD Opteron processors with Solaris 10 and Sun JVM. Also highlights power consumption of featured systems – reminding us that in today’s economy of escalating energy costs raw performance has less meaning to data centers than performance/watt.

http://techreport.com/articles.x/13176/4

An article by TechReport featuring SPECjbb2005 estimates for 2 socket servers running quad-core processors with Windows Server 2003 x64 edition and the Sun JVM. The author states the goal of this performance evaluation was to test relative performance on equal footing.

Taking a closer look at the official SPECjbb2005 scores and the estimates published in the various articles, you can see how confusing a benchmark can be. This serves as a reminder to us that benchmarks are just an indicator of performance and that a benchmark like SPECjbb2005, which allows for a wide variety of configurations, can produce a wide variety of results. And remember – the story being told is not always the one that best reflects reality….

Pat Moorhead is Vice President of Advanced Marketing at AMD.His postings are his own opinions and may not represent AMD’s positions, strategies or opinions. Links to third party sites are provided for convenience and unless explicitly stated, AMD is not responsible for the contents of such linked sites and no endorsement is implied.