I read an interesting take from Forrester the other day about TPC benchmarks. Noel Yuhanna makes the case that “Transaction Processing Performance Council (TPC) benchmarks, once widely accepted as the standard DBMS benchmark, are becoming obsolete.”   Why?  He says:

“ First, all top-tier DBMS vendors such as […] are delivering high performance and scalability to support most large workloads. Second, TPC benchmarks no longer reflect the complex workloads of today’s real-world deployments. Third, customers that need high-end performance often prefer internal benchmarks to TPC benchmarks. Finally, virtualization, cloud computing, and database-as-a-service are changing the way customers deploy databases, and TPC does not address these architectures.” (from “TPC Benchmarks Don’t Matter Anymore:  Features and Cost are Key Factors When Choosing a DBMS” by Noel Yuhanna with Mike Gilpin and David D’Silva, March 6, 2009, copyright Forrester Research)

This touches on a larger issue that we have blogged about before – we are seeing here at AMD that data center demands on the whole are changing. There is less stock being put into how a hardware or software product performs in a lab, and more emphasis on total cost of ownership and how they perform in real-life scenarios.

 I want to be clear, there is a place for benchmarks, and we absolutely value how our products perform against the competition in a standardized setting. For instance, our 45nm Quad-Core AMD Opteron™ 8384 processor just set another world record virtualization score on the VMmark test (29.11@19 tiles on a Sun Fire X4600 M2 server running VMware ESX Server 3.5.0 hypervisor.) We also just set the record on the SPECweb2005 benchmark, measuring web server performance, with 45nm Quad-Core AMD Opteron processors in the four-socket 2RU x64 Sun Fire X4440 server.

However, both AMD and our OEM partners recognize that they are merely a component of the overall value equation. The days of saying that purchasing decisions can be simplified based on a single benchmark are long gone.

Benchmarks don’t always reflect real-world implementations but they have been accepted by and large by the industry as a good initial performance indicator. In fact, benchmark progression can even sometimes be interesting to track as multicore processing is crunching data at unprecedented speeds, resulting in some eye-popping results. AMD recognizes that there are always going to be applications that require the best raw performance. However, recent processor unit data from Mercury Research shows that high-end, performance processors are less than 10 percent of the market today.  This tells us that customers ultimately buy on more than just raw performance. They look at how a system is optimized for software, what level of energy is it going to consume, how scalable is the hardware, and in today’s economic environment, price-performance and TCO are more critical than ever.

Whether it is a DBMS or not, Forrester raises a really good point here, and one that – at a minimum – is worth discussing, regardless of if you agree or disagree. There are many factors beyond TPC performance  - reliability, manageabilty, security, performance to name a few – that impact a decision to buy and are spanning all enterprise computing environments. More specifically, how these factors impact the ongoing demands of your business (something we like to call the “workloads that matter”) is only going to become increasingly important.

John Fruehe is the Director of Business Development for Server/Workstation products at AMD. His postings are his own opinions and may not represent AMD’s positions, strategies or opinions. Links to third party sites are provided for convenience and unless explicitly stated, AMD is not responsible for the contents of such linked sites and no endorsement is implied.