Posts tagged with Nigel Dessau

During the last decade I’ve been in the PC industry, I’ve learned a lot about both the positive and negative sides of benchmarking, or what some people call “benchmarketing”.    Before joining AMD, I worked for another technology company.  At that time, we were early believers that application performance is more than just clock speed.  Back then, there wasn’t much science behind our methodology.   If we could get one successful run of Winstone with all of the performance settings maximized, we’d publish the score to the world.

Once I arrived at AMD, I vowed not to duplicate the benchmarking mistakes we made at that other company.  Performance was the cornerstone for AMD Athlon^TM XP processors, and we went to great lengths to make sure the benchmarking methodology had the utmost integrity.  The two guiding principles we used then are the same ones we use today:

1. Transparency : Provide insight into what the benchmarks are measuring, and
2. Repeatable process: Provide the necessary configuration and setting information for someone outside of AMD to be able to consistently duplicate our results

Like everyone in the PC industry, here at AMD we are dependent on the Business Applications Performance Corporation (BAPCo®) as one of the last remaining sources of application-based benchmarks.  BAPCo has two primary benchmarks:  SYSmark, a performance stress test based on high-end applications, and MobileMark, a battery life test for notebooks. AMD is an active member of BAPCo.

As Nigel points out in his blog, MobileMark is beginning to be used as a battery life indicator with consumers in retail.   Designed to be a test for businesses and using high-end business applications, not everyday consumer applications,  MobileMark, interestingly , measures  battery life while the CPU is inactive for roughly 95% of the time (see the MobileMark whitepaper to get a better understanding of “away” time and “think” time).

Meaning, since the test is not doing much, it outputs an impressive-looking score (read: “benchmarketing” at its best), but doesn’t provide the complete picture. As a result-and as many of us have experienced–the battery life we see advertised, is not always the battery life we get in reality. Many factors influence battery life – screen size & brightness, wireless capabilities, and of course application usage, just to name a few.  So why would MobileMark make sense as a way to educate consumers?

Trying to answer this question ourselves, AMD started looking for a benchmark test that captured the real-world battery life of a notebook in everyday, active use.  But, so far, our search has come up cold…and if you know of a test like this, please let me know.

To see if we can solve the problem ourselves, we’ve started  playing around with the idea of taking a benchmark that does a lot of stuff and keeps the system active, then simply measure how long the battery lasts.  Nothing fancy, we just took 3DMark06 which keeps both the CPU and GPU active more than 50% of the time and logged the minutes the notebook continues to run.

Staying true to my philosophy of being transparent and providing a repeatable process, I want to share the details of how we believe we’ve turned 3DMark06 into an active battery life test. Feel free to provide feedback on our methodology.

One thing to note, all battery life test are very sensitive to screen brightness.  In our labs, we have sophisticated equipment to set the screen brightness to a consistent level (usually 60 nits).  I realize that most people don’t have access to a nit meter, so you may want to either turn your screen brightness down to the lower setting or feel free to test at different brightness levels to see the impact on the battery life scores.

AMD is continuing our efforts with BAPCo with the hopes that the next version of MobileMark will include more robust tests for both idle usage and active usage, but in the meantime, I’d be interested in hearing ideas you may have on how we can better measure the real-world notebook battery life.

Hal Speed is a Marketing Architect at AMD. His postings are his own opinions and may not represent AMD’s positions, strategies or opinions. Links to third party sites are provided for convenience and unless explicitly stated, AMD is not responsible for the contents of such linked sites and no endorsement is implied.