Posts tagged with benchmark
Commercial PC Buyers, How Do You Evaluate Client Software Performance?
Posted by Patrick Moorhead in 8:43 AM
(6 votes, average: 3.50 out of 5)
Loading ...Even though the prices for desktops and notebooks continue to decline year after year, acquisition cost still isn’t insignificant. While in most circumstances software and services outweigh acquisition cost, buyers still want to make the best decision to save their small, medium, large business or government IT shop money. This has been amplified by the overall economy which has led to many reduced IT budgets.
Buyers look at many variables in making their client decision (ie brand, reputation, system quality and reliability, post-sales service and support, energy efficiency, managability), one which is software performance. One way purchase evaluators measure the software performance of the potential systems is through benchmark packages aka “benchmarks”. These are software packages that basically measure the software performance then use the results to compare different PCs being considered.
I wanted to poll the “community” of PC purchase evaluators in business and government to see what they use. Sure, we have quantitative information and have face-to-face meetings with key commercial end users, but the “community” never ceases to amaze me with their insight and answers. Please don’t let me down. 
Loading ...Each IP address can vote only once and you only get one choice. I’ll post a real-time summary of the aggregate results – I won’t be identifying individual voters or their choices.
Thanks for the insight and any details on “why” you chose what you chose would be apprecuated in the comments section.
Pat Moorhead is Vice President of Advanced Marketing at AMD. His postings are his own opinions and may not represent AMD’s positions, strategies or opinions. Links to third party sites are provided for convenience and unless explicitly stated, AMD is not responsible for the contents of such linked sites and no endorsement is implied.
MobileMark 2007, 60 nits, One Nit-Picker and You
Posted by Patrick Moorhead in 3:19 PM
As we have already discussed here, here, and here, the current de-facto standard to measure notebook battery life in consumer marketing materials is MobileMark ® 2007 (MMO7). One thing I want to help educate on are the basic facts behind the notebook brightness settings recommended by MM07 and compare that to my home electronics and what settings others are using for their notebook displays.
The facts about MM07 and notebook display brightness
MM07 requires the following (1): “The display brightness should be measured for a white screen while on battery and be set at the lowest possible setting, no lower than 60 nits.” One needs a nit meter and yes, we have a few of those in our labs. :-)
So what is a nit? Wkipedia defines a nit as a “candela per square meter.” Yes, that really helped me too. Think of a “nit” as a unit of measure of brightness.
How does 60 nits compare to brightness of other home electronics?
I pretend to be a technically-capable business, strategy and marketing guy, which according to the engineers, makes me REAL dangerous. You see, 95% is good enough for my trade, but for theirs, its 99.999% and I respect that.
So I wanted to know what 60 nits really related to, as I didn’t know nits from nuts. You should have seen the look on the engineer’s faces when I asked them to borrow their nit meter. Chad, his real name J, showed me how to use the nit meter and instructed me how to use the nit meter and to make sure to have a white background on all devices.
I literally went around my house testing many of my home electronics and tested them without changing any settings outside shutting off “auto adjustment” on the phones which I am told is best practice. I suppose I can call this my family’s “comfort setting” based on the setting my family and I actually use the devices. So there is variation in room setting, light, etc. that dictates this.
Here are the test results:
Net-net, the lowest nit measurement I observed with my “comfort setting” was still 2.58X brighter than the MM07 brightness level of 60 nits on which notebook battery life is measured.
I then wanted to see how nits related to notebook screen brightness. So I tested a few notebooks I had in my office to see what the “max nits” were when the screen was set to the highest setting.
As you can see from the figures above, the MM07 requirement was between 20-30% of the samples notebooks’ max screen brightness.
What notebook display brightness setting do consumers use?
Now, I know that a sample set of one doesn’t represent anything, and I’m far from normal, as my friends frequently tell me, but the results of my home survey were interesting – interesting enough to make me real curious about what others are doing with their screen brightness settings.
So last week I started Tweeting and Facebooking questions to real users on what is the display brightness setting on which they place their notebooks. I received qualitative and quantitative responses. Yes, I know, this isn’t a 17-country, global discrete choice, double blind methodology survey followed by focus groups, BUT there were telling, directional indicators.
Quantitative: I wasn’t expecting anything like the response I got from Neowin respondents. They actually placed a poll on their community website asking the question, “What brightness level do you run your notebook?“ Over 1,100 community members voted in a few days, and according to Shane Pitman, Editor-in-Chief, “Polls require a member account, and to be logged in to said account. Provides accountability, keeps people from voting multiple times.”
The results were overwhelming in that 75% of the Neowin community member respondents kept their notebook display brightness between 61% and 100%. Somehow, I don’t think that fits into the 60 nit range.
Qualitative: These responses were as valuable as the quantitative as they gave insight into “why” they did this. Some comments gave insight into the folks who use their notebooks at very low display settings. I didn’t use their Twitter names to protect the innocent. :-)
Here was the response to my question, improper grammar and all, “Research question: what display brightness do you run your notebook at? (Please RT)”. Here is just a sampling of public tweet responses.
- “On the machines I see, normally I prefer to have them fully backlit with the slider up to around 75% or so.”
- “I run my laptop at maximum brightness unless the battery is really low, or I have an external screen. “
- “Now running MBP 15.4″ 3.1 at 75% brightness with auto adjust turned ; but was at 100% without auto adjust til you asked! “
- “90 – 100% brightness, 90% of the time. “
- “depends entirely on my battery level and surroundings. “
- “Research question: what display brightness do you run your notebook at? (Please RT). I run full bright 95+% of the time. “
- “oh sorry , 100% unless I’m running low on battery, where I bring it down to a minimum, I’m guessing 50%, but thats rare”
- “usually around 60-70%”
- “i don’t use them much, mainly desktop, but i either have brightness at full or minimum: latter at night and if battery low”
- “When running laptop on battery, i set brightness to 25-40%, when plugged in -100%.”
- “100% brightness. Left default setting (and I appreciate it that way… old eyes). “
- “slammed up to full when possible, monitor screens vary though”
- “mine is usually as bright as I can make it. I hate dim screens. “
- “max brightness notwithstanding battery impact. “
- “I lower my screen brightness to the lowest, usable level that’s comfortable. It’s all about the battery life.
“ - “Agreed…I think mine is usually around 30% or so. “
- “it varies by location due to lighting levels. Usually around 30% or 40%. Never > 50% when on battery. “
- “ME: 90 to 100% on AC, about 50 % on battery when I’m watching movies and about 20 to work. “
I also received some really interesting responses related to interpretation, explanations, tools and resources on brightness after asking about 60 nits. Got to love social media.
- “not all that hard. Need 800+ nits to read screens in direct sun, 30-150 for night highway signs: http://bit.ly/cjxJB“
- “60 nits = table in an office with 300 lux illumination: http://bit.ly/FvoKe BTW sRGB calibration target = 80 nits. “
- “oh and here’s a Kodak guide on how to use a digital camera and gray card as an ad-hoc nit estimator: http://bit.ly/1IMLK3“
You be the judge
I titled this blog, “MM07, 60 Nits, One Nit-picker and you” and now I ask you, am I just nitpicking? I ask you to make that decision for yourself. I have described a few things:
- First, MM07 test requires a 60 nit or higher display brightness. To maximize battery life benchmarks, systems are set at the minimum allowable 60 nit level. This setting is thereby integrated into this battery life benchmark that is then used in consumer advertisings.
- Secondly, worst case, the display brightness of many electronics inside my home at my family’s “comfort setting” is 2.58X more than the MM07 nit setting requirement. Based on the small set of notebooks I tested, MM07’s 60 nits equated to around 20-30% of the max notebook brightness. Your mileage will vary, maybe.
- Third, my unscientific querying last week says many classes of users crank their screen beyond 61% and many at max brightness. Yes, I said max. Some self-selected a lower brightness setting to conserve battery life.
Nuff said, you be the judge. Agree, disagree, I would love to hear your thoughts on this.
(1) MobileMark 2007 White Paper, March 2008, Revision 1.0. 4.2.1 “Display brightness”
Pat Moorhead is Vice President of Advanced Marketing at AMD. His postings are his own opinions and may not represent AMD’s positions, strategies or opinions. Links to third party sites are provided for convenience and unless explicitly stated, AMD is not responsible for the contents of such linked sites and no endorsement is implied.
Why Your Notebook Battery Life Never Quite Seems Equal to the Claims
Posted by Patrick Moorhead in 4:52 PM
Do you ever feel like the actual battery life on your notebook never quite equals the information that appears in promotional material? For example, you may see “up to five hours,” but actually get about half that. Well, you aren’t alone. I hear it all the time, and if you do a quick Twitter search on the topic, you’ll see lots of discussion.
I can assure you that no devious plot exists to mislead you. It really comes down to three simple factors.
#1: Measurements are best case: Like a car’s “highway miles per gallon” which gauges the best case (cruising at a sustained speed for an extended period without stop-and-go driving), notebook battery life is typically based on MobileMark® 2007. This benchmark primarily measures battery life while the notebook is doing nothing – not even wirelessly connecting to the Internet. A “city-driving” equivalent of notebook battery life doesn’t exist…yet.
#2: Different strokes for different folks: Notebook users are different; we all use notebooks differently, and therefore will see different battery durations. Some watch HD web videos on YouTube, some may just do email, and some play more games than others. ALL which will mean varying battery life. You can see this data from AMD here that shows the phenomenon. This even shows that battery life under system use can even vary by component manufacturer.
#3: Battery life varies over time: The longer you own your notebook, use it, charge, and recharge, over and over again, the more the battery loses its effectiveness. So theoretically, your longest battery life will be on the first day you crack open the packaging. See all the people selling new batteries for old notebooks? Some even say, battery life is variable with heat.
So what should a consumer like you do?
- Grade battery life on a “curve”, let’s say 60%. If the label says 10 hours, my guess is it’s probably only about 6 hours in real use. (UPDATE: this isn’t always linear, so be very careful with this.)
- Ask your retailer and systems providers to provide the “city miles per gallon” or using the tried and tested cellphone analogy, “talk-time”. They all have web sites and when all else fails, you can ask them over Twitter.
I may have not added back 40% of your battery life, but hopefully you know why you only get 60% of it!
(This blog was originally published at the Technologizer web site. Updated with full blog June 8, 2009.)
Breaking Records with Dragons and Helium in the Las Vegas Desert
Posted by Patrick Moorhead in 3:55 AM
How do I describe what happened in a tent on the edge of the Desert in Las Vegas on the last night of CES? Completely over the top…. I can say that it was a story that included dragons and helium at temperatures near the point where atoms and molecules cease to move. While I wasn’t at the event, I had droves of people come to me within and hours to tell an account of what happened. After researching the night’s activities, I wanted to bring it to you.
“The Experiment” was held on the edge of Las Vegas with the goal to push AMD Phenom TM II X4 processors to the absolute limits1 using the most extreme techniques and methods available. The testing was done on AMD Dragon platform technology complete with dual ATI Radeon TM HD 4870 X2 and motherboards based on the AMD 790FX chipset.
Overclockers Macci, Hardman, Sampsa and SF3D were given 500 liters of liquid helium and an arsenal of motherboards from DFI and Gigabyte. I am an absolute neophyte when it comes to overclocking and have lots of fun with it, but when you add the pros and put great gear in their hands, the results are stunning. This proved quite true on this special night in the Vegas desert.
A world record score of 45,474 on 3D Mark 05 capped off the evening with all four processor cores at 6301MHz, the memory controller running at 3843MHz, and memory at DDR2-1100MHz. A pair of ATI Radeon HD 4870 X2 graphics cards using ATI CrossFireX™ technology were air-cooled at 800/950MHz.
The team was reinstalling drivers and cold booting on what the meter read as -242C but at that point the meters are only an approximation….amazing stability at temperatures on the verge of absolute zero (-273C at which atomic motion ceases). The team even got a run with one core operating at 6.8 GHz.
Remember, almost all of this stuff is certain to void your hardware warranties and could cause serious damage to your PC hardware. We burn through motherboards, processors and graphics cards at an alarming rate doing it. And liquid helium is incredibly dangerous in untrained hands. Only the most experienced, safe and methodical overclockers should every even think of attempting something this extreme.
There is some great initial posts out there, check it out.
English
http://www.xtremesystems.org/forums/showpost.php?p=3570593&postcount=432
Event Thread (English)
http://www.xtremesystems.org/forums/showthread.php?t=211726&page=18
German
Finnish
http://plaza.fi/muropaketti/3dmark05n-maailmanennatys-suomeen
YouTube Video of The Event
The team would like to break that record as they refine their techniques and the community goes beyond. We appreciate the support from the OC community that AMD has had over the years and hope that our Black Edition processors continue to raise the level of fun and competition. I am hearing that Dragon technology is proving to be a compelling platform for overclockers and we hope that the community enjoys it.
1) AMD’s PRODUCT WARRANTY DOES NOT COVER DAMAGES CAUSED BY OVERCLOCKING, EVEN WHEN OVERCLOCKING IS ENABLED VIA AMD SOFTWARE.
Pat Moorhead is Vice President of Advanced Marketing at AMD. His postings are his own opinions and may not represent AMD’s positions, strategies or opinions. Links to third party sites are provided for convenience and unless explicitly stated, AMD is not responsible for the contents of such linked sites and no endorsement is implied.
Web 2.0 – A New Server World Order?
Posted by Patrick Moorhead in 4:55 PM
You may have seen recent blogs from me on the topic of social networks and their growth. I even subjected my Computex blogs to these media. Why do we spend time with this stuff – we don’t sell software! In the end, it’s because trends in our industry are important for AMD, and personally satisfying to me to stay up to date on these. The creative minds in our industry continue to find new ways to use technology and it’s a challenge just saying current. But what’s equally surprising is the speed that the “under 30” crowd adopts these new media/social networks. This new social model is one of the emerging Internet use scenarios that has come to be known as Web 2.0.
I find a great deal of satisfaction in this fast growing market especially in light of some recent announcements that AMD made regarding the AMD Opteron processor’s web performance. AMD recently announced that Quad-Core AMD Opteron™ processor Model 2356 and Quad-Core AMD Opteron processor Model 8356 now hold the top x86 web performance records for both 2P and 4P servers, as measured by the SPECweb®2005 benchmark. This means that the AMD-based server architecture is able to process and manage more data over a period of time than competing solutions and therefore keep the media rich social networks operating at ever faster speeds. As I was writing this blog, I read a related blog from Nigel Dessau on the subject of workloads and balancing loads. We’ve spent a great deal of time and effort to enable and optimize these new networks but we did not stop there. I blogged recently on 2nd generation notebooks and highlighted the new features that we have enabled through our ATI acquisition including our UVD (universal video decoder) which is optimized to consume rich media such as web video.
AMD continues to focus on solutions. Just to be clear, what this means to us is power and performance optimized silicon, platforms optimized with partners and customers to deliver real benefits. These benefits are provided on the server side as a web backbone and on the notebook and desktop as power/performance optimized clients. So AMD is again leading and enabling major transitions in the industry by leveraging our unique capabilities to create, process, serve up and consume this new-fangled thing called Web 2.0.
Pat Moorhead is Vice President of Advanced Marketing at AMD. His postings are his own opinions and may not represent AMD’s positions, strategies or opinions. Links to third party sites are provided for convenience and unless explicitly stated, AMD is not responsible for the contents of such linked sites and no endorsement is implied.
There are Benchmarks, and there are Benchmarks…
Posted by Patrick Moorhead in 10:08 AM
I’m a fan of benchmarks. I think they are very helpful in allowing consumers to make informed purchase decisions about products. But they generally have some flexibility built into them so you can focus on those elements you want. And this means you can use a benchmark to tell a number of stories – which means you can choose to tell the story you want.
For example, take a recent review by AnandTech entitled “Sixteen Cores, Four Sockets” published on June 17, 2008. This article featured Quad-Core AMD Opteron™ processor-based systems. One of the performance evaluations in this article was a SPECjbb2005 benchmark estimate. What is particularly interesting about this article is that the published estimates list the 4 socket server running AMD Opteron processors model 8356 as 25% faster than the competition while running at similar frequencies and 7% faster than the fastest competitive solution. These results vary widely from the official scores posted on the SPEC site. Now you might ask – how can that be? How can you run what is considered to be an industry standard benchmark and get a different set of numbers? That can’t be right!
Taking a closer look at the SPECjbb2005 benchmark helps to unravel this mystery. SPECjbb2005 is a memory-intensive benchmark that is intended to evaluate the performance of servers running typical Java business applications. Its results evaluate the interaction of the CPU, caches, memory hierarchy, JVM (Java Virtual Machine), and JIT (Just-In-Time) compiler. SPECjbb2005 can be configured to run in a variety of ways, resulting in different performance outcomes. Different configuration = different story. For example, you can get different results based on the operating system used, the version of JVM used, the level of optimization of the JVM and JIT, JVM tuning options, and thread allocations.
The SPECjbb2005 scores published by SPEC tend to be achieved using very aggressive software tuning and processor settings. These settings help to achieve a “best possible score” but do not necessarily reflect how a system would be configured in a data center environment to provide the most stable and efficient performance. The scores published in the AnandTech article, according to the author, are more likely to reflect real world configurations with optimizations used consistent over the different processor architectures.
Indeed – if you do a survey around the internet you can find reference to other SPECjbb2005 scores and estimates that reflect a variety of configuration options and the resulting differences in the benchmark scores:
http://blogs.sun.com/bmseer/entry/sun_fire_x4440_best_opteron
Blog featuring SPECjbb2005 results with the 4 socket Sun Fire x4440 running quad-core AMD Opteron processors with Solaris 10 and Sun JVM. Also highlights power consumption of featured systems – reminding us that in today’s economy of escalating energy costs raw performance has less meaning to data centers than performance/watt.
http://techreport.com/articles.x/13176/4
An article by TechReport featuring SPECjbb2005 estimates for 2 socket servers running quad-core processors with Windows Server 2003 x64 edition and the Sun JVM. The author states the goal of this performance evaluation was to test relative performance on equal footing.
Taking a closer look at the official SPECjbb2005 scores and the estimates published in the various articles, you can see how confusing a benchmark can be. This serves as a reminder to us that benchmarks are just an indicator of performance and that a benchmark like SPECjbb2005, which allows for a wide variety of configurations, can produce a wide variety of results. And remember – the story being told is not always the one that best reflects reality….
Pat Moorhead is Vice President of Advanced Marketing at AMD.His postings are his own opinions and may not represent AMD’s positions, strategies or opinions. Links to third party sites are provided for convenience and unless explicitly stated, AMD is not responsible for the contents of such linked sites and no endorsement is implied.
In Praise of Standardized Virtualization Benchmarks
Posted by Patrick Moorhead in 2:49 AM
I’d like to start off my blog today by extending my congratulations to my AMD colleagues on capturing the VMmark benchmark crown. Our CMO Nigel Dessau covers this achievement today in his blog here.
While the best way in my opinion for enterprises to compare hardware is to load their own data in their own datacenters, it is encouraging to see that “benchmarkship” is already entering the world of virtualization. Because doing real-world testing is difficult with virtualization, this is a good first step. And it’s interesting to see how benchmark developers are begining to develop guidelines around how to properly conduct these virtualization benchmarking activities. These developers are essentially telling the industry that they are going to insist that we standardize our testing methodology, so that we are comparing “apples to apples”. Personally, I think this is a great idea and I hope it will help ensure that meaningfull, truthful data is available to those who need it to make their virtualization purchase decisions.
Of course guidelines are meaningless if they aren’t enforced. That isn’t the case here. We’re seeing more examples of how benchmark developers are willing to enforce these guidelines. Adhering to guidelines is something that AMD learned the importance of firsthand in 2007. And we’re not the only ones. Recently the benchmarks for one of our competitor’s solutions had to be withdrawn from the VMmark results page due to “noncompliance”.
I also want to point out that it is possible to follow the guidelines and still not necessarily tell the whole story. For example, a customer recently pointed out to me that our competitor posted some system benchmarks, but somehow omitted any reference to the AMD-based Dell R905, even though benchmarks for this AMD system had been posted on the VMWare’s site for more than a month. Perhaps it was an oversight?
While real-world datacenter comparisons using real business data is the best way to evaluate systems, I beleive that as vendors we need to take the benchmark high road – we need to use benchmarks as one tool to help customers evaluate technology – so they can make wiser choices. We need to fairly and honestly benchmark systems and report all the pertinent data. Trust and credibility go hand-in-hand, and if we do not keep the process fair and open, then the hard work of many people within my company, at our customers and partners and even within our competitor is at risk. As a demonstration of telling the whole story, let me point you to the “complete” results of the VMWare benchmarks.
Pat Moorhead is Vice President of Advanced Marketing at AMD. His postings are his own opinions and may not represent AMD’s positions, strategies or opinions. Links to third party sites are provided for convenience and unless explicitly stated, AMD is not responsible for the contents of such linked sites and no endorsement is implied.