January 15th, 2008
Beware of flawed CPU efficiency study
Update 2/22/2008 - I originally used the word “rigged” to describe Neal Nelson’s study. My reasoning for using the word “rigged” was due to the fact that the test platforms used in Nelson’s study painted an inaccurate picture. Nelson’s study omitted two generations of Intel products while including pre-shippings products from AMD. Since I cannot know for a fact whether the test subject selection was intentional or merely coincidental, I changed the word “rigged” to “flawed”. Other than this change, I stand by my analysis here.
What if we held a football game involving the Patriots and any other NFL team where we set up a Patriots handicap that prohibited Tom Brady and the rest of his starting lineup from playing? What if the result was a loss for the Patriots and we splash the headline across the sports newswire that the Patriots just lost a football game? Would you think this was ethical behavior? Well that’s precisely what happened yesterday when the Neal Nelson report titled “AMD beats Intel in quad-core server power efficiency” spread across the newswire and got repeated as fact.
This is a classic case where the measurements are most likely accurate, but what’s being measured isn’tNeal Nelson and Associates is a consulting firm that has made it a habit to put out these handicapped reports on processor efficiency. Last year they excluded Intel’s quad-core lineup when AMD didn’t have quad-core processors and declared AMD the winner and got lots of news coverage, now they’re comparing Intel chips released in Q4 2006 to AMD technology that may not be available to the general public until Q2-2008 and the press seems to be falling for it all over again.
Nelson compared AMD’s Opteron 2350 2.0 GHz quad-core processor (may not ship again until Q2-2008 when the TLB bug hopefully gets fixed) to Intel’s older 65nm “Clovertown” E5335 and E5345 processor which were released in Q4 2006. These weren’t even the newest 65nm G Stepping Clovertown processors from mid-2007 with lower power consumption; these were the older stepping released in 2006. But Intel launched their latest 45nm “Harpertown” processors in November of 2007 and these chips were excluded from this “study” on AMD versus Intel energy efficiency. This is a classic case where the measurements are most likely accurate, but what’s being measured isn’t. This is a critical omission because the 45nm chips from Intel made significant improvements in performance and energy efficiency which has a double impact on performance per watt.
Nelson basically took a product from AMD that hasn’t even sorted out the bugs yet and can’t be purchased yet, then compared it to Intel’s 2006 technology while excluding two newer generations of Intel technology that are available in quantity, and he declares AMD the “winner” on energy efficiency. Then in an ultimate twist of irony, Nelson has the gall to question the methodology of the latest SPEC power efficiency standard SPECpower_ssj2008 when his own tests are outright deceptive. But in reality, SPEC doesn’t go out and declare winners or losers for cheap headlines or overstate the importance of their data; they merely present data with full vendor disclosures and provide valuable data points to the public.
When I did my in-depth review of SPECpower_ssj2008, I tempered the results for AMD despite the fact that the early SPECpower_ssj2008 results showed complete domination by Intel over AMD. I stated that the results would have been more competitive for AMD (at least at comparable clock speeds) if a web server version of SPECpower was used and when AMD quad-core Opteron gets its bugs sorted out. I still stand by that assessment based on the fact that AMD does well on a clock-for-clock basis when looking at SPECweb_2005 performance. However, Intel still commands the clock speed advantage which makes them the performance leader but at least AMD can be competitive on web serving duties at the lower clock speeds if they can fix their bug and launch their quad-core parts.
So who should the IT manager believe when it comes to performance per watt? Ideally you run your own tests on your own applications and draw your own conclusions but that may not be an option for everyone. If running your own tests isn’t feasible, I would recommend finding publicly acknowledged reputable benchmarks like those from SPEC or TPC and try to find the benchmark that most closely resembles your workload. While that isn’t perfect, it’s the closest thing to commissioning your own tests. But what you should not do is rely on consulting firms that have of a track record for fixing the game.
George Ou is Technical Director of ZDNet. See his full profile and disclosure of his industry affiliations.
