March 19th, 2007
Inside the belly of the benchmark debate
David Berlind says Intel's benchmarks don't add up. George Ou disagrees. The debate on technology benchmarks–whether it's hardware related like Intel vs. AMD or software–is nearly endless.
This Talkback sums up the state of affairs with benchmarks:
When I look at almost any product I can think of, I would not trust the manufactures claims. Instead, I look for independent reviews. There are a number of sites that do real world usage benchmarks (Toms Hardware) and can provide a lot of information on how hardware performs on tasks that you, George, or I might do. That to me means vastly more than some sideways applied counting contest called a benchmark.
Anyhow, yes I agree, manufactures marketing spin in their claims are outlandish, that is why they should be ignored…
The big problem: Technology buyers don't have much beyond manufacturer claims. Benchmarking standards don't exist. Independent third parties are disappearing as tech trade magazines cut back on labs. Reviews are helpful, but benchmarks are still all over the map.
I'm not sure what the answers are to the benchmarking issue, but the problem needs to be fixed across the entire sector. Here's some required reading as we ponder what exactly that fix should be.
- AMD’s no angel, but Intel’s public usage of benchmark data is feloniously misleading.
- Gallery of the data in question.
- Is AMD being hypocritical about benchmark ethics?
- Initial Core benchmarks look dire for AMD.
- Video: Henri Richard comes out swinging at Intel during AMD press conference.
- Did Intel rig test or out engineer AMD/ATI?
- HP and IBM duel over blade thermal benchmarks.
Larry Dignan is Editor in Chief of ZDNet and Editorial Director of ZDNet sister site TechRepublic. See his full profile and disclosure of his industry affiliations.
For daily updates, follow Larry on Twitter.
Subscribe to Between the Lines via Email alerts or RSS.
