July 22nd, 2009
Will the 'real' Windows 7 testers please stand up?
After two-plus years of development, Windows 7 is almost done. Microsoft officials have been thanking (though not to some testers’ liking — more on that later) the millions who downloaded the various test builds and provided feedback for their help in building the product.
However, Microsoft’s brass isn’t thanking publicly an inner circle of about 40 or so individuals who actually were the “real” Windows 7 testers. This group of hand-picked individuals are the ones on whom the Windows team seemed to count for a lot of the feedback about the various test builds of Windows 7 as it made its way through the development pipeline.
These so-called Test Pilots saw more Windows 7 builds than the rest of the testing community, and they usually saw them earlier. Microsoft officials were especially attuned to their feedback when tweaking the product. In spite of the Windows client team’s repeated insistence that “telemetry, telemetry, telemetry!” means Windows 7 is a customer-driven release, some believe that the Test Pilots and Microsoft’s corporate Technology Adoption Program (TAP) testers were the only external entities that had much of any real impact on the final Windows 7 product.
(Update: For the record, the Windows client team insists that this perception is untrue and that testers’ feedback made a difference in Windows 7. See the Engineering 7 blog posts here, here and here for more on Microsoft’s defense of its testing policies.)
None of the Test Pilots I contacted was willing to be identified by name. Some refused to acknowledge that they were Pilots or knew about the Pilot program, fearing the wrath of the Windows client management. (You can’t blame them; they want to be part of the Windows 8 inner circle, too.)
Microsoft’s decision to create this “elite” group has created considerable animosity among other Windows testers. While the majority of the unwashed masses of public testers don’t assume their suggestions and bug finds will matter much in terms of Windows’ final design, Microsoft’s “technical beta testers” do.
Many of technical beta testers put a lot of time and energy into unearthing and documenting bugs and passing them on to Microsoft. And a number of them were angry that they were provided with only two official Windows 7 test builds from Microsoft over the past three years — and that all they are getting for their work on Windows 7 is a belated “we gave you a chance to get Win 7 at 50 percent off” head-pat and no free copy of the final product.
Up until the start of the Windows 7 development cycle, the technical beta testers were treated as more integral to the creation of Windows. Testers felt like their feedback and guidance mattered and that they were kept in the loop more about what was coming when. Betas of Windows felt more like malleable builds that were subject to user-contributed changes — unlike the case with Windows 7, which a number of testers felt was nearly done by the time it hit its pre-beta milestone builds, let alone the first beta.
In short, the Windows 7 beta testing process was now a lot like Microsoft Office’s, with the first test build seeming more like a final version of the product than something that Microsoft actually expected to be open to tester feedback. (Given the number of former Office managers and execs who now are leading the Windows team, that change isn’t too surprising.)
Was the Windows 7 testing experience any different/better for the Test Pilots?
One Test Pilot with whom I spoke said that while the pilots received new builds of Windows 7 every month or so, they weren’t privy to much information, either.
[More from the Test Pilots] –>
Pages: 1 2
Mary Jo has covered the tech industry for more than 20 years. Don't miss a single post. Subscribe via Email or
RSS. You can also follow Mary Jo on Twitter.
Got a tip? Send Mary Jo your rants, rumors, tips and tattles. For disclosure on Mary Jo's industry affiliations, click here or to see Mary Jo's full profile click here.
