Thursday, October 2, 2008

James Whittaker on Why MS Software "Sucks" Despite Our Testing

A friend turned me on to this post by James Whittaker.  I didn't know he had a blog so now I'm excited to read it.  He has a lot of really interesting things to say on testing so I encourage you to read his blog (now linked on the left) if you are intrigued by testing.

Microsoft prides itself on the advanced state of its testing operations.  This leads to the inevitable question, "If Microsoft is so good at testing, why does your software suck?"  James Whittaker was once a person who asked this question and now that he's at Microsoft he is in a good position to try to answer it.

James gives basically three reasons:

  • Microsoft's software is really complex.  Windows, Exchange, Office, etc. are really, really big projects.
  • Microsoft's software is used by a whole lot of people.  Eric Raymond once made the comment that all bugs are shallow if you have enough eyes.  This applies to closed source software just as much as open source.  Within the first few days of a release of Microsoft software, millions of people are using them.  Windows has an install base in the hundreds of millions.  With so many people looking at it, all bugs are likely to be hit by someone.
  • Microsoft testers are not involved early enough in the process.  This varies throughout the company, but there is still a lot of room for improvement.


  1. Software problems go far beyond what can be found through a test.
    When the specification has a problem at test will only tell you that the program works perfectly according to the specification without crashing.  The test will not tell you that the application is not doing what people what it to do.
    How do you give them what they need instead of what they want?

  2. Hmm, maybe somebody should apply the Five Why's from the later post?