Thursday, October 25, 2007

Testing A Daily Build

It is becoming accepted in the industry that teams should produce a build on a daily basis.  Every project at Microsoft does this as do most projects elsewhere.  If you happen to be on a project that does not, I suggest you work to get one implemented soon.  The benefits are great.  After a daily build is produced though, what next?  What do you do with it?  Here is my suggested work flow.  This is for a large project.  If yours is smaller, feel free to collapse some of the items.

  1. Deal with build breaks - If anything failed to compile, jump on it right away.  Drag developers out of bed and get it fixed.  Either that or back out the offending checkin (you are using a content management system aren't you?) and build again.
  2. Build Authentication Tests (BATs) - The first thing to run against a build are the BATs.  These are test cases that merely ensure that the build is not entirely broken.  These should ensure that all the right files were produced, that the setup works and places files correctly, etc.  Some very basic functionality may be tested here as well.  If the BATs fail, have someone look into the cause immediately and get it fixed.  Do not proceed with any more work on this build until BATs pass.
  3. Build Verification Tests (BVTs) - BVTs are a set of tests to verify basic functionality in your build.  These are not comprehensive tests.  Rather, they are limited in scope and time to the things that matter most.  I'd recommend you ensure that these complete within an hour.  If these fail, the build must not be released for further testing.  Developers must be called in and fixes generated quickly.  I've talked about these previously.  It is worth repeating a little here though.  BVT failures are not acceptable.  These are the canaries we take with us into the mine.  If they fall over, it's time to head for the exits.  Only put tests into the BVTs that meet the criteria that you're willing to block a build's release when they fail.
  4. Functional Tests - This is most of the rest of your test collateral.  These tests are initiated upon completion of the BVTs.  The functional tests contain all of your automation and any manual tests you deem worthy of being run on a daily basis.  These can take all day to execute if you want.  It is acceptable for functional tests to fail.  Each failure should generate a bug that is tracked.  The point of the functionals is to get a feel for the true state of the product.  Everything you want to work should be tested here.  When most (all?) of your functional tests are passing, you know you have a product ready to release to the world. 

That's it for the daily testing regime.  However, that's not all for testing.  There are other tests you probably want before you are ready to release.  These include:

  • Stress Tests - Make sure your code can work under stressful conditions like repeated use, high CPU usage, and low memory.
  • Longhaul Tests - Ensure that your code will continue to work over long periods of time.  The exact amount of time depends on the usage model of your tests.
  • Customer Acceptance Tests - Make sure the customer is happy with the product.  This is usually a series of manual tests that verify that the usability is acceptable.


  1. PingBack from

  2. Steve,
    It appears you are running a very common MSFT daily build / test plan (I worked there 3 years).   So I am going to challenge you and ask why not more frequent?
    Continuous Integration, run your BATs and BVTs at each checkin, or better before developers checkin...

  3. We do often spin a build for each checkin (or batch of checkins), but there are official builds we expect people to install and play with and these are what I'm talking about here.  You make a very good point that building (and testing) on a continuous basis is easily possible and very valuable.

  4. Jeff Beehler on Check out the new Tester Center on MSDN and VS2008 Beta2 VPCs expiring prematurely. Martin...

  5. Max - For what it's worth, there is no difference between a daily build and a continuous rolling build when your product really does take 24 hours to build.