Testing A Daily Build

It is becoming accepted in the industry that teams should produce a build on a daily basis.  Every project at Microsoft does this as do most projects elsewhere.  If you happen to be on a project that does not, I suggest you work to get one implemented soon.  The benefits are great.  After a daily build is produced though, what next?  What do you do with it?  Here is my suggested work flow.  This is for a large project.  If yours is smaller, feel free to collapse some of the items.

1. Deal with build breaks - If anything failed to compile, jump on it right away.  Drag developers out of bed and get it fixed.  Either that or back out the offending checkin (you are using a content management system aren't you?) and build again.
2. Build Authentication Tests (BATs) - The first thing to run against a build are the BATs.  These are test cases that merely ensure that the build is not entirely broken.  These should ensure that all the right files were produced, that the setup works and places files correctly, etc.  Some very basic functionality may be tested here as well.  If the BATs fail, have someone look into the cause immediately and get it fixed.  Do not proceed with any more work on this build until BATs pass.
3. Build Verification Tests (BVTs) - BVTs are a set of tests to verify basic functionality in your build.  These are not comprehensive tests.  Rather, they are limited in scope and time to the things that matter most.  I'd recommend you ensure that these complete within an hour.  If these fail, the build must not be released for further testing.  Developers must be called in and fixes generated quickly.  I've talked about these previously.  It is worth repeating a little here though.  BVT failures are not acceptable.  These are the canaries we take with us into the mine.  If they fall over, it's time to head for the exits.  Only put tests into the BVTs that meet the criteria that you're willing to block a build's release when they fail.
4. Functional Tests - This is most of the rest of your test collateral.  These tests are initiated upon completion of the BVTs.  The functional tests contain all of your automation and any manual tests you deem worthy of being run on a daily basis.  These can take all day to execute if you want.  It is acceptable for functional tests to fail.  Each failure should generate a bug that is tracked.  The point of the functionals is to get a feel for the true state of the product.  Everything you want to work should be tested here.  When most (all?) of your functional tests are passing, you know you have a product ready to release to the world. 

That's it for the daily testing regime.  However, that's not all for testing.  There are other tests you probably want before you are ready to release.  These include:

• Stress Tests - Make sure your code can work under stressful conditions like repeated use, high CPU usage, and low memory.
• Longhaul Tests - Ensure that your code will continue to work over long periods of time.  The exact amount of time depends on the usage model of your tests.
• Customer Acceptance Tests - Make sure the customer is happy with the product.  This is usually a series of manual tests that verify that the usability is acceptable.