Thursday, June 15, 2006

Prescriptive Advice For Successful Unit Testing

   At the beginning of the Vista (then Longhorn) project our team decided that we would implement unit tests.  This was the first attempt in our locale to try to use them.  We had some successes and some failures.  Out of that I have learned several things.  This is an attempt to codify what I have learned and to try to set out a prescription for what I feel it would take to leverage them fully.  What follows are my recommended practices for implementing unit tests:

  1. Unit tests must be written by the developers of the code.  Having a separate test team implementing them doesn't work as well.  First, they take longer to write.  The person developing the code knows how it is supposed to operate and can write tests for it quickly.  Anyone else has to spend time learning how it is intended to be used from the minimal (at best) documentation or conversations with the developer.  Having the developer write the unit tests acts as a form of documentation for everyone to follow.  Second, the tests take a lot longer to come online.  They are available days, weeks, or even months after the code is checked into the product. 

  2. Unit tests must be written and passing before the code is checked in.  Writing unit tests should be considered part of the development process.  Nothing is code complete unless there are passing unit tests for it.  Checking in code without passing unit tests should be treated like checking in a build break.  That is, it is unacceptable.

  3. The unit tests must never be allowed to fail.  Just recently I saw a bug in our database about a unit test that was failing.  The bug was filed several months ago.  This just cannot be allowed to happen if you are going to get the full value out of the unit tests.  Unit tests should act as canaries.  They are the first things to fall over when something goes wrong.  If they ever break, they must be fixed immidiately.  

  4. Before any checkin, existing unit tests must be run and made to pass 100%.  This is a corrolary to points 2 and 3 but I want to make it explicit.  Just as you would get a buddy build, a code review, and a smoke test before every checkin, you must also pass the unit tests.

  5. Unit tests must be granular and comprehensive.  A test case which plays back some media is not really a unit test for a media pipeline and it certainly isn't sufficient.  At a minimum, each external interface should be tested for all expected values.  An even better system of unit tests would verify internal interfaces directly as well.  A rule of thumb is that unit tests should achieve at least 60% code coverage before the first checkin is made.

  6. Standardize a mechanism for storing, building, and running unit tests.  Don't leave it up to each individual or small team.  There should be a standard harness for the tests to be written in.  There should be a convention followed by all for where to check them into the source tree.  Unit tests must be built regularly.  In my opinion, the unit tests should be written in the same harness used by the test team for their tests.  The unit tests should be checked into the build right alongside the code that they are testing and should be built with each build.  A build break in a unit test is just as bad as a build break in the shipping code.

  7. The unit test framework must be lightweight.  If the framework is to be the same one the test team uses (highly recommended), it must be one that can be run easily.  If running unit tests requires anything more than copying some files and running an executable, it is too heavy.  Expecting developers to install a whole test framework to run their tests is a prescription for disaster.

  8. Run unit tests as part of the daily testing process.  If the tests use the same harness, they can be leveraged as part of the daily tests.  The tests for external interfaces can be especially useful in assessing the quality of the product.

Have any other best practices for unit testing?  Let me know.


  1. Re: developers writing their own unit tests...

    Isn't firsthand knowledge of the code a two-edged sword?

    I can see benefits of test-driven development. I wholeheartedly agree that developers should write tests and verify that their code works before checkin.

    But suppose a developer misinterprets a spec, or has a blind spot for a particular failure scenario... wouldn't bugs slip through the same blind spot present in both the unit tests and the code?

  2.   You make a good point.  Developers could have a blind spot and miss something.  That is why unit tests are not sufficient.  I didn't expound on it a lot in this post but unit tests are only one aspect of ensuring quality.  Other tests need to be written and run as well.

      That said, I don't know that this is really a big problem.  Sure, a developer could miss something but then so could a test developer trying to write the same code.  I'm not too worried about the handful of things that might be missed.  Other parts of the release cycle (code reviews, integration testing, exploratory testing, etc.) should catch any bugs that slip through.

  3. Unit tests are for the developer.  They in no way replace any other testing.  They augment the development/testing process.

    Also, in the great book <a href="">Refactoring</a> Martin Fowler suggests that all bugs should be reproduced in unit tests prior to fixing.  This way, you can see the problem and see the fix working.

  4. Steve,

    I remember talking to you about unit tests a few years ago.  At the time you were concerned about "testing the tests" and where does it all end.  Would you mind explaining what has caused this shift in direction?  I'd like to hear the story behind it.

  5. Several months back I wrote about unit testing. Following that I received a question from a reader about