Friday, October 29, 2004

Approaches To Unit Testing

    I recently was involved in a discussion about unit testing.  I'll simplify the issues.  There are many more aguments each way and the area is more complex than presented here.  I will lay out the supposed advantages and disadvantages of side of the issue.  
    Unit tesing has been made quite popular lately with the advent of XP (eXtreme Programming).  The idea is fairly simple.  Have developers write tests to verify that their code works as intended.  It is after this point, that views diverge.  Well, some diverge before that and think that developers shouldn't be bothered to write tests but that, as they say, is a topic for another day.  The XP community seems to think that unit testing should be done at a very granular level.  Each individual function or object should be tested.  Others think that the unit tests should be more wholistic.  
    Granular unit testing is often synonymous with unit testing.  In it, developers test their code directly, trying each input and failing if the expected output isn't returned.  Often scaffolding is used to instantiate the object or call the function apart from the surrounding code.  Mock objects might also be used to insulate the real code from its reliance upon systems below, above, or beside it.  The advantage of this technique is that it is fast and thorough.  Each function, method, or class can be tested fully.  Each piece of code, even those difficult or impossible to reach from the public interfaces, can be tested.  The disadvantage is that testing each piece in isolation doesn't test the system as a whole.  It doesn't test the interaction between the code that will be interacting in the real world.  It is also not useful to test the shipping binary.  As each object is tested by standalone code, you don't get to see how the whole system really works.
    The other form of unit testing is testing only from exposed interfaces.  This is sometimes known as functional testing.  The only item being tested is the shipping binary and the only entry points used are those available on that shipping binary.  The advantages of this type of testing are that it tests the whole system.  It tests the interaction between each part of the system as it will be used in the real world.  These tests are also easily utilized by the test team to run as part of their testing suite.  The disadvantage is that it can be hard or even impossible to test many of the system internals.  Sometimes a simple interface can have a lot of code hidden behind it.  It also requires a greater amount of code to be written before the testing can even begin.
    In this particular discussion that I had, the advocacy was for the second type of testing only.  I advocated a blended approach thinking that would cover all bases.  It was argued that the granular form of testing tests at too low a level--that most bugs are found in the interaction between components and not in the functions themselves.  It therefore made little sense to even write them.    
    What do you think?  Do isolated unit tests have much return on investment or are they better off left undone?  Have you had any experience with either of these approaches?  If so, did it work out well or poorly?