Thursday, November 30, 2006

What To Unit Test

Several months back I wrote about unit testing.  Following that I received a question from a reader about how to actually carry out writing unit tests.  What should be tested?  How much is enough?  There is no single answer to these questions but I can give guidance.  The actual answer depends on the specifics of what you are trying to test and how much time you have to devote to unit testing.  What follows is a list of the items you should test, in the order you should test them.  If you only have limited time, start at the top of the list and work down until you run out of time.

In each case below, you should begin by writing tests for the positive cases and move on to the negative ones.  By positive cases I mean those things you expect a user to do.  This is the way the feature will be used if it is being used properly.  Call each function with all the equivalence classes of data that they would be expected to see in normal circumstances.  Ensure that the behavior you observe is what you expect.  This means actually verifying that the right behavior took place.  If short on time, you can often use the interface to verify itself.  Calling a Get after a Set to verify that the value was set correctly is a good example of this.  Even better is verifying through some alternative mechanism that the right behavior took place.  This could be looking at internal structures, observing behavior, querying a database, etc.

Negative cases might also be called bad data.  These are the inputs you don't expect.  This includes out of range values, invalid objects, null pointers, improperly initialized structures, etc.  Once upon a time we could ignore these and consider the result of these inputs to be undefined.  In the days of networked computers and the security implications now present in many flaws, they cannot be treated lightly.  It is important to test for these and ensure that the proper behavior is observed.  That behavior could be to ignore the data, return an error, raise and exception, or even crash safely.  The important thing is to understand what the behavior should be and to verify that it indeed takes place.

The first and most important thing to hit is the point where your customers will interact with your code.  This could be an API or a UI* or a web service.  This is the highest level of abstraction and perhaps not the easiest or even the best place for unit tests, but if you only have a little time, this is where you'll get the most bang for your buck. 

The next place to go testing is at the internal interface level.  This means testing the places where the internal objects/functional blocks interact.  If you are programming OO, this means testing all of the public (or friend) interfaces on all of your objects.

Finally, test unit test all of your internal utility functions and private object methods.  Crack the abstraction layer on your objects and probe their internal functionality.

If you do all of this, you'll have a pretty good idea that your feature is functioning as desired and will have warning when something breaks.  Remember to run all of the (relevant) tests before *every* checkin.  Also, be vigilant.  Any failures at all should be fixed before checking in.  It is not okay to ignore a failure.  If it is a false failure, fix the test.  If it is a valid one, fix the code.


* You probably cannot unit test the API but if you properly abstract the actual UI, you can write unit tests for the functionality of each UI element.

Friday, November 10, 2006

Bug Taxonomy

When you get toward the end of a product, you have to make the tough calls about what gets fixed and what doesn't.  Most bugs fall into one of two obvious buckets:

  • Must Fix -  Something is really bad here.  We cannot ship the product without fixing it.
  • Won't Fix - The problem is minor, hard to get to, etc.  It won't be fixed for this release.

However, there are a few bugs which don't clearly fit these buckets.  There is no clear call to make on them.  My group has come up with some fun names for some of these bugs.  Among the taxonomy are:

  • Yetis - These are bugs that you'd really like to fix but will be really difficult to find a fix for.  If you find a fix, you'll definitely take it but you won't hold the product until you find one.  Issues in this classification include areas where reproduction is difficult.  Stress crashes, random lockups, and inconsistent behavior all fit into this category.  We thought of using the term Sasquatch but that is too many characters.
  • Limpets - These would be more accurately called Remoras, but why we don't is a long story.  These are bugs that we'd really like to fix but it isn't worth the risk to take a change to a particular binary or even the whole product just for that fix.  Taking a change to something already considered done requires another build, test pass, etc. 
  • Sharks - Remora are also known as suckerfish.  They are known for attaching themselves to the side of a shark.  The idea here is that, while we wouldn't crack open a binary for a limpet bug, if there is a large fix which must go in, we'd like to take these other bugs with it.  Once you have started changing things and have to redo your test pass, you might as well include those nice-to-have issues too.

What nomenclature do you use to describe bugs?

Thursday, November 9, 2006

Vista Goes Gold!

Windows Vista--the product I've been working on for much of the last 5 years--has finally shipped.  Hopefully that will give me some more time to actually blog here.  It has been a long haul and a lot of work--especially over the past few months--but it is now finished.  I'm pretty proud of the final product.  I have been running it on my main machines for a few months now and it has been quite solid. 

A few months back I had a post decribing what the term RC1 meant.  At that point, I also linked to an article showing the times from RC1 to RTM of the various Microsoft products.  Windows 2000 took 166 days.  Windows XP took 53 days.  By my calculations, Vista has taken 68 days.  A little longer than XP but in the same ballpark.

Now there is nothing left to do but wait.  Vista launches to the general public on January 30th.  Office 2007 will launch on that same day.