When to Test Manually and When to Automate

There's a balancing act in testing between automation and manual testing.  Over my time at Microsoft I've seen the pendulum swing back and forth between extensive manual testing and almost complete automation.  As I've written before, the best answer lies somewhere in the middle.  The question then becomes how to decide what to automate and what to test manually.  Before answering that question, a quick diversion into the advantages of each model will be useful. 

Manual testing is the most flexible.  Test case development is very cheap.  While skilled professionals will find more, a baseline of testing can be done with very little skill.  Verification of a bug is often instantaneous.  In the hands of a professional tester, a product will give up its bugs quickly.  It takes very little time to try dragging a window in a particular way or entering values into an input box.  This has the additional advantage of making the testing more timely.  There is little delay between code being ready to test and the tests being run.  The difficulty with manual testing is its cost over time.  Each iteration requires human time and humans are quite costly.  The cumulative costs over time can be very, very large.  If the test team is capable of testing version 1.0 in the development time but nothing is automated, it will take the test team all of the 1.0 testing time plus time for the new 2.0 features to release version 2.0.  Version 3.0 will cost 3x as much to test as the first version, and so on.  The cost increases are unsustainable.

On the opposite end of the spectrum is the automated test.  Development of automated tests is expensive.  It requires a skilled programmer some number of hours for each test case.  Verification of the bug can require substantial investment.  The up front costs are high.  The difficulty of development means that there is a measurable lag between code being ready for test and the tests being ready to run.  The advantage comes in the repeated nature of the testing.  With a good test system, running the tests becomes nearly free.

With those advantages and disadvantages in mind, a decision framework becomes obvious.  If testing only needs to happen a small number of times, it should be done manually.  If it needs to be run regularly--daily or even every milestone--it should be automated.  A rule of thumb might be that if there is a need for a test to be run more than twice during a product cycle, it should be automated.  Because of the delay in test development, most features should be tested manually once before writing the test automation for the feature.  This is for two reasons.  First, manual exploratory testing will almost always be more thorough.  The cost of test development ensures this.  Second, it is more timely.  Finding bugs right away while development still has the code in their minds is best.  Do thorough exploratory testing of each feature immediately.  Afterwards, automate the major cases.

This means that some tests will be run up front and never again.  That is acceptable.  If the right automated tests are chosen, they will act as canaries and detect when things go wrong later.  It is also inevitable.  Automating everything is too costly.  The project won't wait for all that testing to be written.  Those who say they automate everything are likely fooling themselves.  There are a lot of cases that they never write and thus are never run.

Classes Should Exhibit High Cohesion

This is part 4 of my ongoing Design Principles series.

When designing the classes in your model, it is important that they each have a specific role to play.  Cohesion means joining together to form a united whole.  Classes and methods should be highly cohesive.  They should have a single function.  Methods on a class should be single function and together the methods on a class should all work toward accomplishing the same purpose.  Adding an add method to a string class is not being cohesive.  Having a "utility" class isn't either.  Classes (or methods) that try to play more than one role are harder to use, harder to test, and harder to change.  The reason is side effects.  If a class does many things, there is a high likelihood that using one part will affect how another part works. 

It is hard to understand the use model of a multifunctional class.  A class with a purpose has an obvious place in a design.  A class without a purpose implies less of its use pattern and is therefore easier to mis-use.  Classes with multiple roles to play also have more complex interfaces.  Each role subtly influences the design of the other roles and may make the use model more complex than necessary.  Simple, self-documenting use models are going to result in the cleanest code with the fewest bugs.

It is important in testing to make sure that one method call on a class doesn't impact other method calls.  If the number of methods on a class gets large, the combinatorial explosion forced on test becomes unmanageable.  Likewise, every time a change is made, all methods on a class must be tested to make sure that they didn't break.  The more methods--and the more disparate the methods--the more burdensome this becomes for test.  That means longer test passes, less new exploratory testing, and a poorer product. 

Most importantly, complex multi-use classes will be difficult to change.  The code underneath will be interconnected and a change that should affect only one role may end up triggering a cascade of changes that impact all of the other roles.  Tradeoffs may be required.  What should be a simple change may mean untangling various pieces of internal state.  It is hard to be comfortable with a change to a complex class.  Unit tests will ease the discomfort, but there is so much that could potentially break that one can never be sure something subtle didn't just change for the worse.

How can a poor design be recognized?  Here are some questions to consider:

• Is the name generic?  If it doesn't imply a single role, the class may be trying to do too much.
• Conversely, can this class be given a simple name?  If naming is difficult, the class is too complex.
• Classes with a large number of methods often are trying to do more than one thing.  Consider breaking them up.
• Classes should have only one reason to change.
• Methods that take a lot of options are generally poorly designed.  If a method checks a state and then branches right at the beginning, it should be more than one method.

DoSomething(enum Type, int Data)
{
    switch(Type)
    {
           type1: do something
           type2: do something else
    }
}

Is better written:

DoSomethingType1(int Data)
{
    do something
}

DoSomethingType2(int Data)
{
    do something else
}

Podcasts I Listen To Regularly

It's been a while since I posted my list of podcasts and my tastes have changed since then.  Here's what I'm listening to on a regular basis right now:

 

Audio:

• This Week In Tech - Tech news round table
• This Week in Media - Mac news round table
• The Dave Ramsey Show - Personal finance.
• Manager Tools - How to be a better manager.

Video:

• Tekzilla - Tech stories.  Long and short format.
• G4's Ninja Warrior - Snips from the Ninja Warrior show.
• Ask a Ninja - Comedy.
• Geek Brief - Daily tech news.

There are a good number of others that I listen to with less regularity.

Is There Value In Code Uniformity?

There seem to be two opposing views of coding standards.  Some think that they should enforce uniformity in code.  Everyone should use K&R braces, leave two lines between functions, and have a space after the if and before the opening parenthesis.  Others (like me) think that there is little value in uniform-looking code.  The value of coding standards is in their ability to stop bugs from getting into the code.  Patterns that are error-prone, like failing to create a new scope { } for the result of an if statement, should be banned.  Others that are odd but not necessarily error-prone should not.  I'm interested in hearing from you, my readers, on your experience here.  Is there more value in having uniform code than I give it credit for?  Does it really make reading code that much easier?  Do bugs really stand out more or is it just format deviation that stands out?

 

2/20 - Ugh, my spam filter stopped every response to this.  Unblocked now.

HD-DVD: 2006 - 2008, R.I.P.

According to reports, Toshiba will soon stop manufacturing HD-DVD equipment.  The writing has been on the wall since Time-Warner's January announcement that it would go exclusively to the BluRay format.  More recently Netflix and WalMart have announced that they will go exclusively to the BluRay format.  It's unfortunate because, in my mind, HD-DVD was the superior format.  It doesn't have all of the profile issues that are currently plaguing BluRay.  Alas, the market has chosen a winner.  The format war will soon be over.  The question is, will consumers notice?  Will they embrace the new format?  Or will they continue to ignore the expensive high-def disc formats.

[2/19/08] Update:  It's official.  Toshiba announced that they will stop development and manufacturing of HD-DVD equipment.

 

Note:  I'm not on the HD-DVD team here at Microsoft and have no inside knowledge.  This is just my take based on public news sources.  Don't take this as confirmation of anything.

Evaluating Your Skill As A Leader

Someone recently characterized for me one way leaders are evaluated.  This certainly isn't the only way and it doesn't catch everything, but it is a good place to start.  The list is succinct and the questions thought-provoking.  Here is the list:

• Results – How is your day job going? 
• Leadership – Who do you make better?
• Strategic Insight – How far out are you thinking?  A few days?  A few months?  Several years?
• Scope – Who is changing because of you?  Your team?  Your group?  The company?  The industry?
• Sphere of Influence – Who is listening to you?

These are good questions to keep in mind when looking to advance one's career into leadership.

Arc Is Out

Over 6 years ago Paul Graham told the world that he was working on a new programming language called Arc.  It would be some derivative of Lisp, but otherwise not much was known about it.  Graham is the author of 2 books on Lisp and a popular series of essays on topics ranging from programming to startups.  As he had been mostly quiet about Arc for such a long period of time, many people have forgotten about it or dismissed that it would ever come to light.  Two weeks ago Paul announced the initial version of Arc and made it public.  I haven't had a chance to look at it yet but I intend to.  If you are interested, you can find information at arclanguage.org.  There seems to be some excitement building around it.  Only time will tell if it can build a user base or if it fades like so many other new languages.