Tuesday, February 26, 2008

When to Test Manually and When to Automate

There's a balancing act in testing between automation and manual testing.  Over my time at Microsoft I've seen the pendulum swing back and forth between extensive manual testing and almost complete automation.  As I've written before, the best answer lies somewhere in the middle.  The question then becomes how to decide what to automate and what to test manually.  Before answering that question, a quick diversion into the advantages of each model will be useful. 

Manual testing is the most flexible.  Test case development is very cheap.  While skilled professionals will find more, a baseline of testing can be done with very little skill.  Verification of a bug is often instantaneous.  In the hands of a professional tester, a product will give up its bugs quickly.  It takes very little time to try dragging a window in a particular way or entering values into an input box.  This has the additional advantage of making the testing more timely.  There is little delay between code being ready to test and the tests being run.  The difficulty with manual testing is its cost over time.  Each iteration requires human time and humans are quite costly.  The cumulative costs over time can be very, very large.  If the test team is capable of testing version 1.0 in the development time but nothing is automated, it will take the test team all of the 1.0 testing time plus time for the new 2.0 features to release version 2.0.  Version 3.0 will cost 3x as much to test as the first version, and so on.  The cost increases are unsustainable.

On the opposite end of the spectrum is the automated test.  Development of automated tests is expensive.  It requires a skilled programmer some number of hours for each test case.  Verification of the bug can require substantial investment.  The up front costs are high.  The difficulty of development means that there is a measurable lag between code being ready for test and the tests being ready to run.  The advantage comes in the repeated nature of the testing.  With a good test system, running the tests becomes nearly free.

With those advantages and disadvantages in mind, a decision framework becomes obvious.  If testing only needs to happen a small number of times, it should be done manually.  If it needs to be run regularly--daily or even every milestone--it should be automated.  A rule of thumb might be that if there is a need for a test to be run more than twice during a product cycle, it should be automated.  Because of the delay in test development, most features should be tested manually once before writing the test automation for the feature.  This is for two reasons.  First, manual exploratory testing will almost always be more thorough.  The cost of test development ensures this.  Second, it is more timely.  Finding bugs right away while development still has the code in their minds is best.  Do thorough exploratory testing of each feature immediately.  Afterwards, automate the major cases.

This means that some tests will be run up front and never again.  That is acceptable.  If the right automated tests are chosen, they will act as canaries and detect when things go wrong later.  It is also inevitable.  Automating everything is too costly.  The project won't wait for all that testing to be written.  Those who say they automate everything are likely fooling themselves.  There are a lot of cases that they never write and thus are never run.

12 comments:

  1. Hi Steve,
    Manual or automation is tricky decision, if not taken carefully chances are high that you will fail in both. I suggest having mix of both as per project requirements. Manual testing for the modules that don't require regular changes and automation for frequently updating modules.

    ReplyDelete
  2. Hi,
    about "Finding bugs right away while development still has the code in their minds is best.", isn't that the aim of TDD?
    What do you think about this? It seems that you place the testing phase after the developing one, not considering the "mixed" way. I will  appreciate your guru :) response.
    Have a good day

    ReplyDelete
  3. This is so nice overview that I would like to suggest some extension from my experience:
    There is one benefit of automated tests I don't see mentioned - the regression bugs will be detected immediately by them while "development still has the code in their minds" - and so they know which change has caused the regression.
    The benefit “Manual testing is the most flexible” also means that manual tester could change validation approach as they see something “suspicious” even during regression tests (warnings in log file, blinking and lagging UI items, etc., etc.). Automated will only validate what they are programmed to validate.
    This is why when deciding what to automate I also take into account risk (probability and impact:fixing cost) of regression defects and how “smart” the validation should be.

    ReplyDelete
  4. Interesting post. A few questions;
    1. When using a mixture of both, do you have separate resource for each? E.g. a group of testers doing the manual testing and a group of people owning the automation?
    2. Does the effort involved in reviewing the automated tests ever become too high? E.g. in the time spent that 1 person could have been adding more value by running exploratory tests.
    3. For anyone - Do people who dislike scripted tests still use automation?
    I've always wanted to use automated testing but in the past we've never quite got it right - either it's become too much of an overhead or we lacked the resource to keep it going. I'm still searching for the right opportunity to use it.

    ReplyDelete
  5. Lots of good questions.  A few answers.
    @Vijay, even for modules that don't change often, you still have to test them every release because the environment around them will have changed.  Unless they are the lowest object on the stack, any changes below them means you need to retest.
    @crazy - TDD is only a partial solution.  It tests that each function or perhaps module works as expected but not that the whole system works.  Many bugs are in the seams between modules.
    @Simon:
    1.  I've seen it done both ways.  There are advantages to both sides.  I'm partial to separate resources but the danger is that they won't coordinate.
    2.  The cost of reviewing automated tests can become very high.  However, I don't know that the time spent would be better used running exploratory tests.  That would only be true if the automated tests were finding no issues.  If that's the case, automating the review can be done easily.
    To your last question, make sure you aren't automating too early.  If the costs of automation are too high, there is likely too much churn in the project still.  Either write tests that insulate you from the churn or wait until it has died down some.

    ReplyDelete
  6. I keep wondering why manual vs. automation is viewed as opposites and that tests can be performed either manually or can be automated.
    Why not blend the two as part of the same test?  Automation that assists testers with specific testing tasks while leaving other tasks that are part of the same test can often bring more value at a lower cost than trying to automate all or most parts of a test.
    Developers use many tools to assist them in writing, managing, compiling, and building code.  There's no reason that testing tools can't be created to help testers do their jobs better instead of trying to replace them.
    Also, I can think of many cases in which disposable automation has helped me test something better or quicker than I could have done without the support of the automation.  Automation does not always need to be run many times to become cost effective.
    Ben Simo
    http://www.QuestioningSoftware.com

    ReplyDelete
  7. >>>The question then becomes how to decide what to automate and what to test manually.
    When faced with this "when" and "what" about automation, I use following to help me out.
    1. Tell me about the test - Is it ready, meaning documented? in what format? Studying and Analyzing the test can give lots of insight into the approach that I could take.
    2. When I am getting into this kind of questioning? Is it before I design/document the test or at the time of test execution?
    (Assuming that design and execution of tests are separate activities)
    3. What kind of test is this? What kinds of oracles does it need?
    Type of test -- I can think of a confirmatory test say data validation (this field takes only a 6 digit number (0-9)) or a business rule validation or a file comparison and so on ...
    4. Does this require human to watch and observe when the test is being run? Is human intervention mandatory or optional?
    5. Finally - I would ask who is best *agent* to *execute* this test ...
    The notion of "best" would cover aspects like reliability, cost effectiveness, effort, time, chance of human errors and so on ...
    There are some tasks that machine can do better and at some others human excels. That is key point for me determining what I should automate and what I should test manually....
    Shrini Kulkarni
    http://shrinik.blogspot.com

    ReplyDelete
  8. Ben Simo said : I keep wondering why manual vs. automation is viewed as opposites and that tests can be performed either manually or can be automated.
    Let us classify tests in following categories:
    A- Tests that ONLY Automation can execute
    B- Tests that ONLY human can execute
    C- Tests that BOTH human and automation execute with equal efficiency and effectiveness. Either can do the job.
    D- Tests that BOTH human and automation can execute but there are situations where one *agent* scores over the other. Context is key.
    E- Tests that human CAN execute but the accuracy, precision and result tend to diminish over repeated cycles - human fatigue causes "oversight" errors
    F - Tests that automation CAN execute but at higher cost (human execution is cheaper and quicker option)
    G-  ??
    So take a test and see which category it fits. That will give you clues about which agent is best suited to do the job ...
    Shrini

    ReplyDelete
  9. >>>With a good test system, running the tests becomes nearly free.
    What do you mean by a good test system? running tests become free under what circumstances? What parameters make this happen and what parameters threaten this "free" service? Will the test execution be free (assuing that you can good test system) when the system is buggy, and undergoing lots of changes?
    Also you would agree that by automation, you would also loose information/possibly bugs that might come from skilled human testing. Since with automation, you do not invest time
    manual testing - hence free automated testing comes at a hidden risk .. how do you handle that?
    Shrini

    ReplyDelete
  10. Charles Sterling on Rosario Specifications for some of the TFS Features and Radio TFS!. Jason Barile...

    ReplyDelete
  11. Testing started simply. Developers would run their code after they wrote it to make sure it worked. When

    ReplyDelete
  12. Hi Steve,
    If we go for automation tool like selenium it could be less cost than manual testing right?

    ReplyDelete