Tuesday, January 16, 2007

Hiring Great Testers - Tester Roles

In my mind, there are basically three roles on a test team.  These three roles are: developers, scripters, and those who execute the test cases (runtime testers).  In reality there is a spectrum of capabilities on any team but I think most roles will be closely related to one of these three roles.

Test Developers are the heart of a modern test team.  There was a day when you could get away with hiring a few people to just use the product and call that a test team.  This is no longer the case.  Products are becoming more complex.  The lifespan of products is increasing.  More products are being created for developers instead of end users.  These have no UI to interact with so simple exploratory testing is insufficient.  To test complex products, especially over an extended lifespan, the only viable solution is test automation.  When the product is an API instead of a user interface, testing it requires programming. 

Test developers are programmers who happen to work on a test team.  It is their job to write software which executes other software and verifies the results.  The test developers need to be at least as skilled as the developers of the product.  They need to know whatever language their product is written in.  Their job is often as difficult or perhaps even more difficult than writing the product.  Sometimes the job can be done in a simple fashion where each function is just called with various parameters and a simple method of pass/fail is monitored.  This could be a success result or some notification from the program.  A better test developer will not rely on a program to tell him if the right things happened.  Instead, he will monitor the actual behavior.  For a simple function this might just be the return result but more often than not, it requires monitoring the output of the program and making a complex decision based on it.  Test developers are also responsible for creating the test harnesses and systems for their teams.

Scripters are those who have some programming ability but are not as skilled as a test developer.  They will usually know a programming language but on that is less complex than the language the product is written in.  Scripters may know Visual Basic, Perl, Javascript, or even C#.  More important than which language they know, what distinguishes them from a test developer is that their understanding of programming is less advanced.  They will understand the language syntax but their understanding of algorithms and data structures is likely less substantial. 

Scripters will often play a role where they spend a lot of time living the product.  They will use their programming skills to script the product (if it has such a feature) or to utilize tools to drive the product.  The tools could be UI toolkits like Visual Test or something written by test developers.  Because scripters have an understanding of programming they will be able to have a deep understanding of the product.  They will be able to quickly determine the root cause of a failure and see relationships between seemingly unrelated bugs.  Scripters play an important role on any team.  They often handle the bulk of the test triage efforts and closely monitor the needs of the users of the product.

In some teams the responsibility for setting up machines, installing builds, and executing test cases falls to a group I'll call Runtime Testers.  This role involves no programming.  Runtime testers usually don't have any significant programming knowledge and if they do, they don't often get a chance to utilize it.  They play a fundamental role, however.  Because they are the ones running all of the tests, they often have the best knowledge of the state of the product.  If you want to know what really does and does not work, ask one an execution specialist. 

Sometimes this role is relegated to merely clicking buttons.  They run an automated test and log some errors.  This is a waste.  They should be tasked with understanding the product and playing with it.  As I've stated many times in the past, there is a critical need for manual testing on most project.  These are the people best equipped to do that.  To get maximum value out of runtime testers, they should be encouraged to live the product.  To become experts in the functionality the way a customer might use it.  In doing so they can become advocates for the customer and find the usability issues and corner cases that automation cannnot easily find.

Each of these three roles is critical on a test team.  If you build a team with only one or two of the roles, you'll be missing something fundamental.  Each project will require a different mix of people.  Some need more test developers, others more runtime testers.  The importance of these roles comes into play not only when you task your team but also when you hire.  If someone is to be a runtime tester, hiring a CS grad may not be the best idea.  They will likely be bored and quickly move on.  Hiring a self-educated computer geek may be a better fit for the role.  Likewise, if you need test developers, hiring someone who only knows perl is probably not a good fit.  Instead, hire someone who is truly a developer.  My rule of thumb is that if they couldn't cut it as a dev, don't hire them as a test dev.


  1. I'm going to be doing a series not on testing but on the people that carry it out. This will be a post

  2. This is informative blog entry. I liked it. It describes different roles which are complementary to each other.

  3. Surely, This blog information will help us to define the dedicated , long term Automation team strategy. It will help us to build/developed an Ideal Automation Testers Team.

  4. Funny you should bring up this topic.  
    I was at AWTA (Bret Pettichord's Austin Workshop on Test Automation) this last weekend where we ended up discussing tester developers and developer testers in great detail.
    That wasn't our original topic.
    We were there to talk about open source test automation tools.  But the more we talked about tools, the more we got into example test automation code, the more apparent it became that not just programming skills, but true development skills, are essential for good automated test development.
    Being able to crank out a for loop isn't enough.  Knowing multiple languages, understanding good code design, refactoring, and unit testing are important skills for the Test Developer.
    That's a far cry from the ability to edit the code generated by a record-and-playback.  And it's making me think a great deal about what skills someone doing test automation really needs to have these days.
    I appreciate your post.  It's added to my thinking...

  5. Thanks Elisabeth.  Are there any notes or papers or anything that were generated at this workshop that I might be able to access?  The subject of what it takes to make a truly good test developer is one I think about a lot and I'm always looking for more input.

  6. Steve asked, "Are there any notes or papers or anything that were generated at this workshop that I might be able to access?"
    You'll find all the publicly available stuff here: http://awta.wikispaces.com/
    See http://awta.wikispaces.com/Role+of+ToolSmith for some details on the tester-developer role...though not all the conversations were captured, so that's just a subset.
    Perhaps the most significant thing (for me) that may come out of AWTA is...
    ...another conference.  
    Chris McMahon and I have started talking about what it would take to pull together a small conference for developer-tester-tester-developers focused specifically on programming interesting tests.  Not on tools.  Certainly not on commercial tools.  But on good programming practices/tips/techniques for tests.  (Our motto: "Show me the code!")
    We'll see what happens.  We're just noodling around with the idea for now...though if we get a bunch of emails along the lines of "Yes! Do it!" we might be persuaded to stop noodling and start planning...

  7. As a programmer turned tester I enjoyed the 3 articles you linked and really looking forward to reading what else you have to say
    Sadly where I work we're not in a position to hire great testers - or any more testers full stop - so any advice on getting the testers we do have to up their game would be useful
    ( and Elisabeth - stop noodling and Yes, Do it  !!!)

  8. Phil said, "any advice on getting the testers we do have to up their game"
    What do you wish they knew how to do really well?
    BTW, Brian Marick's book "Everyday Scripting with Ruby: For Teams, Testers, and You" is due out any day now.  I read a pre-release copy.  Great stuff.  http://www.amazon.com/exec/obidos/ASIN/0977616614

  9. I forgot to say... good post. It's raised a subject close to my heart.
    More and more testers are finding that in order to compete, they need to develop their coding skills... Hopefully, your post will help spark even more discussion on the subject.
    Competing aside, the demand for tester-developers is growing simply due to necessity. Personally, I think it's a good thing since it is opening up new career paths for many.
    All the best

  10. Thanks for an interesting post Steve.
    I generally work between all three roles you identified depending on what my project requires at any given time.
    When working as a Test developer I find that I make much more effort to use an automated unit tests and automated build/test approach than many of the full time developers I work with.  The majority of developers I work with do not use automated unit tests or automated build tests.

  11. Thanks for the in depth response Antony.  I'm glad you got something out of the post and also had a lot to add.  I agree with some of what you said and disagree with other parts.  Let me try to address some of your points.
    1)  In my mind I was highlighting people but I suppose most of what I said could apply to roles as well.  Certainly when viewed as roles people can slot between them as the needs of the team vary.  It is important to have people on a team willing to switch up roles.  However, as I stated, trying to slot someone with a lot of development knowledge into a runtime role for a long period of time will usually cause discontent.
    2)  This is based on my experience but I suspect that most of what I say is applicable to all teams.  It doesn't really matter to me whether the people doing the test development report to the dev manager or the test manager.  They might even be people that take on the role of a dev at other times.  I do have a disagreement with the concept of having the same person developing the code doing all of the test development.  I'm not against unit testing done by dev.  In fact I'm for it.  However, I don't think unit testing is sufficient.  Perhaps I'll post on that in more detail someday.
    3)  I had in mind more ad hoc testing when I wrote this.  Exploratory testing is most often done in a simplistic, ad hoc manner.  That isn't to say it is restricted to that, but more often than not this is how it is carried out in the real world.  I didn't mean to imply that it was only simplistic.  It can be done in great depth.
    4)  I didn't have an explicit hierarchy in mind when I wrote this but I will admit to a bias and thus an implicit hierarchy which probably pervades my writing.  I don't think that there is necessarily a valuation difference between the roles.  Certainly I state that each role is necessary on most teams.  However, I believe it requires more training to be a test developer than to be a scripter and more training to be a scripter than a runtime tester.  There are scripters and runtime testers who are highly specialized and highly skilled but the barrier to entry into those two roles is less than the the barrier to entry into test development.
      I certainly agree that a team of only test developers is not optimal.  Likewise a team of only runtime testers (or scripters) will not work as well as a balanced one either.  I also agree that a good test developer should be able to play the role of a good runtime tester.

  12. I am unconvinced that all roles must be filled by a test team; the context of the project is important. I don't think a compiler back end test team has a lot of use for a runtime tester as you've described them.

  13. A compiler back end test team probably still has an automated suite of tests that have to be run every day.  The person responsible for making sure that they all run, that the machines are in the right shape, etc. is the person I see in the runtime role on that team.  You are right though.  It's not a huge need and it might even be small enough that another person on the team could absorb it.  This falls under "Each project will require a different mix of people."