Hiring Great Testers - Interviewing

A friend IM'd me yesterday saying he was being considered for a test manager position somewhere.  In that position, he would be responsible for building up his test team.  He solicited advice on what to look for and how best to interview for testing positions.  That's probably as good an incentive as I'll get to write the next piece of my "Hiring Great Testers" series. 

It's "easy" to interview a developer.  The required skillset is obvious and potential questions are posted all over the web.  Sometimes it seems that if you can reverse a string and parse a binary tree, you can be hired as a dev.*  Interviewing testers is a lot harder.  The skills are not as well defined.  Many test teams also hire from a different pool than development teams do.  There are a lot more people hired into testing positions without formal education or experience.  How do we go about sifting the wheat from the chaff in the testing pool?  What follows are some of my observations over nearly a decade of interviewing testers.

When interviewing testers, there are four main areas you want to probe:  motivation, technical skills, problem solving, and testing acumen.  The exact questions you ask in each of these areas will depend on the specific role you are looking to fill.

Motivation

The goal here is to understand why the person wants to work as a tester.  Do they see this as an entry position for a role as a dev later?  Do they enjoy testing?  Do they think it will make them rich?  The best way to get at this is to ask from two angles.  First, just ask straight up why they want this job.  Sometimes I'll use their previous experience to guide the question.  "You have a lot of experience with radio electronics.  Why do you want to work in computers instead of staying in that field?"  Second, you should ask about future plans.  Where does the person want to be in 2-5 years?  This will give you a hint about what they ultimately want to do and why they might want the job.  What you are looking for in both of these is a desire to do (for now), the job you have open.  If this is only a stepping stone, that might be trouble.  Being a stepping stone isn't bad, but it has to be one they will enjoy standing on for a while.  One other thing to look for here is passion for technology.  Do they play with computers on their own time?  If so, how?  Do they have a hobby relating to the problem your software solves?  It is not necessary that they do but the more they care about the area you work in, they better motivated they will be.

Technical Skills

The specifics of what you are looking for will vary depending on the position.  If it is an SDET interview, you'll want to look for programming skills (can they code a linked list?).  If it is more of an STE interview, you'll want to focus on the relevant technologies.  Do they understand computer hardware?  Have they used your application or one like it before?  Can they troubleshoot computer problems?  If they are an experienced tester, you might ask about testing theory (equivalence classes, pairwise testing, model based testing, etc.) or their ability to write test plans.

Problem Solving

It is important for a tester to be able to think like a computer.  They have to be able to recognize not just that things went wrong, but why they went wrong.  It is critical to be able to see two problems and understand that they have the same underlying root cause.  This is not easy to get at directly in an interview but I believe you can approximate it by seeing how well the candidate can work out logical puzzles.  I recommend using logic questions here.  Not all logic questions are good.  Some are more brain teasers than logical puzzles.  By that I mean, they require outside information not given in the framing of the question.  Avoid those.  I'll write another post on this subject soon.  You are not necessarily looking for correct answers here but rather the way the candidate approaches the problem.  Are they capable of breaking it down into its constituent parts?  Do they approach it systematically?  If so, they are probably logical enough to be great testers.  If not, they'll often struggle to understand why the system is breaking.  Not understanding why means they won't know how to identify weak points that might exhibit similar erroneous behavior.

Testing Acumen

How good is the candidate at the act of testing?  If the position requires programming and you have asked them a programming question earlier in the interview, ask them to test their solution.  Pretend it is a black box, what would they do to determine that it behaves correctly?  The best answers here examine not just the errors that could be exhibited in their particular solution but also the errors that other solutions may have.  If you haven't asked a programming question, try asking them to test an object they are familiar with.  Sometimes it works to pick something in the room.  "Test my phone."  When I ask this question, I'm looking to see two things.  First, are they creative?  Do they have a lot of ideas?  Someone who stops after 5 tests doesn't cut it.  Second, are they systematic?  Do they cover all angles?  Are there major areas the person forgets or avoids?  Note that the second is not the same as "Did they organize their tests?"  I've seen many an interviewer say something like, "The candidate had a lot of creative ideas but they didn't group them into areas."  My response is always, "Did you ask the candidate to group them?  Are you convinced they would not have been capable had you asked?"  It's important to judge the candidate on the quality of their answer versus what you asked for and not what you hoped for.  Remember, the candidate cannot read your mind.  An answer that is different is not necessarily wrong.

Those are the major sorts of questions I tend to ask in testing interviews.  If the candidate is well motivated, has the right technical skills, is able to solve difficult problems, and has an ability to come up with good test cases, they'll likely make a good tester.  If they fall short on any of these areas, think twice before hiring them.  It is usually better to forego an acceptable candidate than to hire a poor one.  Hold out for a great candidate.  Settling for a mediocre one is usually not worth the risk.

 

Note that this information should be used as guidance--not prescription.  Take what you want, leave the rest.  There is no one perfect way to interview and no set of questions will always weed out the bad or even allow through all the good.

Have suggestions?  What questions do you find effective?  Share with others in the comments.

 

* I'm kidding here, but these sort of questions do come up a lot.