Interviewing the Experienced

This week there was an interesting conversation over on Slashdot.  The subject of the post is an age discrimination suit against Google.  However, the discussion has gone to other interesting places.  The question is being asked if there is a difference in the way you should interview experienced people vs. those just out of school.  It reflects something I've come to understand after years of interviewing.  The software/IT industry is relatively young.  This is true both in terms of the workforce and the maturity of the work ecosystem.  The result of this is that we don't tend to have very sophisticated interview processes.  To be sure, they are good, but they are not flexible enough.  They are aimed at hiring the young hotshot recent graduate.  They are not usually designed to find someone with experience.  That will have to change as the median experience in the industry raises.

What is it that biases interviews toward those recently in school?  The questions we ask favor those who have graduated in the near past over those who have been in the industry for a decade or two.  It is pointless to ask a newly minted graduate about his/her experience.  They will likely have a few group projects and perhaps a summer internship to talk about, but the projects are all limited in scope.  Even someone who has been in the workforce for a few years will likely have been implementing someone else's design.  Assessing non-trivial design skills is hard and still harder in an hour-long interview.  So often we don't evaluate this.  Instead, we turn to the easily measurable.

We, as an industry, tend to ask a lot of questions that are more about particular knowledge than about problem solving skills.  Sure, we couch them in terms of problem solving skills, but they are really about particular knowledge.  Asking someone to describe OS internals (processes, memory, garbage collection, etc.) is biased toward someone who has recently taken an OS class or the few who work on the guts of operating systems.  It's not that most experienced people can't understand them, it is that they don't need to.  Many coding questions are about syntax or simple tasks no one faces in real life.  Asking someone to swap the words in a string without using extra memory isn't about problem solving as much as it is about knowing the trick.  Expecting someone to be able to characterize the running time of some algorithms is a test for how long it has been since they last took an Algorithms class.  I've even seen people take the simple question of "sort this list" and turn it into "can you implement quicksort in your head?"  Why would anyone need to?

As developers grow in experience, their work changes.  While the specifics of an implementation are still important, the harder work is about the interaction of objects.  Design becomes more important than syntax.  Strategy more important than tactics.  This has an impact on their specific knowledge.  Everyone knows that most of what we learn for a CS degree is of little value in the industry.  Why then should it be surprising that those whose graduation date has a different 3rd digit than that of the current year don't have fast recollection of such facts?  There is only so much information a mind can hold at a time.  It is thus more important to know where to find an answer than to be able to recall the answer off the top of one's head.

If the test of tactical information is not what we need on bigger projects and is not what experienced programmers are best at, how do we change the interview process to dig out the strategic skills we should be looking for?  Unfortunately, I don't have any great answers.  I'm hoping some of you can help here.  My best advice is to think about the questions you are asking.  Ask yourself if the skill required to solve it is something an experienced programmer would use every day.  If not, don't ask the question.  Instead, look for questions that are more central to work people actually do.  Also, make sure to expand the repertoire of questions to include design questions.  These take more time to think up than "code a fifo queue" but are a better judge of the utility someone will bring to the team.  Don't be afraid to ask softer questions about a person's experience.  What were the difficult problems the candidate ran into on their projects?  How did they handle them.  These questions will elicit low-value responses from a recent grad but a cornucopia of information from someone who has been around the block a few times.