Through a few twists of fate, I ended up at Microsoft as a test developer (lead). It’s not something I ever considered doing before landing here and I’m sure it is not something a lot of you have thought much about. It is the goal of this post to introduce you to who test developers are, where they fit into the team, and what sort of work they do.
Most people seem to have a view of the software world divided into two sorts of people: developers and testers. Devs are the real heroes of software. They get to write code. Testers are a necessary evil. They help to keep the developers from screwing up too badly but don’t produce any positive results. I won’t go into the role of test in this post. I’ll save that for another day. The thing is, most people think of testers as the people who push a lot of buttons and try to break things. They run tests. Perhaps they even cobble together some script and batch files or a VisualTest “program” to help them out. Those sort of testers do exist. I’ll call them runtime testers or software test engineers (STEs). At Microsoft, however, a lot of our testing ranks are filled with the rare breed known as the Test Developer (SDE/T).
So what is this test developer then? Is he someone who just isn’t quite good enough to be a “real” developer? I posit that the answer is no. A good test developer is as good as any developer writing production code. A good test developer is worth his weight in gold. A test developer’s job is to develop software which exercises a product to ensure its correctness and gauge its stability. He has to exercise every corner of the code before it ships and usually before it is documented. On my team, test developers are developers who know how to test, not testers who know how to develop.
In my business, which is multimedia (audio and video rendering), we produce not polished applications but the APIs which underlie such programs as Windows XP Media Center Edition and Windows Media Player. Countless other applications use our APIs underlying their games, video editing suites, and media players. Our work product forms the building blocks of many an application. Whenever you see video or hear sound from your computer, we’re probably underneath doing something.
So, who is it that makes sure that these APIs actually work as advertised? It is the role of the test developer to make sure that developers using our APIs have a pleasant experience. We are the front line of defense for the developers out there in the “real world” trying to use Windows to make a buck or two. We are the first ones to try to implement anything on top of those interfaces. It is our job to verify that they work as advertised.
We accomplish by writing a lot of code to exercise the interfaces. While it starts with the fairly mundane testing of each method, it doesn’t end there. It includes writing sample code (which may ship in the SDK) to do exactly what those coming after will try to do. It involves trying to simulate the end-to-end scenarios this component is likely to be used for. Most interesting, is trying to validate that the right action just took place. Anyone can call an API and determine if it reports that it did the right thing. It is a whole other ballgame to try to verify that the audio mixing component you are testing really did mix the audio correctly or that the video decoder really output the picture you expected and not some unintelligible pattern of greens and pinks. The role of the test developer is to do all of this, and to do it without an SDK to guide him. He’s the first one trying it, after all.
Test developers are tasked with one thing: writing code and writing it first. They are always breaking new ground: going where no one has gone before. It is a challenging and yet rewarding endeavor. Great software is not written by some developers and their unit tests. It takes someone to go beyond the granular and ensure that the whole package works together. When your deliverable is not a whiz-bang UI but rather a set of building blocks for someone else, that task falls upon the shoulders of the test developer.
That is probably enough for now. Anyway, if you are out there looking for work as a developer, don't skip over all those SDE/T jobs. Our code may not ship in the box but it's not a job for someone who isn't good enough to be a “real” dev.
[This posting is provided "AS IS" with no warranties, and confers no rights.]
ReplyDeleteInteresting. I have a couple questions :
- How do you rank methodology among the top 5 criteria for a good job as a SDE/T person?
- Isn't this dev root a problem, someone with a dev state of mind tries to figure out things may have a much harder time (especially finding new bugs) than someone from a usability group, not technical at all?
- Regarding the team you work in, does it relate to DirectShow? It really looks like no one cares about DirectShow anymore... :-(
- In test cases, do you test in MS-environments? or in other words what is the amount of testing performed on real world machines, with a bunch of components, codecs, players and so on that interfere with the core OS components?
- Without saying anything regarding the windows media infrastructure, do you have a way to trace what's being done underneath by just registering a COM component or a service? How much do you think this would benefit developers outside?
- How much time do you spend visually testing the results, versus the time spent on those that can be checked silently and with no human intervention?
Let me take these in order:
ReplyDelete- I'm not sure quite what you mean by methodology. I'll assume you mean testing methodology. If that's the case, I rank it pretty low. The things I would rank ahead of that include strong coding skills, problem solving skills, and communication skills (are you willing to defend your position?). I find that with a modicum of curiosity and some interest, I can convert mode developers into test developers. I've had plenty of chances to try. We often bring in contractors who are SDEs and make them test developers. I've had very few who can't handle the testing. On the other hand, most STEs can't be good devs.
- Absolutely not. How do you think extreme programming and unit testing work if developers cannot think like testers? If someone has some curiosity, they can be a good tester. To be a good developer means you probably spent a lot of time on your own tinkering with computers. Where does that urge to tinker come from? Curiosity of course. That's the same curiosity that makes a good tester. While there is a place for a wholly creative mind, I have yet to meat anyone from a usability group that can write a DFT or calculate the PSNR of two frames of video.
- Yes. The work I do relates to DirectShow. I wouldn't say no one cares any more. We're still fixing bugs. There's even a feature or two being added over time. A lot of people still use it.
- Where the test cases are run is generally more a responsibility of runtime testers than test developers. We write the tests, they execute them in numerous environments. When I was an STE, we used plenty of real-world machines.
- In DirectShow, everything revolves around the graph object. GraphEdit (a tool which ships with the DX SDK) can be used to snoop any graph that is added to the running object table (ROT). The <a href="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/directx9_c/directx/htm/loadingagraphfromanexternalprocess.asp">code</a> to add a graph to the ROT can be found on MSDN. This is easy if you have the source to the app. If you don't, it is harder but still possible. Perhaps I'll write an article about how to do that sometime.
- When I first began, most of it was visual. Today, the majority of the tests we write can validate the output without human intervention. We still do a lot of looking with real eyeballs though. There's a lot about video a human can see that a programmer may not think to look for.
ReplyDeleteWheeew, a lot to digest :-) Thanks for the answers.
Regarding methodology, I was talking about testing methodology but may be even more general methodology than that, the ability to execute a plan without being taken off-track by the technical constraints of the environment. Let's take an example : writing a robust test with VisualTest. Since most if not all screen recorder tries are not even replayable, mostly lack invariants aimed to be robust enough and adapt the tiny changes, and so on, I think it's pretty much impossible to write a testing framework based on VisualTest without being undermined, constrained, stuck, or whatever else by this environment. Of course, pretty much any environment has its limits, but do you think those persons using VisualTest know what was the state of mind of the person who made that environment? (like testing owner-drawn controls consistently, although VisualTest obviously cannot guess the things buried in the target app).
Regarding the dev state of mind, I also have a problem with with how it is supposed to work. I mean, developers implement solutions so they work. Because of their focus, their scope has to be limited to what they do the most during the day. Testers will implement solutions so they break (unless they are doing regression checking). And their scope can be an order of magnitude different. How do you reconcile the parties under a single SDE/T job ?
I can't comment a whole lot about Visual Test. I think I used it once. It was interesting but I'm not really sure how useful much of it is. If you really want to test your app, you should be testing the "business logic" apart from the UI. Each of the underlying components should be tested separately. That way, when you get to the UI, all you need to test is the UI itself. Visual Test can be useful there but it was really designed for older Windows Apps and not the new custom-UI stuff that makes up most apps nowadays. One way to test the UI is to have a common dispatching mechanism (be is a message queue or callback functions) that drives all of the UI elements. That way, a program can interact with the UI without having to actually click on buttons, etc.
ReplyDeleteTest Developers also implement solutions so that they work: they exercize the unit under test in ways that will break it. All it takes to be a good tester is some curiosity. If you think about where most programmers began, it was messing around to see what their computer could do. That is curiosity. A SDE/T is merely someone who continues to exercize that curiosity while they get paid to program rather than simply implementing what a program manager told them to.
I've hired over a dozen seasoned developers over the past few years. None had done formal testing before. With few exceptions, they all made good testers with a bit of training. The ones who failed probably weren't good programmers either. The advantage of a test developer is that not only can he think of a cool way to break the app, he can actually do something about it.
I'm going to be doing a series not on testing but on the people that carry it out. This will be a post
ReplyDeleteOn his blog, Microsoft employee Steve Rowe discusses the top three reasons to consider being a Test Developer
ReplyDeleteSDET … SDE/T … Tester … Software Development Engineer in Test. Whatever you want to call them, here’s
ReplyDeleteOn his blog, Microsoft employee Steve Rowe discusses the top three reasons to consider being a Test Developer (aka a Software Design Engineer in Test or SDET.) Reading his post prompted me to look at other entries on his blog, and I noticed he’s also
ReplyDeletePingBack from http://eduardk.wordpress.com/2008/02/27/rolul-de-software-design-engineer-in-test-la-microsoft/
ReplyDeletePingBack from http://greenteafatburner.info/story.php?id=3477
ReplyDelete