Information Technology Leaders

There is a local cable television station run by the University of Washington.  On it there airs a show called Information Technology Leaders hosted by a woman named Laura Shildkraut.  On the show, Laura spends an hour interviewing, you guessed it, leaders in the information technology field.  The guests are CEOs, CIOs, VPs, COOs, CTOs, etc.  Topics covered include not only what they are doing now but what they did in the past and how they got to where they are.  Laura gets some interesting guests.  For instance, she interviews all three Renegades of the Empire (Alex St. John, Eric Engstrom, and Craig Eisler), John Connors, Rick Devenuti, etc.  I find the stories fascinating.  There is also a lot to learn from people who have "made it."  If you have some time, I recommend you check out the streaming video version of the show at http://www.informationtechnologyleaders.com/video.html.

Too Much Test Automation?

            There was a time when testing software was synonymous with manual testing.  Now with the rise of test development and the advent of unit testing, automation is becoming more and more prevalent.  Test automation has many benefits but it is not a silver bullet.  It has its costs and is not always the right answer.  Software development organizations must be careful not to let the pendulum swing too far in the direction of automation.  Automating everything is a way to guarantee you miss many bugs.  It is incumbent upon those in charge of test organizations to find a balance between automated and manual testing.

            It is best to start out with a definition.  Test automation is code or script which executes tests without human intervention.  Simple automation will merely exercise the various features of the product.  More advanced automation will verify that the right actions took place.  Results of these tests are stored in log files or databases where they can be rolled up into reports listing a pass/fail rate.

            Automation has many advantages.  Manual testing is expensive.  It requires people to click buttons and observe results.  This isn’t terribly expensive the first time through but the costs stay fixed as the product progresses.  If you have a daily build (as you should), you pay the cost daily.  This quickly adds up.  Automation is expensive to create but the incremental cost is very cheap.  Automation is also consistent.  No one will ever forget to run a test case with automation.

            Because it is cheap, automation can be run in places where manual testing cannot.  Extensive tests can be run on daily builds.  These can even be run in the wee hours of the morning before everyone shows up for the day.  The inexpensive nature of automated testing is what allows unit tests and test driven development to be possible.

            Some test cases cannot easily be done manually.  When testing an API such as the DirectShow API I often work with, there is no UI to drive it.  At least some minimal coding must be done to expose the API to a user.  In these cases, automation is the obvious choice.

            Despite its advantages, automation is not a panacea.  First, it is not free.  Automation is expensive to create.  If you are able to amortize that cost over a lot of runs, the incidental cost becomes low.  On the other hand, if the test is something that will only be run a few times, automation may be more expensive than manual testing.  Decision makers must consider the high initial cost before committing their organization to automated tests.

            Secondly, automation is limited in scope.  After you have run your automated tests for the first time, you are done finding new bugs.  Never again will you find a new issue.  You might catch a regression, but if you missed the fact that clicking a particular combination of buttons causes the app to crash, you’ll never find that issue.  This is true no matter how many times you run the automation.  On the other hand, high quality manual testers will take the opportunity to explore corner cases.  In doing this, they will find many issues that would otherwise go unnoticed until real users find them after the product is shipped.  This lack of exploratory ability is, in my mind, the Achilles heel of automated testing. 

            The third drawback of automation is that there are things that simply cannot be automated well.  I work in the field of audio-video playback.  Trying to automate video quality tests is hard.  It is very expensive—specific media must be pared with specific test cases.  If you have to account for variable content (say, testing television tuners) or if the algorithm is not fixed (varying video cards or decoders), the task becomes even harder.  Using something like PSNR is possible but is not well tuned to the human visual system and thus is susceptible to false positives and false negatives.  Sometimes there is no replacing a person staring at the screen or listening to the speakers.

            Other problems also exist with automated tests.  Bugs in an automated test may mask a bug in the product and go unnoticed for long periods of time.  Automated testing does not emulate real users using a product nearly as well as a real person.  Manual tests can be brought online more quickly than automated tests allowing bugs to be found sooner.

            What then, is a test manager to do?  Manual testing is too expensive but automated testing is imperfect.  It is imperative that the two sorts of testing are mixed.  Automated testing should be used to verify basic quality but cannot be used exclusively.  There need to be real people simulating the experience of real users.  With the hiring of developers and not scripters as test developers, the quality of automated tests can be made very high.  Unit testing can ensure that automated tests are available early in the cycle.  With the increased sophistication of test developers and automation techniques, automated testing can take a larger role than in the past.  It cannot, however, replace real testers.   

Language Choices

It has been asked several times which language is better for a test developer at Microsoft, C# or C++?  The answer really depends on what group you will be working in.  Different groups require different skills.  It is my perception that C++ is by far the dominant language today but that will not always be the case.  I have friends that are SDE/Ts who work exclusively in C#.  I'm sure there are some on the VB.Net team who do all of their work in VB.  There are probably some on the Office team who use VBA for most of their work.

For now most things are being written in C++ because of the extra runtime speed and the availability of libraries and APIs in the language.  In theory C# could become as fast as or faster than C++.  It has the ability to dynamically compile and thus take advantage of processor and system features not available to the least-common-denominator approach of a traditional compiler.  Most likely, in the real world, it will never quite catch up.  The C# team is working hard to minimize the difference between the two.  See Rico Mariani’s blog for some posts on the subject.

This isn't to say that C# won't become the dominant language though.  Hand-tuned assembly is often faster than C code and C code can be faster than C++.  Those advantages didn't stop C++ from becoming the dominant language, however.  Why is that?  It is because C++ is faster to code in.  It offers native support for object-oriented coding and abstract data types.  These increase the efficiency of the programmer.  This increase in efficiency is accepted at the price of code that runs a bit slower.  C# offers a similar bargain.  Coding efficiency is up but performance is down.  As processors continue to increase in speed, the need for that extra ounce of speed will be less and less advantageous and the increased coding efficiency relatively a greater and greater advantage.  When the APIs for coding Windows applications become managed with Avalon and WinFX, the usefulness of C# will be magnified.

Over the long term, I expect that C# will become a dominant language.  Will it fully displace C++?  That is hard to say.  Certainly not in the near term.

I must end this by saying that I am no expert on C#.  I've done some coding in it but nothing too serious.  There are others on blogs.msdn.com that know a lot more about the subject.

Disclaimer - Please Read

These postings are provided "AS IS" with no warranties, and confers no rights.  This site contains my personal opinions and does not necessarily reflect the viewpoint of my employer, Microsoft Corporation.  The thoughts and opinions expressed herein are solely my own.

Three Reasons To Consider Being a Test Developer

            When it comes to careers in the world of software most people think of programmers or what are more formally known as developers.  Developers are the people who write the software which is consequently sold or utilized by the organization.  I’ll call them dev-developers to distinguish them from test developers.  Sometimes people will also think of testers.  Testers are the people who don’t program but run the software written by developers to find bugs.  What gets lost in the shuffle is a specialized class of developers who are also testers.  We like to call them test-developers.  These are the people that write software to test software.  It is they who will be the subject of this essay. 

            The term “test developer” is sometimes used to refer to a tester who knows a scripting language like Javascript or Perl or maybe even knows VB.  Usually this person has no formal training and takes on only simple tasks.  That is not what I refer to in this essay.  The test developers I am referring to are a specialization of developers.  They write complex code in production languages utilizing computer science techniques.  See my previous essay on why test developers are real developers. 

            So, why should you consider becoming a test developer?  Why not just become a dev-dev instead of a test-dev?  That is the subject of this essay.  There are three primary reasons to become a test-dev rather than a dev-dev.  These are that it makes you a better programmer, the code you write is more broad, and it is at a sweet spot in the development cycle.

            Programmers who are or have been test developers are, on average, better programmers than those who have not.  They have a feel for what is likely to go wrong with software and so code for failure instead of coding for success.  All too often those who have not been testers write code until it works and then stop.  They write so that it can work but not so it will always work.  Test-developers have practice breaking programs and so know where they will encounter problems.  They are thus more successful at anticipating what error conditions may happen and writing resilient code.

            Secondly, test developers tend to write code which exercises the product at a higher level.  Instead of focusing all of your effort on a way to make a widget spin, instead you get to see what happens if multiple widgets spin at the same time or how spinning widgets interact with the IFoo class.  Test developers write code that exercises the product as a whole which is often more interesting and more rewarding than spending time optimizing one little corner case.  You get a more wholistic view of the product which leads to better understanding of how the various pieces interact.  Additionally, you often get to do more fun work like determining ways to break the system, put various pieces together, analyze its performance, etc.

            Finally, test development is at a sweet spot in the product release cycle.  Dev-developers work furiously in the early cycle getting their pieces code complete.  They then work furiously late in the cycle fixing the final few bugs.  The end result is often very long hours.  Test developers, on the other hand, are generally under less pressure.  In the early product cycle, you can only work as fast as code becomes available to you.  In the late cycle, your tests are already in place.  If they aren’t, it is too late to add them.  I don’t mean to imply that test developers don’t work hard.  They do.  They just tend to feel less deadline pressure than dev-devs.

            If you have experiences as a test developer that you feel may benefit someone trying to make a career decision involving test development, please share them in the comments section.

So You Want to Be a Test Developer

            So you have an interest in becoming a test developer, eh?  You’ve come to the right place.  In this paper I will lay out what I think is the best path to take to becoming a test developer or really any sort of developer.  Test developers are really just developers who happen to report in the test org. Becoming a good test developer requires learning not only a programming language but also what to do with it.  You’ll need to learn not only C++ but also about operating systems, algorithms, object-oriented programming, etc.  Becoming a test dev won’t be easy.  It is going to take a lot of reading and a lot of practice.  If you stick with the regimen, you’ll be a real developer.  You’ll have much of what someone would have if they had acquired a CS degree.  If not, well, you’ll be where you are today. 

Starting With a Language

            The place to start is obviously with a language.  If you don’t know how to program, you won’t be much of a programmer.  It is important to know, however, that knowing a language is just the beginning.  Too many people equate knowing C/C++ with knowing how to program.  Sadly, this is not the case.  Think of an author as an example.  Learning the English vocabulary and the rules of grammar will make you a proficient English speaker.  That does not, however, make you a great writer.  Mark Twain wasn’t a great writer because he knew how to diagram sentences.  Knowing what to say is even more important than knowing how to say it.  It is the same way with a programming language.  You can know C/C++ forward and backward and still be a bad programmer.

            The first decision you should make is what language to use to start your journey.  As you progress, you will probably pick up other languages but you have to start somewhere.  You should start with a language that encourages you to do the right things.  C# is an easy language to begin with.  C is simple and easy to learn.  C++ is its more complex big brother.  Most system-level work is done in C++.  That’s probably where you want to start.  If something like VB or Perl strike your fancy, you can use them but know that you’ll be limited in where you can utilize your skills.

            To learn C++, I recommend starting with a book.  The two best ones I know of are C++ Primer by Stanley Lippman and Thinking in C++ by Bruce Eckels.  I prefer these over the “24 Hours” or “21 Days” sort of books.  They may claim to teach you C++ quickly but they won’t.  C++ is huge.  It will take some time to master.  Pick one and read it cover to cover.  Do the exercises if you are that sort of person.  These are both thick books.  Don’t be intimidated.  They’ll both cover everything you need to move onto the next phase of programming.

Don’t stop there though.  You need to put this into action for it to make sense.  Come up with a project and start implementing it.  The project can be anything.  Implement a tool you’ve been itching to have, write a game, whatever.  I started by trying to implement the game of Robo Rally.  I never finished.  You might not either.  The point isn’t to complete the project.  It is to come to an understanding of how the parts work together.  Books teach us to program in discrete little packets.  Each packet makes sense atomically.  The point of writing a real program is to understand how these packets interact.

If you want to become wiser in the ways of C++, there are several other books you might want to read including Effective C++ by Scott Meyers and Inside the C++ Object Model by Stanley Lippman.  Neither book is essential but both will help make you a better programmer.

Going Beyond Language

            As I said, learning a language is just a first step.  Once you are here, you are proficient in some language (C++ most likely).  You are conversant in a language.  You are still a terrible writer.  The next step is to learn what makes a good program. 

This is where things get more complicated.  Learning pointers is hard but understanding algorithms is harder.  If your math is weak, you’ll struggle with some of these concepts.  Fear not.  None of this is rocket science.  It just may take you a bit longer.  If necessary, put down the book you are reading, read some math, then come back.  This should help you get further.  If you run into a roadblock, learn more math and continue until you get through.

There are three major subjects you need to come to understand.  The first is data structures and algorithms.  The second is operating systems and processors.  Finally, there is object-oriented programming.  Once you have these three areas understood, you’ll be well on your way to becoming a good programmer.

Algorithms are what make your programs run fast or slow.  When you are dealing with only a small amount of data, the algorithm you use doesn’t matter too much.  However, when you are dealing with hundreds of thousands or millions of items, the algorithm can matter a lot.  It is important to understand the types of algorithms available and to understand how to measure their efficiency.  Different ways of sorting would be an example of algorithms.  Data structures are ways to organize large amounts of data and maintain accessibility.  Examples are linked lists, trees, hash tables, etc.  Two books which are used a lot for this are Introduction to Algorithms by Thomas Cormen and Algorithms in C++ by Robert Sedgewick.

Understanding how processors and operating systems work will help you understand why the language and the OS do what they do.  Many things that C/C++ do make more sense once you understand how the underlying system works.  Understanding the OS and being able to read assembly language will come in invaluable when it comes time to debug.  The best OS book I know of is Operating System Concepts by Abraham Silberschatz.  It is accessible and covers the critical topics.  For processor/assembly topics, look to Computer Organization and Design by David Patterson. 

Finally, there is object-oriented programming.  Learning about OO techniques will help you understand how to write a program that is maintainable and extensible.  The best place to start is with Design Patterns by Gamma et al.  I suggest using to book to understand how patterns work in general, not to memorize every pattern.  A good book showing the application of patterns and good OO design techniques is Design Patterns Explained by Alan Shalloway.

Where to Next? 

            So you read all 5,000 pages and you still want more?  What can you study next?  The tree branches from here.  There are lots of possibilities.  Some good places to look are for general programming are debugging and good programming practices.  You can also start going down a specific path such as windows programming, graphics, ATL/COM programming, etc.

            Programmers spend an inordinate amount of time debugging the code they just wrote or even code written by others.  Debugging is an art, not a science.  The only way to get good at it is practice.  There are some books you can read which will give you a jump start though.  The best of these is Debugging Applications by John Robbins.

            Learning how to become a better programmer also takes practice but there are a few books written which give you some good pointers.  I won’t list them all here.  Everyone raves about Code Complete by Steve McConnell.  I don’t.  I find it long-winded.  A better book is The Practice of Programming by Brian Kernighan and Rob Pike.  Another book which is really useful is Refactoring by Martin Fowler.  He explains how to take old code and make it better.  Like Design Patterns, I find the first several chapters more useful than the reference that makes up the bulk of the book.  Finally, go find a copy of your groups coding standards document.  Often times these are chock full of good techniques.  They will usually tell you what sort of things to avoid and what good practices to emulate are.

Formal Training

            If you decide that self-driven book learning isn’t for you, are there alternatives?  Yes.  I prefer the classic CS education over professional certificate programs, workshops, or even IT degrees.  To that end, I have found a few programs I believe are worthwhile.  Each of these will allow you to get a full degree or just take some classes.  All are based online.

 

Regis University – Offers a Bachelor of Science in Computer Science.  http://www.regis.edu/regis.asp?sctn=onl

 

Colorado State University – Offers a Master of Computer Science program.  http://www.cs.colostate.edu/cstop/csacademics/csdegrees/onlinemcs.html

 

University of Illinois at Urbana-Champaign – Offers a Master of Computer Science degree.  Students participate in on-campus classes via newsgroups and online videos.  http://online.engr.uiuc.edu/

Bibliography:

Cormen, Thomas.  Introduction to Algorithms.  Thorough introduction to algorithms.  The content is often advanced.

Eckels, Bruce.  Thinking in C++.  Focuses a lot on STL which isn’t used much at Microsoft.  You can download a free copy here:  http://mindview.net/Books/DownloadSites.

Fowler, Martin.  Refactoring.  How to spot good and bad code and what to do about the bad stuff.

Gamma, Erich, Richard Helm, Ralph Johnson, John Vlissides.  Design Patterns.  The first few chapters are the most useful.  They will help you understand how OO designs should work.

Kernighan, Brian, Rob Pike.  The Practice of Programming.  Tips and techniques for being a good programmer from one of the authors of the C language.

Lippman, Stanley.  C++ Primer.  Great introduction to C/C++.

Lippman, Stanley.  Inside the C++ Object Model.  The how and why of C++.

Meyers, Scott.  Effective C++.  50 tips for writing C++.  Each tip is explained.  Along the way, you learn a lot of C++.  Skip More Effective C++ though.

Patterson, David.  Computer Organization and Design.  Covers the MIPS processor not the Intel but the concepts are the same.  Don’t get bogged down in the measurements chapter.

Robbins, John.  Debugging Applications for .Net and Windows.  Great book explaining how to debug programs on windows.  Focuses on the Microsoft programming environment but the general techniques are applicable everywhere.

Sedgewick, Robert.  Algorithms in C++.  Sedgewick has a series of Algorithms books covering everything from C to C++ to Java to Pascal.  The latest versions come in two volumes.

Shalloway, Alan.  Design Patterns Explained.  How to use design patterns in design.

Silberschatz, Abraham.  Operating System Concepts.  Good introduction to how operating systems work.

 

If you have suggestions for additional books, subjects, etc.  Please post them in the comments below.

Advice for CS Students

Joel Spolsky (of JoelOnSoftware fame) is well-spoken and often has a lot of good advice for programmers.  This recent essay gives advice to college students on what to study.  His advice consists of:

1. Learn how to write before graduating.
2. Learn C before graduating.
3. Learn microeconomics before graduating.
4. Don't blow off non-CS classes just because they're boring.
5. Take programming-intensive courses.
6. Stop worrying about all the jobs going to India.
7. No matter what you do, get a good summer internship.

If you want to find out his reasons, read the whole article.