Why building software isn’t like building bridges

I was having a conversation with a friend the other night and we came across the age-old “software should be like building buildings” argument.  It goes something like this:  Software should be more like other forms of engineering like bridges or buildings.  Those, it is argued, are more mature engineering practices.  If software engineering were more like them, programs would be more stable and projects would come in more on time.  This analogy is flawed.

Before I begin, I must state that I’ve never engineered buildings or bridges before.  I’m sure I’ll make some statements that are incorrect.  Feel free to tell me so in the comments section.

First, making software, at least systems software, is nothing like making buildings.  Engineering a bridge does not involve reinventing the wheel each time.  While there may be some new usage of old principles, there isn’t a lot of research involved.  The problem space is well understood and the solutions are usually already known.  On the other hand, software engineering, by its very nature, is new every time.  If I want two bridges, I need to engineer and build two bridges.  If I want two copies of Windows XP, I only engineer and build it once.  I can then make infinite perfect copies.  Because of this software engineering is more R&D than traditional engineering.  Research is expected to have false starts, to fail and backtrack.  Research cannot be put on a strict time-line.  We cannot know for certain that we’ll find the cure for cancer by March 18, 2005.

Second, the fault tolerances for buildings are higher than for software.  More often than not, placing one rivet or one brick a fraction off won’t cause the building to collapse.  On the other hand, a buffer overflow of even a single byte could allow for a system to be exploited.  Buildings are not built flawlessly.  Not even small ones.  I have a friend who has a large brick fireplace inside their room rather than outside the house because the builders were wrong when they built it.  In large buildings, there are often lots of small things wrong.  Wall panels don’t line up perfectly and are patched over, walls are not square to each other, etc.  These are acceptable problems.  Software is expected to be perfect.  In software, small errors are magnified.  It only takes one null pointer to crash a program or a small memory leak to bring a system to its knees.  In building skyscrapers, small errors are painted over.

Third, software engineering is incredibly complex—even compared to building bridges and skyscrapers.  The Linux kernel alone has 5.7 million lines of code.  Windows 98 had 18 million lines of code.  Windows XP reportedly has 40 million lines of code.  By contrast, the Chrysler building has 391,881 rivets and 3.8 million bricks.

Finally, it is a myth that bridge and building engineering projects come in on time. One has to look no further than Boston's [thanks Mike] Big Dig project to see that.  Software development often takes longer and costs more than expected.  This is not a desirable situation and we, as software engineers, should do what we can to improve our track record.  The point is that we are not unique in this failing.

It is incorrect to compare software development to bridge building.  Bridge building is not as perfect as software engineers like to think it is and software development is not as simple as we might want it to be.  This isn’t to excuse the failings of software projects.  We can and must explore new approaches like unit tests, code reviews, threat models, and scrum (to name a few).  It is to say that we shouldn’t ever expect predictability from what is essentially an R&D process.  Software development is always doing that which has not been done before.  As such, it probably will never reliably be delivered on time, on budget, and defect free.  We must improve where we can but hold the bar at a realistic level so we know when we've succeeded.

Recommended Books On Testing

Another question I was asked via e-mail.  "Do you know of any good books that explain the testing process in detail? I noticed you mentioned Debugging Applications for .Net and Windows, but I am looking for a book that really explains the basics of 'I have this API/application and I want to test it'. "

Let me say up front that I’m not a big fan of testing books.  Most of what they say is obvious and they are often long-winded.  You can learn to test more quickly by doing than by reading.  Unlike programming, the barrier to entry is low and exploration is the best teacher.  That said, Cem Kaner’s Testing Computer Software gets a lot of good marks.  I’m not a huge fan of it (perahps I'll blog on my disagreements later) but if you want to speak the language of the testing world, this is probably the book to read.  Managing the Testing Process by Rex Black gives a good overview of the testing process from a management perspective.  Most testing books are very process-intensive.  They teach process over technique.  How to Break Software by James Whittaker is more practical.  I have read some of the book and have heard James Whittaker speak.  As the title indicates, the intent of the book is to explain where software is most vulnerable and the techniques it takes to break it at those points.

Part of the difficulty with testing books is that there are so many kinds of testing.  Testing a web service is fundamentally different than testing a GUI application like Microsoft Word which is again wholly different than testing a multimedia API like DirectShow.  Approaches to testing differ also.  Some people have a lot of manual tests.  Some automate everything with testing tools like SilkRunner or Visual Test.  Others write code by hand to accomplish their testing.  The latter is what my team does.  Most books on testing will either distill this down to the basics--at which time you have no real meat left--or they will teach you a little about everything and not much about anything.  Read the 3 books I call out above but make sure to adapt everything they say to your specific situation.  Treat them as food for though, not instruction manuals.

Do you have a favorite testing book that I haven't mentioned?  Share your knowledge with others in the comments.

FCC Broadcast Flag has a bad day

In 2003 the FCC ruled that digital television signals could carry a flag which, when set, required specific protections be in place in all hardware/software that handled the signal.  This flag has become known as the broadcast flag.  Recently, several members of the D.C. appeals court heard a lawsuit seeking to block the ruling.  At least two of the judges had unkind words for the FCC.  The flag is currently scheduled to go into effect this July.

News.com:

"You're out there in the whole world, regulating. Are washing machines next?" asked Judge Harry Edwards. Quipped Judge David Sentelle: "You can't regulate washing machines. You can't rule the world."

Reuters:

You crossed the line," Judge Harry Edwards told a lawyer for the Federal Communications Commission during arguments before a three-judge panel of the U.S. Court of Appeals for the D.C. Circuit.

"Selling televisions is not what the FCC is in the business of," Edwards said, siding with critics who charge the rule dictates how computers and other devices should work.

What is a Test Architect?

I was asked a few questions via mail.  Here is the first of some quick answers to these:

   What is the role of a Test Architect?  There is not a single definition of the test architect role.  A test architect is an advanced test developer whose scope is larger and who solves harder problems than the average SDE/T.  The specifics of what they do varies greatly.  They might be solving hard, particular problems.  They might be the one called in to solve the seemingly intractable issue.  They may be designing a new test platform.  Or, they may be determining group test policy.  Any and all of these fall in the scope of the test architect.  The work they do is often similar to that of an SDE/T.  The difference is often one of scope.  An SDE/T will often own a specific technology or be responsible for implementing a part of a system.  A test architect will own the approach to testing or developing entire systems.

   Are you a test architect or do you have a different idea of what one is?  Please leave your opinions in the comments section.  I'd love to see a dialog develop on this subject.  It's something I'm interested in but that isn't all that well understood yet in most quarters.

TDD - First Impressions

      I spent the last two days in a class covering test driven development, unit testing, and refactoring.  I hope to provide a more detailed discussion of what I learned at some later point but for now I thought I'd post my initial impressions.   I've read a lot about TDD, Unit Testing, Refactoring, etc. but I'd never actually *done* test driven development.  The class had several hands-on exercises.  We were using C# and NUnit.  Let me say right here that NUnit is a slick app.  It is unobtrusive which is exactly what you want in a test harness.  This is the report of that experience.

      First off, it felt really strange.  Generally when developing a program, you think about the big picture first and work your way to the details.  With TDD, you end up taking the opposite tack.  Because you have to write a test, fail the test, then write the code, you cannot start with the big parts, instead you start with the small internals.  Of course it is necessary to put some thought into the bigger picture, but very quickly you are driven into the implementation details and that creates a feedback loop for your big picture design.  This feels strange but you become accustomed to it.  The designs that come out seem clean.  Forcing testability causes you to think about cohesion, coupling, redundancy, etc.

      When you are doing test driven development, you are constantly switching between the editor, compiler, and test harness.  You are compiling often.  This is a good thing.  It means that your code is always working.  You don't go on long coding sprees followed by long bug fixing sprees.  Instead, you intermingle the two a lot.  I find it easier to fix an issue as I'm writing it rather than later when I compile some larger chunk of code.

      TDD feels good.  Every time you run the unit tests, you get positive feedback that things are working.  When you make a change, even a radical change, you know that if all the test cases pass that everything is working.  It gives you peace of mind that you cannot get in older coding models.  If you've never done any test driven development, give it a try.  Who knows, you might like it.

      The class was taught by Net Objectives.  This is the 3rd class I've taken from them.  I've also attended a few of their free seminars.  If you have an interest in OO or Agile techniques, check them out.  I highly recommend their work.

Information Technology Leaders

There is a local cable television station run by the University of Washington.  On it there airs a show called Information Technology Leaders hosted by a woman named Laura Shildkraut.  On the show, Laura spends an hour interviewing, you guessed it, leaders in the information technology field.  The guests are CEOs, CIOs, VPs, COOs, CTOs, etc.  Topics covered include not only what they are doing now but what they did in the past and how they got to where they are.  Laura gets some interesting guests.  For instance, she interviews all three Renegades of the Empire (Alex St. John, Eric Engstrom, and Craig Eisler), John Connors, Rick Devenuti, etc.  I find the stories fascinating.  There is also a lot to learn from people who have "made it."  If you have some time, I recommend you check out the streaming video version of the show at http://www.informationtechnologyleaders.com/video.html.

Too Much Test Automation?

            There was a time when testing software was synonymous with manual testing.  Now with the rise of test development and the advent of unit testing, automation is becoming more and more prevalent.  Test automation has many benefits but it is not a silver bullet.  It has its costs and is not always the right answer.  Software development organizations must be careful not to let the pendulum swing too far in the direction of automation.  Automating everything is a way to guarantee you miss many bugs.  It is incumbent upon those in charge of test organizations to find a balance between automated and manual testing.

            It is best to start out with a definition.  Test automation is code or script which executes tests without human intervention.  Simple automation will merely exercise the various features of the product.  More advanced automation will verify that the right actions took place.  Results of these tests are stored in log files or databases where they can be rolled up into reports listing a pass/fail rate.

            Automation has many advantages.  Manual testing is expensive.  It requires people to click buttons and observe results.  This isn’t terribly expensive the first time through but the costs stay fixed as the product progresses.  If you have a daily build (as you should), you pay the cost daily.  This quickly adds up.  Automation is expensive to create but the incremental cost is very cheap.  Automation is also consistent.  No one will ever forget to run a test case with automation.

            Because it is cheap, automation can be run in places where manual testing cannot.  Extensive tests can be run on daily builds.  These can even be run in the wee hours of the morning before everyone shows up for the day.  The inexpensive nature of automated testing is what allows unit tests and test driven development to be possible.

            Some test cases cannot easily be done manually.  When testing an API such as the DirectShow API I often work with, there is no UI to drive it.  At least some minimal coding must be done to expose the API to a user.  In these cases, automation is the obvious choice.

            Despite its advantages, automation is not a panacea.  First, it is not free.  Automation is expensive to create.  If you are able to amortize that cost over a lot of runs, the incidental cost becomes low.  On the other hand, if the test is something that will only be run a few times, automation may be more expensive than manual testing.  Decision makers must consider the high initial cost before committing their organization to automated tests.

            Secondly, automation is limited in scope.  After you have run your automated tests for the first time, you are done finding new bugs.  Never again will you find a new issue.  You might catch a regression, but if you missed the fact that clicking a particular combination of buttons causes the app to crash, you’ll never find that issue.  This is true no matter how many times you run the automation.  On the other hand, high quality manual testers will take the opportunity to explore corner cases.  In doing this, they will find many issues that would otherwise go unnoticed until real users find them after the product is shipped.  This lack of exploratory ability is, in my mind, the Achilles heel of automated testing. 

            The third drawback of automation is that there are things that simply cannot be automated well.  I work in the field of audio-video playback.  Trying to automate video quality tests is hard.  It is very expensive—specific media must be pared with specific test cases.  If you have to account for variable content (say, testing television tuners) or if the algorithm is not fixed (varying video cards or decoders), the task becomes even harder.  Using something like PSNR is possible but is not well tuned to the human visual system and thus is susceptible to false positives and false negatives.  Sometimes there is no replacing a person staring at the screen or listening to the speakers.

            Other problems also exist with automated tests.  Bugs in an automated test may mask a bug in the product and go unnoticed for long periods of time.  Automated testing does not emulate real users using a product nearly as well as a real person.  Manual tests can be brought online more quickly than automated tests allowing bugs to be found sooner.

            What then, is a test manager to do?  Manual testing is too expensive but automated testing is imperfect.  It is imperative that the two sorts of testing are mixed.  Automated testing should be used to verify basic quality but cannot be used exclusively.  There need to be real people simulating the experience of real users.  With the hiring of developers and not scripters as test developers, the quality of automated tests can be made very high.  Unit testing can ensure that automated tests are available early in the cycle.  With the increased sophistication of test developers and automation techniques, automated testing can take a larger role than in the past.  It cannot, however, replace real testers.