What Is Test Automation?

I talk about it a lot, but I don't know that I've ever defined it.  A reader recently wrote in and asked what exactly this was.  I suppose that means I should give a better explanation of it.

Long ago in a galaxy far, far away, testers were computer-savvy non-programmers.  Their job was to use the product before customers did.  In doing so, they could find the bugs, report them to the developers, and get them fixed.  This was a happy world but it couldn't last.  Eventually companies started shipping things called SDKs which were Software Development Kits full of programming primitives for use by other programmers.  There were no buttons to click.  No input boxes to type the number 1 and the letter Q into.  How was a non-programmer supposed to test these?  Also, as companies shipped larger and larger products and these products built upon previous products, the number of buttons that needed pushing and input boxes that needed input grew and grew.  Trying to test these was like running on a treadmill turned up to 11.  The cost of testing grew as did the amount of time developers had to wait for the testers to give the green light to ship a product.  Something had to be done.

The solution:  Test Automation.

Test automation is simply an automatic way of doing what testers were doing before.  Test automation is a series of programs which call APIs or push buttons and then programmatically determine whether the right action took place.

In a simple form, test automation is just unit tests.  Call an API, make sure you get the right return result or that no exception is thrown.  However, the real world requires much more complex testing than that.  A return result is insufficient to determine true success.  A function saying it succeeded just means it isn't aware that it failed.  That's a good first step, but it is sort of the check engine light not being lit in the car.  If there is an awful knocking sound coming from under the hood, it still isn't a good idea to drive.  Likewise, it is important to use more than just the return value to verify success.  Most functions have a purpose.  That may be to sort a list, populate a database, or display a picture on the screen.  Good test automation will independently verify that this purpose was fulfilled.

Other advanced forms of test automation include measuring performance, stressing the system by executing functionality under load, and what we call "end to end" testing.  While unit tests and API tests treat methods and modules as discrete pieces and test them in isolation, end to end testing tries to simulate the experience of the user.  This means pressing the buttons in Windows Media Player to cause a DVD to play and then verifying that it is playing.  Sometimes this can be the most challenging part of testing.

Here's an example of something we had to automate.  Think about how you might approach this.  Windows Vista offers per-application volume.  It is possible to turn down the volume on your World of Warcraft game while leaving Windows Media Player playing loud.  To do this, right-click on the speaker icon in the lower-right-hand corner of your screen and select "Open Volume Mixer."  Moving the slider for an application down should cause its volume to attenuate (get quieter).  Testing this manually is easy.  Just play a sound, lower the volume, and listen.  Now try automating it.