Test Automation in Crackdown

An interesting report over at Gamasutra.  Jami Johns from Microsoft Game Studio gave a talk at Gamefest 2007 about how they tested Crackdown.  The thing that strikes me most from the article is the increasing need for automated testing and thus test developers who are true programmers.

A few excerpts:

"For Crackdown, the team changed over to a PC application that would rip the data from the game instead of making people report it manually. This generated maps with big red blotches -- making it easy to find the places where performance took a real hit. One block -- where the developers weren't expecting much traffic -- turned out to be a big performance drain. It turned out that the area offered a great vantage from which to look out over the city, so the testers were flocking there."

"The decision was made to create a new tool -- one that could make dealing with the bugs much, much faster. The team came up with a tool called SWARM. This allowed bugs to be tracked easily: each one had a text description and a screenshot, and it could track every bit of relevant data for each bug. Since it was easy to see the bugs, this stopped duplicate bugs and also made them easy to check. Metadata was stored in each bug's jpeg iamge, which meant when that data was dropped into the Crackdown build, they'd teleport to the bug and verify it."

It sounds like they automated not only the bug finding but also the reporting and the reproduction.  Quite a ways from the days of playing a game with a VCR attached and writing your findings on paper.