Standard software development calls for a large suite of test cases as you're programming. The idea is that you write the tests for whatever you're going to build, then build it and check it passes the tests. The many benefits of this technique include:
- If someone else plays with your code they have more of an idea as to what the outcome should be.
- When something just won't work, running all the tests can highlight where the issues lie.
- you can automate tests so people get annoying emails if the code the checked in doesn't pass all the tests.
- It forces developers really understand what you're doing before you start coding.
The problem lies in the fact that it is very difficult to unit test a mostly visual medium. Does the character look right? Is she triangulated nicely? Do the buttons line up? What happens when you move the mouse slightly while clicking a button, do the characters shatter when you drag the window around the desktop?.
Standard agile practice would have us adding a unit test for each instance, there are techniques that we can use for this - the java.awt.Robot takes you so far - you can automagically move the mouse to a location and click, or check the colour at a location. But it becomes very fragile - if a button moves location or the artists change the skin tone you "red line" on the tests. Keeping in the green takes an intolerable quantity of time, compared to eye-balling the output as you go.
What are we doing at the moment? We've got a really really fast turn around between the programmers (~minutes) - "you forgot to add that jar to the build paths". Then there's a quick turn around between the QA guys and the programmers (~hours) - "I can't save my movie". We've got a slightly longer turn (~days) around between the QA guys and the in house machinimators (product users) - "My video file has more artefacts than it did". And we've got ~weeks between QA and our cutting edge beta testers - "you can't save the movie when you try to open door #23 on a tuesday". Each layer removes some of the problems and the most annoying problems are flattened first.
It's not a textbook operation, but with a small team it works surprisingly well.