Why automate game testing?

If you've been gaming over the last 20, 15, 10 or even 5 years you would probably agree that video games have become exponentially more immersive, engaging and most importantly, more fun with every year and with every new console or PC architecture that hits the market. *deep breath*

With many exceptions, of course.

But with the increasing size of game worlds or levels comes increasing complexity. Games aren't just a collection of models, maps, music and dialog. They are a system, and the more components and intricate behaviors that are added to that system, the more fragile it can become. Even after years of incremental improvement, a change to the underlying rules of that system can have far reaching implications to the gameplay experience; sometimes with a very negative effect on the player.

That's just what happened in World of Warcraft when a recent change designed to reduce the scale of player health and damage caused "a slew of issues" from the time required to level up characters to the amount of time needed to kill a boss; turning the game into a "demoralizing grind".

To give a sense of the change, back in 2005 just after the game had launched and already amassed millions of players, the average player HP or "health" stat was around 5000. That's how much "damage" an opponent or bad guy needed to deal in order to kill your player and force you to run from the nearest graveyard to revive your corpse. Today, a player's health can reach several million points, which leads to huge numbers on the screen during gameplay. To combat this inflation, Blizzard changed some of the underlying math in the game, thus reducing the scale by some factor. It would appear the change had unintended consequences (see comments for player feedback).

Changes like this are hard to test for. Any sort of unit test would likely produce the kinds of numbers that the developers were expecting (in isolation), and players testing individual encounters or new content wouldn't have seen the issue. Only when comparing the pre-patch leveling time as a whole, or comparing the duration of major encounters would the issues have been apparent. In a game with literally hundreds of such encounters, and enough content to last a single player or group years, that can be... challenging.

This is where end-to-end test automation can find problems that would otherwise be missed.

Manual testers typically have a series of regression tests that need to be run for every build. Over time, that list of tests grows but only enough to allow testers to focus on new functionality and content. Using GameDriver, tests that would normally be run manually can be captured or coded as automation, and executed with each and every build. This allows QA teams to focus on new features while still covering the majority of existing content, and preventing underlying defects from escaping into production. GameDriver can even run the tests at higher-than normal speed to allow for faster execution of large automated regression test suites.

Now testing a game as massive as World of Warcraft takes a lot planning and investment in the creation of test automation. The best approach is to develop test automation alongside development, but that doesn't mean a game that has already shipped can't benefit from this approach to improve time to market, test coverage and overall game quality.

According to the Chinese proverb;

"The best time to plant a tree was 20 years ago. The second best time is now."