Barren Planet: Putting it to the Test

Tuesday, 1st August 2023

I've learned from previous projects how best to tackle testing, and I'm still learning now. I favour private beta testing over public. Public beta testing might get more volunteers, but I've found I get lots of downloads and next to no feedback that way. With a private beta test I know who the testers are, and I can contact them directly from time to time to get feedback.

So when Barren Planet was ready, I contacted about a dozen people who had been supportive of the project to see who would be interested in trying it out. Nine volunteered. I'd be happy at four active beta testers, so nine volunteers was a good number. I duly sent out the first beta. If even a third of those volunteers sent me feedback, that would be very useful.

Testers often come up with interesting ideas for improvement, but I have found that the beta test phase is not the best time for adding new features. Not only does this open the door to new bugs, but the addition of ad-hoc features at this late stage can lead to problems like imbalance, an untidy user interface, and feature creep. So I tend to tell the testers that the project is feature-frozen at the start of the beta test, and that ideas for additional features will be reserved for sequels or future projects. Most testers are understanding about this.

So what does that leave for the testers to do? Bugs are the obvious answer. But there are also issues like game balance and user interface problems to be identified. In the case of Barren Planet, some users found the units of one side difficult to tell apart from their enemies. Some testers were also finding the computer player quite easy to beat. These were the kinds of issues we fixed during testing.

During the beta phase of Barren Planet another lesson was impressed upon me. Round-robin messages are convenient for sending out the test versions to everyone, but not an effective way to gather feedback. A couple of beta versions of Barren Planet passed with very little feedback, and only from a couple of players. So I've taken to contacting everyone individually, shortly after each version is sent out, to check how they're getting on with it. Lots of interesting feedback was generated from doing this.

I gave each beta test version a week. I built expiry dates into the betas to make sure that testers were playing the latest version, so I wouldn't get too many reports of bugs I'd already fixed (and testers wouldn't miss new bugs I might introduce). Keeping on top of feedback, after one week without new bug reports I decided that the game was ready for release.