Rescued by a Centaur
Darknet is the most ambitious project I’ve ever started, and I’ve made plenty of mistakes over the course of production. One of the biggest mistakes (so far) was that I waited far too long to begin intensive playtesting. The conventional wisdom is that you should get other people to start playing your game as soon as possible, and although I fully intended to take that advice when I began, it quickly fell off my radar. I was so focused on getting the game done that, by the time I realized how badly I needed to start playtesting, there was barely any time left. Luckily, at the last minute I was rescued by a group of friendly local centaurs.
The Research Centaur is a UX/QA testing lab that originated as part of the renowned game studio The Behemoth (of Castle Crashers and Battleblock Theater fame). Nestled in downtown San Diego, The Research Centaur runs a professional game testing operation, and that was exactly what my game needed. I had met a few members of The Research Centaur at local game dev meetups, so I floated the idea of doing some user testing on Darknet. It seemed like a good fit: they wanted to get some experience testing VR games, while I wanted to get feedback and had no time to handle the testing on my own. Once it became clear that our schedules were compatible (and that they understood the limitations of an indie budget), the tests were arranged. I dropped off some Oculus Rifts and went back to development.
Saving time was my original motivation, but it wasn’t the only advantage that The Research Centaur offered. The quality of their tests was also clearly beyond anything I was able to put together myself. This was most obviously reflected in the group of players that they recruited to test Darknet. When I ran my own playtests in the past, I inevitably tested my game with the people who were most available: other game devs, my family and friends, and whoever else happened to be in the room. My testers tended to be a lot less diverse than my intended audience, and at this point in development, most of my potential playtesters had already been exposed to the game. The Research Centaur, on the other hand, went out and recruited a group of fresh players who were much more diverse, both demographically and in their gaming experience and preferences.
The centaurs also ran their tests in a much more professional manner than any test I had ever handled myself. They made sure that the testing experience was consistent for all the players, and they used a survey to make sure that the feedback was being gathered fairly. Another big advantage was the simple fact that I wasn’t present to skew the results. When you’re giving feedback directly to the developer of the game, you’re probably keen to spare the dev’s feelings. Since I wasn’t in the room, the players were free to praise or bash the game however they saw fit.
At the end of the tests, I was presented with three big deliverables. First: a complete set of annotated videos for all the playtest sessions. I watched several of these (partially to evaluate them personally, partially because I just enjoy watching people play my games), but I discovered that the attached notes did a good job of covering most of what I needed to know.
Second, I got the raw survey results from each of the players. The depth of their responses varied a lot from player to player, but they all offered useful information. I also got some gratifying bits of praise (“I totally dug this game”), which was a welcome motivational boost at a busy time.
Lastly, The Research Centaur compiled a final analysis, which took all the contents of the videos and surveys and boiled them down into a set of core findings and recommendations. They did a great job of offering a manageable set of intelligent suggestions (several of which went straight into my to-do list) without making me feel like they were intruding on my territory as game designer.
Overall, I was impressed with the amount of work, skill, and experience that went into the whole process. There were a few technical hiccups (mostly my own fault), but overall, it went very smoothly. I saved a ton of critical time, and I feel certain that the game will turn out better as a result. If I manage my schedule more carefully on future games, I might try to handle the playtesting myself, but if so, I think that my experience of observing the pros has left me better equipped to do it right.
