This week the Quality Assurance cycle ends for our first batch of testers. Overall it was a success! In the beginning, we had a team of about twenty inexperienced QA testers and over a few weeks and a handful of training sessions they managed to write about 325+ Bug Reports. In addition to this, there were hundreds of confirmations and reports filed for Regression Testing and attempted to reproduce existing bug reports. All of this spanning a steady roll-out of new content each week, and the testers juggling outside work and study.
What went right?
We had a lot of success regarding the team having excellent collaboration skills. Especially considering everyone involved was working remotely. We had a few key testers step up to help direct QA Specialists who felt lost or weren’t sure where to focus their energies. Several other testers developed a knack for developing creative workarounds to game breaking bugs, or finding ways to improve the bug submission system!
The Adventure Pals team also took time out of pure bug fixing to toss us some primitive QA tools. The tools enhanced the speed at which bugs could be identified and allowed us to pinpoint the cause of bugs more quickly.
I was earnestly blown away by the amount of passion and energy they poured into this. Quality Assurance is at best intensive problem solving, and at worst grinding. This rag-tag group of testers did a phenomenal job.
What went wrong?
Due to the staggered introduction of testers, I had a very difficult time as QA Lead keeping track of which individuals were in need of one on one training. In addition to this, I simply struggled at times to keep on top of reviewing and organizing bug reports at the rate they were pouring in. The biggest time sink was running the testers through one on one training sessions. These workshops demanded about 20-40 minutes of hand-holding and walking through the already written documentation. In hindsight, it would have been far more appropriate to record introductory videos and tutorials for different methods of testing and bug reporting.
One big challenge we faced were the communication issues with the rest of the Adventure Pals team. There was often a fair amount of uncertainty surrounding when new builds were distributed to the testers. The lack of a changelog also led to confusion as we transitioned to new versions of the game. Some of the content that was expected to be playable was locked or required specific methods to access.
And finally, while I’m impressed by much of the work of our testers it was challenging for them to stay on top of their studies while dedicating time to testing. I made it very clear that study and academics come first. Unfortunately, this left us at about 10-15 testers on average participating per week. The sporadic participation also led to drops in morale when testers felt they had failed the team or worse, begun to fall behind in their studies.
What is next?
Our playtesting uncovered a lot of critical bugs for the game. Only in the final week did the game reach a point where it can be played all the way through. In my opinion, the game could use another month or two of intensive QA to prep the game for launch. Post-release the game should get a lighter group of testers to help reproduce and identify bugs and issues identified by the community.
At least now, we have a cleaner method of organizing large quantities of testers. We also have learned from a lot of the mistakes made throughout this quality assurance cycle. I hope we get the opportunity for further testing moving forward. The experience has greatly improved the quality of the game, the testers, and my management abilities. Thanks for reading!