Why Beta Testing: Why It Matters

I had a discussion recently with one of our clients around what a beta release looks like. The term “beta,” which I’ve seen in many organizations, is a misunderstood term or in some rare instances, an almost taboo term. I’ve found that this is especially true in organizations where the stakeholders of the development team haven’t been exposed to Agile or common software practices. Commonly, what you end up seeing in those organizations, is the beta is boiled down to an appeasement to stakeholders because of a deadline — i.e: we can’t release everything because we don’t have time/money/capacity, so we’ll release this small subset and call it a “beta.”

For us at LeadingAgile, we think about beta releases as a long, controlled experiment where, as a development team, you are trying to understand which of the solutions you are trying to release to market will succeed and drive the right types of behaviors for your users. This experiment is a controlled environment (a beta branch of your code and a beta test environment, for example) and the beta users that you use are a sample population of your general population of users. If you have rigorous/scientific testing standards, you would add analytics to test/measure specific behaviors in your users. Or, in a world where you don’t have analytics, you may use qualitative feedback (we have questions we need answers to, for example) as the method for being able to understand if the solutions you build will be successful for the general population of users.

To use an applicable example – I worked on a video game called EverQuest back in the early 2000s. This was one of the first SAAS products on the market – we were an online massively multiplayer game that had many players playing on the same server. Every 6-9 months, our team would release an expansion pack that players would buy to add more content to the base game. For the final 4-6 weeks of our development cycle, we’d go into Feature-freeze and then promote all of our work from our Dev environment to the Beta environment and server. Then our team would send out beta server invitations to players that we deemed would be a good representation of the players we were aiming to sell our expansion to, and used those players as our test subjects to find out if our expansion pack was going to be successful. Our testing plan included when we were going to test, with who, and what we were trying to get a better understanding of in our testing (i.e.: what unknowns need to become known in testing).

The success of the expansion for our company ultimately came down to how many boxes of the expansion we sold (this is a lagging indicator). Like with any other entertainment medium, such as movies, or literature, the goal is to invoke an emotional response that gets the user engaged with your product. For video games, this boils down to a simple, yet very complex question, “Is the game fun?” So when we brought our control group into our controlled environment to test the content we created, we are trying to answer the question, “Is this expansion fun to our players?”All of our testing for 4-6 weeks is trying to answer multiple facets of the non-quantifiable feeling that is the emotion “fun.” And so we look at each piece of our content in isolation, and we look at all of our content in concert with each other, to see if the content is rewarding, challenging enough, providing an engaging story, has the right pacing, invokes the right emotions at the right times, such as the feeling of accomplishment after beating the big boss, or the feeling of dread going deeper and deeper into a tough dungeon. And then we repetitively test each of these concepts against our content using this controlled set of players, until we have a fairly confident answer that, “Yes, this is fun.”

So when you are planning a beta release, and when you later do a beta test, spend rigorous thought around what types of behaviors are you trying to measure within your users, or simply, what types of answers you are trying to get from the unresolved questions you have around the solutions you want to release to your general population of users. And then your beta cycle should be a process of increasing the confidence level that the solutions you are trying to release will be successful once released. A good litmus test you can use to determine if your teams are correctly planning beta releases and correctly running beta tests is to observe what the team is doing during the beta phase. Is the team spending the majority of their time and effort on supporting the beta test – trying to answer the questions that need to be answered, so that you can launch a successful product – or are they trying to play “catch-up” on all of the features that didn’t get into the beta test? If they are doing the latter, then they really aren’t beta testing.