Getting Testing Involved

Testing anti-pattern

There is a common anti-pattern I consistently run into involving testing. In this anti-pattern testing is about finding technical defects near the end of a project. Testers view their job as preventing a buggy product from being shipped to the customer. To accomplish this, testers define test cases from the requirements documents. Since requirements are going to change during the project, the testers need to wait until near the end of the project to produce their tests.

What’s wrong with this?

This model doesn’t work. The requirements documents don’t tell a perfect story in the first place. And the documents never stay in perfect alignment. Sometimes the tester interprets the requirement differently than the developer. Inevitably, development delivers something that doesn’t match what testers expect and they file a defect. To avoid defects, the developers have to anticipate everything that might be meant by the requirements. A result is that developers significantly overbuild or over-engineer the product.  The bigger and more “dynamic” the product is, the bigger the testing effort.

A result of the anti-pattern is that the cost of producing working, tested software is significantly increased. We are less predictable and we don’t really know where we are until very late in the project.  Because we test late we find out late in projects that we aren’t meeting the schedule, or scope or cost. We end up in the blame game trying to decide if defects are a requirements problem versus a development problem versus a test problem. There is vicious cycle that produces a culture of low trust and a lack of collaboration and coordination. This is very common – I see this low collaboration environment every day.

Define the acceptance criteria before you build

Test (n): A procedure for critical evaluation; a means of determining the presence, quality, or truth of something.

Testing isn’t about finding bugs. And it isn’t just about evaluating.  A critical activity in testing is defining the criteria for the evaluation – we will call these the acceptance criteria. As we pointed out above it isn’t effective when someone in a silo sets the criteria for evaluation after the product has been built. We want to define the acceptance criteria before we build the thing. This doesn’t incur more cost to the testers. It is no more initial work writing the test cases whether you do them before or after.

Define acceptance criteria as a whole team

Defining them before isn’t enough. Everyone needs to have a shared understanding of the outcomes. There is a lot of benefit in the discussion around defining acceptance criteria. When we agree on the tests upfront developers can build the smallest and simplest thing that will solve the problem – not overbuild trying to anticipate everything that requirements could possibly mean.  When we agree on how we will evaluate the product before we build it we will have fewer defects so there is less rework.

Progressively elaborate acceptance criteria

Defining the acceptance criteria before you build something isn’t big up front design.  We want to do whole-team just-in-time detailed design. In Scrum, this happens while grooming the backlog and in Sprint Planning. In our Enterprise Agile approach, we break the product into a series of releases in product road-mapping. We break releases into features during release planning. We break features into stories and we elaborate the stories further before Sprint Planning.

As we progressively elaborate the requirements – we progressively elaborate acceptance criteria. In fact, acceptance criteria are an effective way to express business rules at the story level. We also find that non-functional requirements, compliance requirements, and operational requirements can be expressed as acceptance criteria at the feature level so we can ensure we are finishing all aspects of the product as the project progresses.

A better testing strategy

We’re on the way to solving the testing problem. When the acceptance tests are elaborated jointly the testers, analysts and developers don’t have different understandings. When we elaborate them before we build developers can build the smallest and simplest thing that will solve the problem. When we elaborate acceptance criteria at the same time we elaborate requirements we don’t have as much change control required and we actually get better requirements to communicate. This strategy can have a significant impact on reducing the cost of testing – and the time to release of the product.