Acceptance Criteria
Did We Build the Right Product? And, Did We Build the Product Right?
Acceptance Criteria are important. Unfortunately, we often overlook or undervalue it as an aspect of the iterative planning process. It is super important because projects succeed or fail based on the ability of the team to meet their customers documented and perceived acceptance criteria. When we clearly define the criteria up front, we avoid surprises at the end of a sprint, or release, and ensure a higher level of customer satisfaction. In other words we’re able to answer these two important questions: Did we build the right product? And, Did we build the product right?
What are Acceptance Criteria?
They are the conditions that a software product must satisfy to be accepted by a user, customer, or in the case of system level functionality, the consuming system.
Acceptance Criteria are a set of statements, each with a clear pass/fail result, that specify both functional and non-functional requirements, and are applicable at the Epic, Feature, and Story Level. Acceptance criteria constitute our “Definition of Done”, and by done I mean well done.
We’re not talking about horseshoes here, and there is no partial acceptance: either the acceptance criteria is met or it is not.
When to define our Acceptance Criteria?
A trap that I encourage my teams to avoid is writing acceptance criteria after development has started. This leads to merely verifying that the functionality built works rather than verifying that the functionality meets user needs and expectations. If we write and review the criteria before implementation begins, we’re more likely to capture the customer intent rather than the development reality.
What makes good Acceptance Criteria?
Acceptance criteria define when a work item is complete and working as expected. Express criteria clearly, in simple language the customer would use, without ambiguity regarding the expected outcome. This sets our testers up for success, since they will be taking our criteria and translating them into automated test cases to run as part of our continuous integration build.
What. Not how.
Another trap that I coach my teams to avoid is the how trap. Criteria should state intent, but not a solution. (e.g., “User can approve or reject an invoice” rather than “User can click a checkbox to approve an invoice”). The criteria should be independent of the implementation, and discuss WHAT to expect, and not HOW to implement the functionality.
Formats
The Given/When/Then format is helpful way to specify criteria:
Given some precondition When I do some action Then I expect some result
When writing acceptance criteria in this format, it provides a consistent structure. Additionally, it helps testers determine when to begin and end testing for that specific work item.
Sometimes it’s difficult to construct criteria using the given, when, then, format. Particularly when dealing with system level user stories. In those cases, I’ve found that using a verification checklist works well.
Another advantage to verification checklists is that they are also simple to individually mark as complete as we implement functionality.
Comments (26)
Christine Schubert
I see challenges in helping non-software development teams (business teams) to understand the meaning of acceptance criteria. Checklists are useful to illustrate the end result. I sometimes recommend that teams try backing into a user story. So begin with listing the items that are desired first. It can help business teams new to agile understand the purpose of the user story and what the acceptance criteria will look like.
Bob Tarne
I also think acceptance criteria can be written to early. I’ve had clients try to write all acceptance criteria at the beginning of the project when user stories are first capture. I advise them to have acceptance criteria ready to review at the iteration planning meeting for the iteration the stories are going to be developed
Justin Mancinelli
>>> “A user can approve or reject an invoice” rather than “A user can click a checkbox to approve an invoice” <<<
Isn't a checkbox user-facing and therefore something a user may expect?
When I think of "HOW the functionality will be implemented", I think of things which are not user-facing such as "The approved or rejected invoice will be stored in a Postgres table".
That said, deciding on a checkbox is probably too small a detail to include in acceptance criteria unless user testing has already shown this is what users expect. Otherwise, I think small details like this can be left out of acceptance criteria and, after the feature is implemented, the users can give feedback such as "I really wish this were a simple checkbox" then you can change it in the next sprint.
Michael
I think the point is that when outlining a feature, it’s not typically necessary to describe the implementation details, but rather the set of criteria that need to be met in order to deliver business value. The same business value of approving or rejecting an invoice can be achieved with a checkbox, or with a dropdown selection, or even with a swipe gesture. Regardless of which method is implemented, the user should still be able to achieve their goal of approving or rejecting an invoice.
Quite often, the implementation details may be implied as part of maintaining a consistent UX across the application, in which case the development team will need to adhere to such details when implementing ANY story. But unless a specific implementation is critical to delivering the business value (i.e. the same value for the end user couldn’t be achieved with a different implementation for the same functionality), it’s often not necessary to mention the “how”.
Mike Hillman
If you are not specific then how does QA know what to test?
Ray Edgar
This is not a specification. They collaborate with PO/BA/Devs. That is the point of a story in the first place.
Veronica
I agree — in my experience, acceptance criteria are interchangeable with test cases and tend to need to be very specific, down to the user INTERACTION — which does need to list things like “this should be a dropbox”. In an ideal world, I would be able to provide guidance to design that says “user needs to be able to SELECT from a list of options”, and that may or may not translate to a dropdown. In practice, design, engineering, and test almost always demand a lot more detail. Also for analytics, for example, I’ve been asked to specify what event should fire — certainly not user facing but very important detail for getting analytics requirements right (and testable).
David Murphy
I find that acceptance criteria at the level Mike suggests is enough at the story level and then flesh out the details as the story evolves into delivery with specific features (eg the checkbox v some other method. It does depend though on how the UI evolves and when you know the detail of it, but it is key to focus on the business value rather than detailed features early in the process.
Daria Johnson
Thank you for the article. Really acceptance criteria must be created before the beginning of development. It doesn’t make sense to do vice versa.
Keith Collyer
I have been referred to this post as providing “the agile view of acceptance criteria” by a colleague whose team is (mis)using acceptance criteria to provide another level of detail of requirements. It seems to me that the post and the discussion is missing a couple of points.
1. Acceptance criteria are generally more specific than requirements, but they are not another level of detail. That might appear contradictory, but what it means is that criteria should refer to an instance (or instances) of the requirement, whereas the requirement itself is more general. Consider a tax preparation program. Probably the most important requirement is that it correctly calculates tax due given incomes and outgoings of various sorts. Clearly, you are not going to be able to test all possible combinations, so your acceptance criteria will specify particular values, or how to generate a valid set of random values. More specific, but not another level of detail.
2. Sometimes the acceptance criteria ARE simply a restatement of the requirements. The example of approving or rejecting an invoice is one such. The requirement might be “as a finance officer I want to be able to accept an invoice”, the acceptance criteria might be “when I perform the accept action, the invoice is accepted (check by examining the record for the invoice)”. No more detail really.
3. On the point of whether the tester needs to know whether to click a checkbox, that is part of the definition of the test case, not of the criterion for acceptance.
Keith, thank you for adding your points. I’m sure our readers will find them helpful.
Keith Collyer
I should have said that I do agree with the main thrust of the article.
Andy
Typo alert: “When to define are acceptance criteria?”.
“are” should be “our”.
Fixed.
Mateo Diaz Granados
Thanks for the article.
I noticed you have a typo in the first sentence.
Acceptance criteria are an important. Unfortunately, we often overlook or undervalue it as an …
Acceptance criteria are an important?
Javi
Who should define/write the acceptance criteria?
David Murphy
Teams I have led have followed a collaborative approach where the QA and the PO work together to decide and evaluate. Developers may also be part of this and also BA ( if the project has BAs). The PO may involve other business users if needed.
Where UIs are concerned also need to think about usability and accessibility and this may require some larger up front work to define the design principles for the project and ensure affected people are involved.
Vijayanand
Thank You. This is really helpful.
Jaroslavas
good idea
Will
“The criteria should be independent of the implementation, and discuss WHAT to expect, and not HOW to implement the functionality.”
In reference to this sentence, I am wondering what step of the agile process defines “how to implement the functionality?” I started investigating with stories but that wasn’t it, then I got to acceptance criteria and thought “aha, this must be where you define how to implement” but then…no, it seems this is not where that is done. I understand Agile does not rely on specifications but surely in order to implement functionality, someone, at some point, needs to define how specifically that will be done. I’m just wondering if there is a name for this step in the Agile process and if it’s supposed to be documented and if so if there is a name for that documentation.
Thanks so much, this article was definitely helpful in getting me closer to to answering my questions on the Agile process and workflow.
Nate
My understanding of Agile is that it focuses mainly on User Value and Business Value and thus User Stories and Features will define the What only. The How is mostly up to the Dev Team to figure out, and this can be done by a discussion between Dev and UX/UI. Most often this will be discussed prior to the Development phase and the team will know more or less what it has to develop and how it has to be developed. But a major point of Agile is that the Dev Team is free to implement a functionality in the way it seems the best to satisfy users (checkbox, drop-down, etc), AC are here to ensure the What is correctly delivered. If the how needs to be changed then it can be done in another iteration.
OBloodyHell
I would assert Nate is correct.
Agile emphasizes NOT defining HOW something is to be done except in very broad terms, which are generally understood by the dev and testing team:
We’re going to use Github, along with Oracle as the DB, and use Confluence tools for the other main actions. Our APIs will be set up to work with Postman, using JSON. We will define the interface with (pick your tool), and use Balsamiq for the wireframes. Testing will be done using Browser Stack for cross-browser testing.
None of these need be defined in the AC, as they are understood as applying across the whole project. Devs, BAs, and QA all know this, and thus will plan their descriptions and coding using those tools.
The nitty gritty of HOW may still be defined more exactingly in regards to making a consistent UI, and this involves more than likely the wireframe definitions and even direct UI mockups created by a graphic designer on the team. There may be specs (the backgrounds will be color #F8FCFA, with text color #E6FC00 — yes, those are totally random hex codes) for specific style things but on the whole, the implementation beyond that is up to the dev, with the power to suggest Other Great Ideas for anything that they think may be outside the expected style envelope.
But the code-writing and most of the implementation details are still much more up to the dev for their story.
Hopefully, you also have a team that works together to code to a “house style” for most things, so everyone can grasp everyone else’ code.
B
Question – Do you feel that acceptance criteria is absolutely mandatory? my thought would be “no” and to back that up I would say that if the vision of the user story is absolutely clear and my team is high performing and understands the product owners request very clearly than acceptance criteria can be left out. I have got push back from people that were inexperienced scrum masters and they said that is unacceptable. But acceptance criteria is just business level requirements needed to define that the story is complete when all those needs are met.
OBloodyHell
I would thoroughly disagree. AC is critical to forcing the BA/PO, etal to define exactly what they want, and to clarify to QA and the Devs what it is, exactly, that the PO/BA etal expect the story to do.
You can use either the detailed list method or the BDD GWT means, or even both, but you should always have one or the other at a minimum, AND every single one of the AC should be fully binary — either it passes or fails, it works or it does not.
John T.
You stated “Acceptance criteria constitute our “Definition of Done.” Not sure if this is a typo or really what you meant. I would make it criteria for Definition of Ready. The user story isn’t ready to be estimated until AC has been specified.
OBloodyHell
No, AC define the Definition of Done for that user story. Providing them defines part of the “Definition of Ready”, from the Dev point of view.
But they define “Done” because, when the software does what they say it should, that individual story is complete and can be pushed into the “completed” backlog, and not before.