I doubt whether I understand what “acceptance criteria” are.
I have no doubts how I interpret the term. In my interpretation, they have nearly nothing to do with requirements and certainly nothing to do with requirements engineering. Looking around in the real world of requirements engineering, I encounter the term being used with a completely different meaning. I doubt that introducing this meaning to the term is needed and useful.
In my simple world of software engineering, there are requirements and there are test cases. The tests of the system against the requirements are often called “acceptance tests”. What matters for acceptance tests, is who carries them out, rather than whether they are against system requirements. The test of a GUI component by a prospective user can be declared an acceptance test. Hence, what is of importance is the consequence of the test result. If it is used to decide whether the subject under test is fit for productive use, then the test is an acceptance test. By the way, the supplier can never take this decision.
What role do acceptance criteria play in my simple world of software engineering?
Acceptance criteria are a kind of counterpart to the contract. The most abstract acceptance criterion would be: are all contractual agreements fulfilled? To arrive at this decision, a checklist is usually developed and used to determine whether the overall criterion is satisfied.
The checklist contains acceptance criteria concerning:
completeness of delivery (code, documents, training material, etc.),
completeness and correctness of the functionality and properties of the software (e.g. records of executed tests),
readiness of the organisation to use the system (e.g. successful training, availability and mastering of operational procedures in the user environment),
readiness of the organisation to operate the system (e.g. availability and mastering of operational procedures in the operations unit), or
readiness of the organisation to maintain the system.
This list is not exhaustive. The checks are an investigation into whether the delivered items are o.k. and audits of the organisations that need to be ready.
We thus have a clear distinction between acceptance, acceptance test, and acceptance criteria. Acceptance is the decision whether the contractual obligations have been fulfilled. Acceptance test is a test with the purpose to find out whether the subject under test is fit for use. Its results will be used to decide upon product acceptance if they are subject of acceptance criteria. Acceptance criteria are the conditions that are checked in order to be able to take the decision to accept or reject (not yet accept) a product.
How are acceptance criteria used in the actual world of requirements engineering?
I encounter requirements engineering practices where “acceptance criteria” are a deliverable of the requirements engineer. My observations revealed in this context two uses for acceptance criteria:
acceptance criteria define conditions that are not stated in the requirement
acceptance criteria reformulate the requirement and mention data to be used to test whether the requirement is satisfied
The first case is rather unfortunate, since the requirement seems incomplete without the acceptance criteria. How should one decide what shall be included in the requirement and what shall be expressed as an acceptance criterion? Why do we need two vehicles for the same thing? What is the benefit of expressing acceptance criteria as an addendum to a specification versus a complete specification of the requirement?
The second case is more adequate, but not quite. In this case, the so-called acceptance criteria are nothing else than the list of test cases the requirements engineer thinks should be executed. Instead of declaring the required coverage of the requirement and letting the test designer determine the test cases, the requirements engineer determines the test cases (and usually fails to declare which coverage criterion was applied). I don’t mind the requirements engineer (partially) doing the job of the tester. But why should we call what he is doing specification of acceptance criteria and not specification of a minimal set of (acceptance) test cases? (Hint: The required coverage is actually a management decision – it has to do with risks and money.)
The second approach resembles Specification by Example, though it has a different order of activities and less rigor in expressing the test cases (the acceptance criteria).
I doubt whether I grasped all the uses of the term “acceptance criteria” out there in the field. I have no doubts that what I did observe (mostly the half-hearted description of “test cases”) does not justify the use of the term “acceptance criteria” instead of test cases.
By the way, it is a shame I could not find the original source of acceptance criteria in the literature, neither in “agile” nor elsewhere. Thus, I have another doubt: acceptance criteria have no theoretical foundation. They are a pragmatic invention that cause more confusion than necessary.
I will be happy if you prove my ignorance or otherwise dispel my doubts.
Karol Frühauf
P.S: Thanks to Oliver Hoeffleur who shared first my doubts and helped me to sharpen them
Give Feedback
Your feedback is important to the RE community! Therefore we are keen on receiving your opinion about the RE Magazine, on individual articles or on the RE topic in general.
Suggest missing topic
You are missing articles on a particular topic? Please let us know so we can perhaps publish a matching article on it soon. We appreciate your input very much!
Become an author
You are invited to submit an article! In order to make an initial assessment of the ideas you intend to contribute, we are looking forward to receiving your abstract first.
Comments (4)
From:Suzanne Robertson
Date:07. July 2016
Dear Karol,
Thanks for your response, here are a few thoughts about the questions that you raise.
"So it seems that the Fit Criteria add the necessary preciseness to the description of the thing that should be built. But, why it is not part of the requirement?"
The answer is yes, the Fit Criterion is one of the attributes of an Atomic Requirement. If the Fit Criterion is specified so that Developers and Testers do not need to interpret what it means (in your example about a scale of measurement of colour) then it is true that you do not need a description attribute that says exactly the same thing.
The problem arises because often the source of the requirements are business experts/subject matter experts who do not communicate in precisely measurable terms. Yes, it is the responsibility of the analyst to derive the Fit Criterion by doing requirements analysis and once that is done then maybe the Description attribute is superfluous. However, in practice, I have found that if you lose the connection between the measurable Fit Criterion and the original business description then it is much more difficult to involve the business expert when there are changes/questions/negotiations. The Description is what they recognise and provides the cognitive thread for communicating with them.
Another vital attribute of an Atomic requirement is the Rationale. The Developers and Testers need to have an unambiguous understanding of the Requirement (via the Fit Criterion) so that they know precisely what to build and test. They also need to know why the Requirement is important so that they can make informed choices about which requirements are most vital/valuable to the business.
Along the Requirements Food Chain different people/roles have different interests in creating and consuming different attributes of a requirement. It's the Business Analyst's responsibility to make sure that the overall picture of how the requirements fit into the larger world is not lost along the way. Without this story it is very difficult to respond to change.
Dear Suzanne,
many thanks for your insights.
I hope our discussion inspires the readers of the magazine to contribute.
All the best
Karol
From:Suzanne Robertson
Date:16. June 2016
Hello Karol,
Thanks for your thought provoking piece. I certainly share your experiences where the word "acceptance" is used in a very elastic way.
I agree with your distinction between: Acceptance, Acceptance Test and Acceptance Criteria. However, in the way that we write requirements there is an Acceptance Criterion (or Fit Criterion) for each Atomic Requirement. In other words it is an attribute of the requirement along with other attributes like: ID, Description and Rationale. The Business Analyst is responsible for writing the Fit Criteria. The Developers develop a solution that meets the Fit Criteria. The Testers decide how they will test the solution to discover whether it fits or does not fit the fit criteria.
The lights went on for me on this subject when I read a book - Notes on the Synthesis of Form by Christopher Alexander. To paraphrase Alexander "The fit criteria provide an unambiguous way of dividing all possible solutions into two classes: those that fit or meet the requirement and those that do not fit. In order for this to happen each requirement must be unambiguously measurable - that measurement is the Fit Criterion.
Alexander is an architect and the book I refer to was first published in 1964. Another book he is better known for is "A Pattern Language" that was inspirational for the Design Patterns movement.
many thanks for your feedback and sorry for the delayed reaction.
The distribution of work you suggest is:
BA defines the requirement and the Fit Criterion
Tester defines the test case
I still have doubts whether I understand the need for Fit Criterion. I assume that the Fit Criterion is needed only when the requirement is vague. Then it can define the range of tolerance for “fitting” the requirement. A trivial example:
Requirement: All straight lines shall be displayed in blue colour. Fit Criterion: The lines are displayed in blue colour darker than the summer sky in CH and less dark than ripe plums in September. Test Case: Compare visually each displayed straight line whether it is darker blue than summer sky in CH and less dark blue than a ripe plum bought in the supermarket in September.
Counterexample is an unambiguously specified requirement. Requirement: All straight lines shall be displayed in blue colour with RGB = 0 0 255. Fit Criterion: The lines are displayed in blue colour with RGB = 0 0 255 Test Case: Compare visually each displayed straight line with a line displayed in blue colour with RGB = 0 0 255
In the counterexample I can’t see the benefit of having a Fit Criterion defined, because the requirement is clearly defined. So it seems that the Fit Criteria add the necessary preciseness to the description of the thing that should be built. But, why it is not part of the requirement?
The architects like Alexander are facing most likely much vaguer requirements than I would like to see in specifications of software.
All the best
Karol Frühauf
Author's profile
Karol Frühauf
Karol Frühauf is co-founder of INFOGEM AG in Switzerland, since 1987 consulting in the field of software project and quality management (http://www.infogem.ch). He worked 12 years for BBC Brown Boveri & Cie and helped since 1987 many companies to improve their processes and products. He co-authored two books and is a frequent speaker, tutor and teacher in the field of software engineering. His main interests are test and quality management and requirements engineering.
He initiated and directs the "Bridge Guard Art / Science Residence Centre" in Štúrovo, Slovakia (http://www.bridgeguard.org).
Comments (4)
Dear Karol,
Thanks for your response, here are a few thoughts about the questions that you raise.
"So it seems that the Fit Criteria add the necessary preciseness to the description of the thing that should be built. But, why it is not part of the requirement?"
The answer is yes, the Fit Criterion is one of the attributes of an Atomic Requirement. If the Fit Criterion is specified so that Developers and Testers do not need to interpret what it means (in your example about a scale of measurement of colour) then it is true that you do not need a description attribute that says exactly the same thing.
The problem arises because often the source of the requirements are business experts/subject matter experts who do not communicate in precisely measurable terms. Yes, it is the responsibility of the analyst to derive the Fit Criterion by doing requirements analysis and once that is done then maybe the Description attribute is superfluous. However, in practice, I have found that if you lose the connection between the measurable Fit Criterion and the original business description then it is much more difficult to involve the business expert when there are changes/questions/negotiations. The Description is what they recognise and provides the cognitive thread for communicating with them.
Another vital attribute of an Atomic requirement is the Rationale. The Developers and Testers need to have an unambiguous understanding of the Requirement (via the Fit Criterion) so that they know precisely what to build and test. They also need to know why the Requirement is important so that they can make informed choices about which requirements are most vital/valuable to the business.
Along the Requirements Food Chain different people/roles have different interests in creating and consuming different attributes of a requirement. It's the Business Analyst's responsibility to make sure that the overall picture of how the requirements fit into the larger world is not lost along the way. Without this story it is very difficult to respond to change.
Thanks for an interesting discussion.
Kind Regards
Suzanne
Dear Suzanne,
many thanks for your insights.
I hope our discussion inspires the readers of the magazine to contribute.
All the best
Karol
Hello Karol,
Thanks for your thought provoking piece. I certainly share your experiences where the word "acceptance" is used in a very elastic way.
I agree with your distinction between: Acceptance, Acceptance Test and Acceptance Criteria. However, in the way that we write requirements there is an Acceptance Criterion (or Fit Criterion) for each Atomic Requirement. In other words it is an attribute of the requirement along with other attributes like: ID, Description and Rationale. The Business Analyst is responsible for writing the Fit Criteria. The Developers develop a solution that meets the Fit Criteria. The Testers decide how they will test the solution to discover whether it fits or does not fit the fit criteria.
The lights went on for me on this subject when I read a book - Notes on the Synthesis of Form by Christopher Alexander. To paraphrase Alexander "The fit criteria provide an unambiguous way of dividing all possible solutions into two classes: those that fit or meet the requirement and those that do not fit. In order for this to happen each requirement must be unambiguously measurable - that measurement is the Fit Criterion.
Alexander is an architect and the book I refer to was first published in 1964. Another book he is better known for is "A Pattern Language" that was inspirational for the Design Patterns movement.
Thanks for opening up this discussion.
Kindest Regards
Suzanne Robertson
Dear Suzanne,
many thanks for your feedback and sorry for the delayed reaction.
The distribution of work you suggest is:
I still have doubts whether I understand the need for Fit Criterion. I assume that the Fit Criterion is needed only when the requirement is vague. Then it can define the range of tolerance for “fitting” the requirement. A trivial example:
Requirement: All straight lines shall be displayed in blue colour.
Fit Criterion: The lines are displayed in blue colour darker than the summer sky in CH and less dark than ripe plums in September.
Test Case: Compare visually each displayed straight line whether it is darker blue than summer sky in CH and less dark blue than a ripe plum bought in the supermarket in September.
Counterexample is an unambiguously specified requirement.
Requirement: All straight lines shall be displayed in blue colour with RGB = 0 0 255.
Fit Criterion: The lines are displayed in blue colour with RGB = 0 0 255
Test Case: Compare visually each displayed straight line with a line displayed in blue colour with RGB = 0 0 255
In the counterexample I can’t see the benefit of having a Fit Criterion defined, because the requirement is clearly defined. So it seems that the Fit Criteria add the necessary preciseness to the description of the thing that should be built. But, why it is not part of the requirement?
The architects like Alexander are facing most likely much vaguer requirements than I would like to see in specifications of software.
All the best
Karol Frühauf