Karol Frühauf

Sharing My Doubts on Acceptance Criteria

Do you know what acceptance criteria are?

I doubt whether I understand what “acceptance criteria” are. I have no doubts how I interpret the term. In my interpretation, they have nearly nothing to do with requirements and certainly nothing to do with requirements engineering. Looking around in the real world of requirements engineering, I encounter the term being used with a completely different meaning. I doubt that introducing this meaning to the term is needed and useful.

In my simple world of software engineering, there are requirements and there are test cases. The tests of the system against the requirements are often called “acceptance tests”. What matters for acceptance tests, is who carries them out, rather than whether they are against system requirements. The test of a GUI component by a prospective user can be declared an acceptance test. Hence, what is of importance is the consequence of the test result. If it is used to decide whether the subject under test is fit for productive use, then the test is an acceptance test. By the way, the supplier can never take this decision.

What role do acceptance criteria play in my simple world of software engineering?

Acceptance criteria are a kind of counterpart to the contract. The most abstract acceptance criterion would be: are all contractual agreements fulfilled? To arrive at this decision, a checklist is usually developed and used to determine whether the overall criterion is satisfied.
The checklist contains acceptance criteria concerning:

• completeness of delivery (code, documents, training material, etc.),
• completeness and correctness of the functionality and properties of the software (e.g. records of executed tests),
• readiness of the organisation to use the system (e.g. successful training, availability and mastering of operational procedures in the user environment),
• readiness of the organisation to operate the system (e.g. availability and mastering of operational procedures in the operations unit), or
• readiness of the organisation to maintain the system.

This list is not exhaustive. The checks are an investigation into whether the delivered items are o.k. and audits of the organisations that need to be ready.

We thus have a clear distinction between acceptance, acceptance test, and acceptance criteria. Acceptance is the decision whether the contractual obligations have been fulfilled. Acceptance test is a test with the purpose to find out whether the subject under test is fit for use. Its results will be used to decide upon product acceptance if they are subject of acceptance criteria. Acceptance criteria are the conditions that are checked in order to be able to take the decision to accept or reject (not yet accept) a product.

How are acceptance criteria used in the actual world of requirements engineering?

I encounter requirements engineering practices where “acceptance criteria” are a deliverable of the requirements engineer. My observations revealed in this context two uses for acceptance criteria:

1. acceptance criteria define conditions that are not stated in the requirement
2. acceptance criteria reformulate the requirement and mention data to be used to test whether the requirement is satisfied

The first case is rather unfortunate, since the requirement seems incomplete without the acceptance criteria. How should one decide what shall be included in the requirement and what shall be expressed as an acceptance criterion? Why do we need two vehicles for the same thing? What is the benefit of expressing acceptance criteria as an addendum to a specification versus a complete specification of the requirement?

The second case is more adequate, but not quite. In this case, the so-called acceptance criteria are nothing else than the list of test cases the requirements engineer thinks should be executed. Instead of declaring the required coverage of the requirement and letting the test designer determine the test cases, the requirements engineer determines the test cases (and usually fails to declare which coverage criterion was applied). I don’t mind the requirements engineer (partially) doing the job of the tester. But why should we call what he is doing specification of acceptance criteria and not specification of a minimal set of (acceptance) test cases? (Hint: The required coverage is actually a management decision – it has to do with risks and money.)

The second approach resembles Specification by Example, though it has a different order of activities and less rigor in expressing the test cases (the acceptance criteria).

I doubt whether I grasped all the uses of the term “acceptance criteria” out there in the field. I have no doubts that what I did observe (mostly the half-hearted description of “test cases”) does not justify the use of the term “acceptance criteria” instead of test cases.

By the way, it is a shame I could not find the original source of acceptance criteria in the literature, neither in “agile” nor elsewhere. Thus, I have another doubt: acceptance criteria have no theoretical foundation. They are a pragmatic invention that cause more confusion than necessary.

I will be happy if you prove my ignorance or otherwise dispel my doubts.

Karol Frühauf

P.S: Thanks to Oliver Hoeffleur who shared first my doubts and helped me to sharpen them