Options:

Thorsten von Ramsch

Classical requirements and test analysis a discontinued model?

Endeavours to improve the situation are finally rewarded


In times of increasing "agilization", the question arises as to whether classic requirements engineering and test analysis are still relevant at all. Since the IT of banks and insurance companies is subject to strong control, development here is usually carried out according to the V-model. This article deals with the interaction of requirements engineering and test analysis in such a classical approach and highlights the importance of observing set standards such as ISO/IEC/IEEE 29148 in conjunction with ISO/IEC/IEEE 29119. Due to the importance of this topic, the article demands unconditional knowledge of the above-mentioned standards for all project participants and makes it clear that comprehensive training, error frequency and subsequent costs can be reduced. To support this thesis, both positive and negative practical examples are given from project work at a large commercial bank, a large insurance company and a large logistic provider and these are contrasted with studies of the frequency and remediation costs of software errors caused by faulty requirements specification. The article ends with the demand for comprehensive training of all project participants (business analysts from specialist departments, functional analysts from IT, (system) integration testers from IT, acceptance testers from the specialist department and possibly also the developers) who have to deal with requirements in the areas of requirements engineering and test analysis in order to take into account the high priority of this topic. Requirements engineers must be trained in test analysis in order to be sensitized to the topic of "testable requirements"; test analysts must be trained in requirements engineering, because those who can write requirements can also read and understand them and derive meaningful test cases from them.

Agile development

"Agile" is on everyone's lips these days and articles about agile procedures such as Scrum, Kanban, XP etc. are successively replacing articles about "classic" procedures in the technical literature. Even institutions such as the ISTQB and the IREB are taking this development into account and, due to the increase in the number of agile projects, have developed curricula for the training of agile testers[1] and Requirements[2] Engineers. The reader is now justified in asking whether an article on "classical methods" still seems at all contemporary. One can say it right away: it does. Another question is whether agile methods can be combined with classic methods: they can.

Classical (and hybrid or agile) methods

In addition to the medical and avionics sectors, the financial sector is under particular scrutiny. In Germany, the financial sector, especially in banks and insurance companies, is highly regulated by the Federal Financial Supervisory Authority (Bundesanstalt für Finanzdienstleistungsaufsicht - BaFin) with their regulatory requirements on IT: "Bankaufsichtliche Anforderungen an die IT" (BAIT) and the "Versicherungsaufsichtliche Anforderungen an die IT" (VAIT), not only because of its security-critical multi-systems. For this reason, special requirements arise when adapting existing systems and applications or developing new ones, which affect the entire project or program structure. Due to the regulatory framework, the V-model is usually still used for such adaptations and new developments.

Figure 1: V-model

In the classical V-model, the requirements documents, as well as the design documents, test plans, test cases and other artefacts, go through a review and approval process in which existing uncertainties are discussed and eliminated in the rework. According to pure doctrine, the work of the test analysts already begins with the review (static tests: reviews, walkthroughs, inspection)[3] of the business department's requirements documents; the annotations, as mentioned above should then find their way into the documents to be created. This is how it continues in all subsequent phases until the project is completed.

For some time now, hybrid models have also enjoyed increasing popularity in the banking and finance industry. They often consist of the V-model in their basic structure, whereby the development part is "agilized" and operated according to Scrum, for example. These hybrid models are increasingly used in situations where development is not carried out by in-house development departments, but where development is outsourced to external service providers, or where software is developed and/or purchased by vendors who themselves choose an agile approach to the development or adaptation of their software.

Figure 2: V-model based hybrid model

In the case of a hybrid development approach, special attention must be paid to translating the requirements of the incoming documents into the documents needed for the agile process, because in the case of Scrum for example, the requirements are rewritten into user stories, which in turn should have such a reduced complexity that a user story can be processed within a sprint and thus an executable increment can be created. This presents the product owner with the challenge of cutting the stories in such a way that, on the one hand, the required functions can be implemented in accordance with the requirements and, on the other hand, the stories are "small" enough to be created in one sprint and are also extensive enough to be able to implement a meaningful and testable function in this increment. In addition, after all sprints at the end of the development period, exactly those functions must have been implemented that the business unit has commissioned in its business requirements.

Another point that is not primarily at issue here, but should nevertheless be mentioned in passing, is the input quality of the delivered software for the SIT test. In the V-model, the organisation prescribes when the test maturity for the next higher test level is reached. This is then captured in the organisation's own test strategy and forms the guideline for all test levels present in the development process. For example, it specifies that the unit tests are created, executed and documented with a C₂c coverage. If, for example, Scrum is chosen as the (agile) process model for development within a V-model (hybrid model), these organisational specifications are just as binding for the Scrum teams and must also be explicitly included in the Definition of Done[4] (this applies not only to hybrid models, but also if an organisation uses Scrum as its predominant process model, but also specifies coding conventions or quality criteria). It becomes more difficult here when the development vendors cannot be seen as part of the development process, but develop in an agile manner themselves as service providers. In this case, the vendor delivers a trade and is therefore no longer subject to the control of the development organisation. Here, "standard software" is integrated at once, which then in turn has to be checked for its functionality within the specific multi-system. Here it is not possible to fall back on the test documentation of the unit tests, as it simply does not exist. In this case, one can try to request test documentation from the vendors, but this is usually so generic that little or nothing can be done with it and the test maturity of the development cannot be clearly determined on this basis. This, in turn, creates additional effort, since integrative tests must sometimes be extended by tests that should have been executed in the unit test in order to be able to assess the test maturity of the delivered increment.

In reality, however, a somewhat modified form is observed due to time and cost pressures. Often, test analysts are not integrated into the project and program structure until the requirements documents have been approved and implementation has already begun. Yet the test analysts from the business unit who write the test cases for the acceptance test are the very ones who also work with the application, so they could already provide important information for the commissioned changes during the concept creation phase. This phenomenon can also be observed in the upstream test stages. And here we inevitably come to the weak point that such a scaled approach entails. It is the supposed cost reduction. Of course, there are fewer costs upfront if the authors of requirements documents are not trained beforehand and the quality assurance experts are only integrated into the project when the dynamic tests begin. However, this is a fallacy. Such procedures inevitably lead to rework or change requests, which in turn have a negative impact on the pre-calculated project budget as well as the pre-calculated completion date.

A real example (from a project of a large commercial bank): "Optimally on dd.mm.yyyy, or later, the checking logic should be carried out with X.X%. The value should be administrable for future changes." illustrates the above point quite well.

Figure 3: Requirements specification in a real project context [5]

Here the use of the sentence template is (unconsciously) dispensed with, which means considerable degrees of freedom for both development and quality assurance. This means that the expectations and the reaction of the software can differ considerably. No specific date is mentioned, but only that on date X this checking logic can be carried out. Of course, requirement specification should not deprive development of all the degrees of freedom they need to implement these requirements in the code, but it must at least be defined in advance to what extent degrees of freedom are granted. From the author's point of view, the degrees of freedom granted here are too high to be able to derive binding instructions for development and testing. For example, for an implementation that meets the requirements, the requirement should be formulated as follows: "Optimally on dd.mm.yyyy, latest on dd.mm.yyyy, the checking logic has to be carried out with X.X%. The value must be administrable for future changes.". These small changes do not unduly reduce the degrees of freedom for development, but the requirement still sets clear guard rails that development must follow.

Another example is the continuous changing of requirements (despite approval and finalization of the documents and already started development work). During development, further adaptation requests and additional changes found their way into already released requirements documents, so that development had to be stopped: tests already created had to be adapted and the project plan had to be adjusted.

Figure 4: Requirements adjustments within a short time

In this real-life example, the requirements were to be available in the accepted state at the end of the year. These requirements were made available to the development vendor, who then started work. As can be seen from this example, there were then various adjustments, all of which were marked as approved, until finally the final version (v9!) was available in the month in which the development was to be completed according to the project plan. If you look at the time specifications between v8 and v9, you will notice that the validity of v8 was just 2 hours, which in a program context has a significant impact, not only on the project of the application in question, but also on all projects of the supplying and consuming applications. Those changes in turn led to changes in the functional specification, changes in the low-level design, the development, new creation or adaptation of the test cases already created, which in turn led to additional work and effort in the development, test analysis, test execution which was not planned, as well as a delay in the planned go-live and thus to a considerable increase in the project and program costs.

Figure 5: Number of software errors due to unclear requirements

The real number of 1,791 errors shown above, which resulted purely from improperly defined requirements, led in this project context to additional effort in re-specification, re-design, re-development and re-testing. This observation is not only confirmed by the author, but also supported by other professional articles[6]. In this example, costs increases of approximately 1,225,000 euros through re-specification, re-design, re-development and re-testing. These points make it clear how important, especially in the context of a security-critical multi-system, the precise requirements analysis, requirements capture and the final acceptance and release of these requirements are.

Another example comes from a large German insurance company that wanted to, or has, partially become "agile" in development in the life insurance sector. Here, the V-model was also used as a basis, where the business requirements were also specified by the department, but an agile approach similar to Scrum was chosen from the functional analysis onwards. The approximate procedure was as follows:

Figure 6: Semi-agile Model of a German Insurance Company

The impression that emerged here is that an agile process model was not used because it was suitable, but rather to jump on the agile bandwagon. The weak point of the procedure was that an attempt was made to put an agile cap on an existing structure. Long-standing project managers with leadership functions could not cope with the fact that they suddenly no longer had the reins in their hands, but that the development team would henceforth determine what could be implemented in a certain amount of time. In this case, it was decided that the requirements analysis, development and testing should be carried out in a two-week sprint. The article does not want to go into the misjudgment of the success of such a procedure, but it seems clear to the attentive reader that with such a procedure the sprint goal cannot be achieved at all. Apart from that, the backlog only contained vague "ideas", the business requirements were not completely taken over into the backlog, the functional analysts were supposed to transform them into user stories during the sprint, development was supposed to implement them, and quality assurance was supposed to test them. Here it is immediately apparent that this procedure cannot be effective, as a mini-waterfall has been established under the guise of agility. Even if one wants to have it this way, it should be clear that at least the functional analysts are able to describe requirements in such a way that they are generally understandable, and code is technically convertible and testable. If this is not the case, it leads to unwanted iterations, which in turn lead to analysis, development and testing efforts.

The last example describes the attempt of a large German logistics service provider to establish a confidential, binding and reliable communication option electronically. The project started as a V-model development, but in 2012 an attempt was made to make the development agile (using Scrum) in order to be able to react more quickly to market changes and results of customer surveys regarding manageability, requests for additional services, design, and the like. For this purpose, a separate GmbH was founded, which was outsourced but subordinated to the group. Agile development was now carried out here, but contrary to the Scrum principle, the development approach was not aligned with that of the commissioning organization[7]. This led to the fact that (according to the comments in the code, and even as per the C₂c coverage in developer tests) the stipulations of the development process were consistently disregarded. Now one may ask whether this disregard is really such a big problem. Due to the fact that electronic communication was not free of charge for the consumer, an account was set up to which the customer either deposited money or where a direct debit authorization was required, which in turn required a "small banking license". This banking license then led to the communication product being regulated by BAFIN and thus subject to control. In addition, the decisions to use development and test tools that were not considered audit-proof and therefore did not comply with the specifications of the parent organization did not help this process. Furthermore, decisions were made during the development process in such a way that the requirements of the organization, i.e. requirements in the form of a style guide and in the form of predefined security requirements, were not taken into account and the use of predefined fonts and RAL color codes was consciously or unconsciously ignored, and a "stateless" application was chosen. This led to the development of several sprints being put to the test by the organization, not being accepted by the organization (despite acceptance in the individual sprints) and, as a result, rework having to be carried out in three sprints (6 weeks). This might have been avoided if the group requirements had been read correctly, taken into account when creating the user stories and implemented in the development. With regard to the total costs, we can only speculate here, but the adaptation costs amounted to approximately 240,000 euros just for the additional development work. The costs of the time delay incurred cannot be quantified here.

Especially in this context it becomes clear that requirements engineering (RE) is not practised for its own sake, as it is also postulated by the International Requirements Engineering Board (IREB), but must be given special attention in the development process. Based on the above example, it can already be guessed that the requirements of the functional area represent both the basis for the work of the functional analysts and also provide the basis for the user acceptance tests to be carried out at the end of the development process. If these requirements are insufficiently specified, this already leads to considerable additional work in the static tests that take place before development (document tests such as review, walkthrough, etc.), which are necessary so that the functional analysts, for their part, can write high-quality documents. If these discrepancies are not discovered in the review and approval process, e.g. due to the absence of test analysts from the business department, they find their way into the design and programming of the software. Under certain circumstances, these "inaccuracies" continue into the acceptance test, or even worse, into the go-live phase. These errors can then cause such serious errors that even result in recourse claims by customers against the bank or insurance company, which may be fought out in court. These recourse claims then inevitably lead to damage to the image or loss of image, which the institution in question would certainly not want to experience, as financial losses are also to be feared as a result.

There are various articles in the literature that deal with the merging of agile methods[8] with classical[9] models[10] and how well the integration of agile development components into the V-model works, but here the focus is more on the process level than on the training or skills of the business or functional analysts. A point that cannot be generally applied by Kriescher and Markgraf is the "evolutionary collection of requirements with the involvement of the customer"[11], which in agile procedures requires the involvement of the client and forces him to cooperate. This is certainly not possible in the context of banks or insurance companies, since the clients of regulatory changes are not the banks or insurance companies themselves, but rather these changes are initialised by BaFin. In such a situation, the bank or insurance company will certainly not contact BaFin to discuss the required software changes and offer to participate in the requirements and development process. Here, there is no discussion, but the required changes are implemented. In this case, BaFin represents the client and does not allow for discussion, regardless of whether it is an agile approach or not. Canditt, Rauh and Wittmann also disregard this point, although they refer to "close and trusting cooperation with the client"[12] and do not address "dictatorial" points that arise in so far as changes to the law should occur.

This should make it clear that the discussion here is not primarily about whether the V-model, hybrid approaches or agile development are the right choice here, but rather about the fact that with all methods, the correct formulation of the requirements is decisive for the success of the program or the project - regardless of whether these are highly regulated sectors or not. But just to resume the title of the article, classical requirements engineering and classical test analysis is a discontinued model. Especially here, it should be clear that the approaches are not competing models and mutually exclusive: that this is not black and white. Rather, classic and agile methods are compatible with each other, especially with regard to the development of hybrid approaches. Nevertheless, there are organizations that prefer one approach or the other. However, one thing remains the same for all of them: the project rises and falls with the way the requirements are formulated. If requirements are formulated properly, there will be fewer inquiries in the design and development phase, fewer errors in the test phase, and fewer productive errors will be reported, which will then lead to rework in the form of post-project changes.

This article, however, deals with requirements in sectors that are not debatable. This also means that the requested changes must be implemented, regardless of which model is used for development. And it is precisely here that the emphasis is on the statement that organisations should not save in the wrong place, as what is supposedly saved is a boomerang. When it comes to regulatory changes to banking or insurance software, the client cannot afford to have any weak points and for this reason those responsible for requirements development must be able to overview the scope of the changes, put them into words and, even better, formulate them in such a way that both developers and test analysts can implement them. For this reason, the training of the persons developing the requirements should not be skimped on under any circumstances. If investments were made in the training of the people involved in the project, economies of scale would be generated in the medium and long term, project times would be reduced and thus project costs would be lowered.

Result

Even in highly regulated industries (banking, insurance, aerospace, medical), where documentation is eminently important to be able to show the regulating authorities when, how and why a change to the artifacts was either necessary or useful, or to show that requirements of the regulating authority were implemented as expected, both the V-model and hybrid or agile approaches have their raison d'être. The prerequisite here is, of course, that the rules of such processes are adhered to. Provided that the basic processes are mastered by the applying organization, it is of course possible to deviate from them, to adapt or change the process model. It is important to know, however, that adaptations mean that the procedure no longer corresponds to the proven original model and that this can result in unexpected side effects that can have a significant negative impact on the success of the project or program[13]. For this reason, it is incredibly important that the requirements, whether business, functional or technical, are elaborated to such a quality that they are, as mentioned above, easy to read and unambiguous, but still offer the developers enough freedom in their implementation. Regardless of whether it is an agile approach or a project procedure according to the V-model or a hybrid approach, requirements development and requirements management are the central components with which a project or program stands or falls. The developed requirements are the basis, just like building a house. If these are not correct, the project milestones will inevitably not be met; there will be change requests, which in turn will jeopardise the date for going live, or there will be errors in production. This is the worst-case scenario, which in turn means that the entire cascade has to start all over again in order to eliminate this error. Of course, this is clear to every Requirements Engineer, but it is important to make this clear to all project participants. Only well-trained project participants are able to identify weak points, as mentioned above, and thus avoid them.

Conclusion

For years, the ASQF, for example, has offered certification as a QAMP (Quality Assurance Management Professional), in which quality assurance professionals receive an additional certificate through an additional qualification as IREB Requirements Engineering, and can thus prove their expertise. This advance pursues a laudable goal, but does not go far enough and is so far only reserved for quality assurance professionals. Particularly with regard to agile or hybrid approaches, more attention must be paid to the training of all project participants. In agile methods in cross-functional teams (e.g. Scrum), every task must be able to be completed by every team member. And especially in this point a lot of potential is lost if there is no comprehensive training of the team members in the area of requirements engineering. But also in V-model development, where not only developers and testers but also program and project managers have to deal with business, functional and technical requirements, comprehensive training is indispensable in order to take up the project and its requirements in their entirety, to understand them and finally to be able to implement them. This leads to the fact that all project participants can judge what is involved with development or adjustment projects, can estimate expenditures, as well as judge the quality of the provided requirement documents, in order to estimate and possibly even quantify expenditure and impact. Due to the facts described above, every project member (project and program manager, functional and technical analyst, architect, developer and the technical and functional test analysts) should be trained and certified in requirements engineering in order to be able to correctly assess the scope and significance of the program content and thus to be able to give realistic estimates of the required project budgets and timelines.

References

  1. IREB Certified Professional for Requirements Engineering – Foundation Level – Syllabus Version 3.1.0 ireb.org/content/downloads/2-cpre-foundation-level-syllabus-3-0/cpre_foundationlevel_syllabus_en_v.3.1.pdf
  2. IREB Certified Professional for Requirements Engineering – Advanced Level Requirements Elicitation – Syllabus Version 3.0.0 ireb.org/content/downloads/9-syllabus-advanced-level-requirements-elicitation/cpre_elicitation_al_syllabus_en_3.0.pdf
  3. Handbook Advanced Level Requirements Elicitation ireb.org/content/downloads/11-cpre-advanced-level-elicitation-handbook/advanced_level_elicitation_handbook_en_v2.0.pdf
  4. IREB Certified Professional for Requirements Engineering Requirements Modeling, Advanced Level – Syllabus Version 3.0.0 ireb.org/content/downloads/17-syllabus-cpre-advanced-level-requirements-modeling/syllabus-cpre-al-requirements-modeling_en_v3.0.pdf
  5. Handbook Requirements Modeling ireb.org/content/downloads/19-handbook-cpre-advanced-level-requirements-modeling/ireb_cpre_handbook_requirements-modeling_advanced_level_en_v2.0.pdf
  6. ISTQB Lehrplan Certified Tester Foundation Level Version 2018 V3.1D german-testing-board.info/wp-content/uploads/2020/01/CTFL-DE_Syllabus_2018_V3.1.pdf
  7. ISTQB Certified Tester Advanced Level Test Analyst (CTAL-TA) Lehrplan v3.1.1 german-testing-board.info/wp-content/uploads/2021/07/ISTQB_CTAL-TA_Syllabus3.1.1-DE.pdf
  8. ISTQB® Certified Tester Foundation Level Extension Syllabus Agile Tester VERSION 2017 german-testing-board.info/wp-content/uploads/2021/01/Certified-Tester-Foundation-Level-Extension-Deutsch.pdf
  9. IREB Certified Professional for Requirements Engineering - RE@Agile Primer - Syllabus and Study Guide ireb.org/content/downloads/34-cpre-re-agile-primer-syllabus/ireb_cpre_re@agileprimersyllabusandstudyguide_en_v1.2.pdf
  10. Kriescher, Markgraf: Agiles V-Modell - ein Widerspruch? (Entwicklung: Agiles V-Modell - ein Widerspruch? - computerwoche.de)
  11. Canditt, Rauh, Wittmann: Brückenschlag: Das V-Modell XT mit Scrum inside (canditt_rauh_wittmann_OS_Agility_11:maas_OS_agility_09.qxd (sigs-datacom.de))
  12. Sutherland, Schwaber: The Scrum Guide scrumguides.org/docs/scrumguide/v2017/2017-Scrum-Guide-US.pdf
  13. Weck, Martin: Requirements Engineering Anforderungen als Basis effizienter Softwareentwicklung (Requirements Engineering — Anforderungen als Basis effizienter Softwareentwicklung – INNOQ)

Footnotes