The Potential of User Tests for Requirements Engineering

Introduction

Let us suppose that a company wants to build new software. Managers start by collecting all the needed information from the market and learn that there is a demand for this particular solution. Following this is a convincing business case, which then is a basis for deciding to give the green light to a new project.

Then there is a question: how to ensure that the new software the company will develop will be what customers want to buy and use? How to avoid the situation where the software is released but sales are low and customers' opinions are not as high as expected? The answer is: to test software with users.

Why test with users?

There are multiple benefits of tests with users. Adequately conducted tests help in learning the software’s idea and about the software itself. It means that the software can still be under development, but nevertheless its concept can be validated via user tests in the middle of the process. In further sections in this article, techniques on how to test unfinished software with users are described.

There is a wide variety of possible user tests. The article describes three ways to use user tests to (1,4) elicit, (2) validate and negotiate, and (3) re-check requirements (Picture 1). These types of tests are particularly interesting when it comes to software tests. They can provide feedback helping to make improvements without complicated and long-lasting analyses of the results. It is also possible to conduct them without many preparations and colossal spending. They are quick, inexpensive, and useful.

Types of users

Testing software needs participants - testers. There is a simple definition of a test user, with whom it is recommended to conduct research.

End-users.
The end-user is someone that does not know the software at all. Most of the time, end-users are selected from the potential future customer base.

"By observing real people attempt to use design, we see the interface from the users' perspective. Since average users lack insider knowledge about how the system is 'supposed' to work, they often encounter problems that the creators of the design did not foresee." ^[1]

Internal-users.
The internal-user is basically every person that works on the software. It includes project managers, designers, developers, QA. Also, stakeholders, investors, and every business person with knowledge of the software’s idea.

It is essential to understand the difference between the end-user and the people that work on the software. The latter think that they can put themselves in the shoes of customers. However, as the person involved in developing the product, it is hard to "forget" all the information. Then, the results of tests are biased and should not be used to make decisions over the software’s future.

Tests can be used as a tool to elicit requirements. A common practice is to engage end-users for that purpose. To validate and negotiate requirements, to be sure that the software will fulfill them, it is recommended to use internal-users as testers. Finally, to re-check the software’s compliance with requirements at the final step of the development, it is an excellent choice to rerun tests with end-users.

How can tests help to elicit requirements?
Tests with end-users can answer questions that can be the ground truth for requirements collection.

For example:

• What problems of end-users the future software could resolve, and how (to do it in the most efficient way)?
• What features in the software are the most exciting according to the MoSCoW method ^[6] ("must-haves" versus "could-haves")?
• What prices can be accepted by potential customers? Requirements may note the wish, for example, to avoid too greater expense in the implementation, or, conversely, to do better integrations or designs, when users are willing to pay a higher price for that.

Collecting requirements via tests with end-users can be done in various ways:

• Surveys and interviews. This is a basic form of gathering knowledge of end-users needs.
• Wireframes. They are an elemental form of documentation of software requirements ^[2]. It is a way to "picture" many of the requirements which to describe might need hundreds of words in a written specification. Wireframes are a prompt and understandable solution to document functional requirements. To use “Wireframing (...) is for translating requirements and ideas into a visual language that can be easily understood." ^[3]
• Prototypes. There are two ways of preparing prototypes:
  • When the prototype presents a fundamental concept of needed software, it shows simple functionalities via an elemental customer journey. This kind of prototype is moderately easy to prepare. It is a sketch of the software's idea in the form of simple wireframes that are not fully interactive at this stage.
  • When the prototype presents the more advanced software concept, then it is usually based on final designs. In most cases, the prototype is interactive: it shows all the features and functionalities and covers not only the elemental customer journey but also edge cases. This kind of prototype presents almost final software but still at the stage when changes are not expensive.

Tests with end-users are a way to gather information about what they need from the software. The results can be a first draft of requirements, and provide user input to them.

These kinds of tests should be conducted at a very early stage of development. Then it is less hard and less expensive to implement certain changes based on the results of the tests. It is undoubtedly more affordable to check the software's idea before it is available on the market. It gives time to adjust at the very early stage of development. As a consequence, it helps increase the chances of the software's success in general. It needs many cycles to get it right, and with every cycle the design is improved leading to better and better software. It is possible because of listening to the users' feedback and then adjusting to it. Then, the risk that the software will not meet expectations from the user's perspective is smaller.

How can tests help to validate and negotiate requirements?
An essential factor is to investigate requirements within a company. It includes validation and negotiation of requirements.

The first technique is to use tests with internal users. Research of that kind requires the usage of wireframes or prototypes (described in the section above). Thanks to internal presentations of future software visualization, it is possible to ensure that requirements are valid and up-to-date. The other benefit is that engaged internal users negotiate the software's scope to ensure the best outcome and future profitability. It is an aspect of the project where business requirements best meet users' needs and are combined with the technical requirements. It is the best phase in which to make decisions and set the goals to be achieved by releasing the software to the market.

The second technique is to review all user stories and compare them with the software from the current development stage. Tests of use cases show if the software is heading in the right direction.

How can tests help to (finally) re-check requirements?
This phase of tests re-check requirements with the almost final, or even the final, version of the software. Tests with end-users are promising ways to check if potential customers think that the product matches its purpose.

During tests with end-users, at the final stage of development, it is possible to answer these questions:

• Is it simple to explore the software (in general)?
• Is it easy to understand the software at first glance: what it does, what problems can it solve, or what actions are available?
• Are there any distractors that could prevent users from buying something, registering, or performing whatever action was required?
• Is the software confusing at any point?

During these tests with users, it is also possible to re-check whether the software fulfills the requirements and is compliant with the software's idea. Additionally, these tests can be expanded with questions about the business model: "Does the price match the value this software brings to customers?"; "What features should be improved or developed, to increase chances of customers buying the software?", and so on.

Requirements re-check can be done by using an MVP.

MVP - minimum viable product.
Providing a minimal product for the market as early as possible is one of the core ideas of an Agile approach to development. Furthermore, when the most basic version of the software is ready to be presented, it is recommended to run tests with end-users (again). Tests with MVP provide the possibility to quickly (and moderately cheaply) check if the software is understandable for users at this stage, and if it is fulfilling requirements. There is still the possibility of modifying the software to reach the best fit in the market.

Note.
It is worth mentioning that the implementation of external analytical tools into MVP will be beneficial. With the use of recording sessions, it is possible to observe: cursor movements, clicks, way of consuming the content, or even users' struggles—analysis based on that data helps to improve the further development.

End-users tests help to validate the future. When the MVP is available on the market for customers, and there are many ideas on what to develop next, tests with end-users can help. Research conducted with real customers could be divided into two parts. First: checking MVP to get to know if there is anything that could be improved within the version that currently is online. The second part would be a survey or questionnaire, containing questions on possible future features and users' needs that the MVP does not cover. In this way, it is possible to deep dive into customers' needs and know what they would be likely to buy. Asking end-users what will be a must-have and a nice-to-have in the future can genuinely change the software’s trajectory. The whole team involved in the development process lacks this perspective. Customers can bring that information to the project and help improve it.

Software tests - another benefit

There is an additional benefit of user tests for requirements managers. While ensuring the best software with the best market fit is a goal, there is a long road ahead. This road can be full of "bumps", whether internal or external problems. Both should be resolved by the team working on the project. Considering that sometimes there is politics involved (meaning fluctuations of power and members within a company ^[4]), user tests results can be used as a critical argument. Test outcomes can prove which requirements should stay in force even if there is an argument among team members.

Summary

As the article proves, tests with users provide many benefits. One that should not be omitted is the impact of creating, validating, negotiating, and finally re-checking the requirements. Lack of user input (both internal users and end-users) equals incomplete requirements. Of course, requirements are crucial to provide the software that will fit the market and fulfill users' needs. Tests are one of the tools that help to confirm the software's compliance with requirements.

Having in mind the benefits of user tests described in the article, the author hopes readers will implement this approach within their projects. She is also convinced that incorporating user tests as a part of every project can improve the quality of requirements, and as a result, the final software available on the market.

Note.
PRINCE2, PMP, or DSDM Agile have a well-described process for how to manage projects. It is worth mentioning that not all of them sufficiently highlight the importance of end-users in the process. Tests with users and research of what users need and expect are somehow missing from PRINCE2 ^[5], and partially from PMP. From the above approaches to project management, only DSDM includes a guide to the product's business testing. Agile includes a variety of techniques to gather requirements by user testing or ways to ensure the product's quality by re-checking requirements vs. the final product ^[6].

Sources

• [1] Whitenton, Kathryn. "But You Tested with Only 5 Users!: Responding to Skepticism About Findings From Small Studies." 2019. Retrieved from nngroup.com/articles/responding-skepticism-small-usability-tests; last visited 29 March 2021
• [2] Malecka, Katarzyna, "How Wireframing & Prototyping Can Help Product Managers Find Balance." 2019. Retrieved from pmcolumn.com/wireframing-prototyping-product-management; last visited 29 March 2021
• [3] Barnard, Leon, "Wireframes Are not Just for Designers: Wireframes as a Tool for Product Managers." 2020
• [4] Siadati, Rana, Wernick, Paul, Veneziano, Vito. "The case of Software Requirements Engineering." 2019. Retrieved from re-magazine.ireb.org/articles/learning-from-history-the-case-of-software-requirements-engineering.
• [5] Directing Successful Projects with PRINCE2®, 2014
• [6] Agile Project Management Handbook v2, 2016

Recommended reading

Krug, Steve. "Don't make me think. A common-sense approach to web and mobile usability." 2014.

Moscichowska Iga, Rogus-Turek Barbara. "Research as a base for designing a user experience." 2017.

Nielsen, Jakob. "Why You Only Need to Test with 5 Users." 2000. Retrieved from nngroup.com/articles/why-you-only-need-to-test-with-5-users; last visited 29 March 2021.

Norman, Don. "Observe, Test, Iterate, and Learn." 2018. Retrieved from youtube.com/watch?v=JgPppwsocRU; last visited 29 March 2021.

Nunnally, Brad, Farkas, David. "UX research. Practical Techniques for Designing Better Products." 2016.