Think Like a Scientist

Successful software systems from the combined effort and brainwork of a visionary business organization and a technically skilled delivery organization. Optimal solutions to the wrong problems are not likely to achieve the desired business results. Therefore, software projects need to broaden their scope and not only work on solutions but also commit to the discovery of the right problem. Strangely though, most software development methodologies focus only on how to produce new features and functions. But the difficulty lies not only in making the software per se, but also in finding out which problem to solve.

We want to build what we can sell rather than sell what we can build!

To accomplish this we shall introduce the art of hypothesis testing into the toolbox of the requirements engineer. Hypothesis testing comes from the world of science as a method of objectively determining the validity of uncertain claims. Then, by defining and building the smallest possible product, and shipping it as a hypothesis test, we can learn about our customers and avoid wasting our resources on futile products and focus our efforts on the ones that will achieve desired business results.

Hypothesis Testing

A business model reflects the intent of a company to deliver value to customers, entice customers to pay for value, and to convert those payments to profit. At design-time, a business model is just a chain of hypotheses about the cause and effect relationships that enable the long-term business effects. It might look good on paper and the work put into its design might be substantial, but it does not make it a solid foundation to craft new businesses from. Yet, high level models are strikingly often taken as hard fact, while in reality, it is ’terra incognita’ and needs to be tested incrementally. The problem is that people tend to stick to their beliefs and defend, justify, and rationalize them with a battery of intellectual reasons, cogent arguments, and rational explanations. We simply have a hard time letting go of ideas we have fallen in love with.

Originating from the world of science, hypothesis testing is an essential part of the scientific method. The scientific method guarantees that a claim becomes a fact only when it is confirmed to such an extent it would be reasonable to offer general agreement. The process steps of the scientific method are simple but effective:

1. Ask a question; State the problem to be solved.
2. Do background research; Gather information and investigate known facts about the topic.
3. Construct a hypothesis; Based on the research, construct an educated guess about the inner workings and what the answer to the question might be.
4. Test with an experiment; Design a test that will either confirm or refute the hypothesis.
5. Analyse results and draw conclusions.

Although software development is more art than science, we can design products with better market fit by applying the fundamentals of the scientific method. In order for a software solution to work under real market conditions and not just as a clever idea, we need to confirm the following general hypotheses:

Hypothesis 1: People have this problem
How to test: Pre-studies and market research are often used to answer a question like this. The bigger the study-group, and the more elaborate techniques invoked, the higher the confidence in the results.

Hypothesis 2: Our solution addresses the problem
How to test: If the problem is well-defined, system testing will validate the claim by checking it against the specified requirements.

Hypothesis 3: People will use our solution
How to test: Build the smallest thing you can and ship it as a hypothesis test.

Many projects tests hypotheses of type 1. Feasibility studies and market research before committing to a project investment is an effective insurance not to waste unnecessary resources on futile product development.

Basically, all projects test hypotheses of type 2. This is where the ordinary functional verification process comes in play and this type of testing is relatively easy to do.

Very few projects build and ship small product increments to track and act on market metrics and will have no notion of the business effects until the product is finalized. By then, all resources will be spent.

Requirements engineers are often (or should be) deeply involved in assisting testers develop tests for functional verification. These are important tests, but they will only validate the system in terms of functional quality, while we also need to validate whether people will buy, sign-up or use our product. This is where hypothesis tests matter and the requirements engineer should be deeply involved, or even the driving force, creating them as well.

How to write hypothesis tests

Since requirements are concocted from hypotheses, then requirements are hypotheses themselves. This offers a good opportunity for the forward thinking requirements engineers who are able to think about, understand, and act on information in a critical way and adept enough to reason about hypotheses and hypothesis testing. Let us take a look at how we can design a good hypothesis test.

Per definition, a hypothesis is something we believe (or wish) to be true but have not yet proved correct. For example, ”we believe customers will sign up for our new online gaming service”. Our job, as requirements engineers, is to make sure we do not waste time building features that do not stand the test of their hypotheses. In the example above, there should be a smallest thing we can build and ship in order to learn whether the online gaming service will attract customers or not. We should not need to build the complete system up front, but rather focus on the few features that can prove or refute the claim.

A hypothesis test consists of the three components; the hypothesis formulation, the test itself and the evidence of a successful outcome. Using sentence templates to formulate them is a good way of starting off.

Since the basis for the hypothesis test is the formulation of the hypothesis, you should put in some effort into describing it well and unambiguously. Here are some other tips that can help you construct good hypothesis tests:

• Developing a good hypothesis test is not a one man job. They should always be discussed among all key project roles, so always remember to write in a simple and clear manner, to avoid misunderstanding.
• Write as statements, i.e. as ’to the point’ claims of your beliefs. It will make it easier designing the actual tests.
• Establish participants (who is involved), variables (what is involved) and prediction of the outcome (evidence).
• Evidence may be of a quantitative nature (resulting in metrics) or qualitative (observable outcome). Always aim for the measureable; metrics is the undisputed champion to objectively interpreting test results.
• Most important of all, make hypothesis tests testable.

A simple requirements process

In order to carry out hypothesis testing we would define an increment of the product requirements, build and ship it as a ’tool’ to measure user behaviour and learn from that. Let us define a simple requirements process and include hypothesis testing as an integral part. Such a process would iteratively work on increments of product requirements, fractional compared to the full-scaled system. Because of the small-scale approach, the requirements can quickly be constructed and tested at a fraction of the risk and cost. The hypotheses we test will either be true, false or to some extent true. Depending on the outcome we might need to modify our product or feature ideas, the hypotheses derived from them or the requirements themselves. When a hypothesis is confirmed, we dig into the next important hypothesis and repeat the process.

This simplistic process starts by asking you to choose the metrics you want to improve. Your choice of metrics will drive the process and which requirements set to develop in the current increment.

Conclusion

Hypothesis testing is the natural way of confirming or refuting hypotheses. It comes from the world of science as a method to objectively determine the validity of uncertain claims. By defining, building and shipping the smallest possible product increment that can shed light on the hypotheses, we do not need to waste unnecessary resources on futile products.

One might argue that it is an approach more conceivable in certain lines of business than others, and yes, that is the case. It is much easier to run a continuous series of hypothesis tests for a web-based service than on a banking system running on mainframe. The point is, however, that the speed and frequency by which hypothesis testing can drive the requirements process is all relative to the competition. If one bank ships a new version of its’ Internet banking service once a year, a more competitive bank might do it every three months as a hypothesis test.

The art of requirements engineering is the glue that binds together hypothesis testing, continuous shipment and business effect tracking. As a requirements engineer, with the right insight and equipped with the right tools you can make a world of difference and a real impact for your future organizations by embracing these techniques and incorporating them into your requirements engineering toolbox.

References

• 1. Lean Analytics; A. Croll, B. Yoskovitz; O'Reilly Media; ISBN 978-1449335670
• 2. Exploring the Scientific Method; Steven Gimbel; University Of Chicago Press; ISBN 978-0226294834