Joy Beatty Candase Hokanson

Project Value Delivered

The True Measure of Requirements Quality.

The quest to measure the quality of requirements through requirements metrics has been around almost as long as requirements themselves. Researchers have found that good and clear requirements help lead to successful projects; so many organizations are trying to measure whether or not they have good requirements. However, we believe this is fundamentally the wrong thing to measure. The real target of our requirements efforts is the value a project actually delivers to an organization, and how that compares with the desired value the project was supposed to deliver initially — that is the thing we should be measuring.

Many organizations wonder how they know if their requirements are “good,” and search for metrics they can use to measure the quality of their requirements. Even in our own organization, we continually assess how we can or should measure the quality of our deliverables. Understanding how to measure requirements quality seems important because it indicates that requirements practitioners want to do good work and show their value. This could be the symptom of often feeling our role is undervalued or subordinate to other positions. Quantifying whether we as practitioners are doing a “good job” helps us feel good about what we are doing, and justifies our presence on projects.

So how do we measure requirements quality? Most researchers and practitioners would point to some variation of this list: well-written, complete, atomic, testable, unambiguous, validated, verifiable, clear glossary, unique identifiers on every requirement, and document consistency. These things are important, but we think deciding how to measure the quality of the requirements is an uninteresting problem to solve. In fact, asking how to measure the quality of the requirements is asking the wrong question.

A Better Question

Requirements are only a means to an end. Perfect requirements do not guarantee project success. If we could deliver a successful project every time without any requirements, we would. If we could do it with sloppy, poorly written requirements, we would. We spend money on getting requirements right because they tend to help deliver successful projects. However, it’s not requirements that help attain desired goals on projects – it’s business objectives.

With that in mind, a better question to ask, and a more interesting problem to solve related to the quality of our requirements, is “Are we going to achieve the business value?” We want to measure whether we have the right requirements that will ultimately meet a project’s business objectives because requirements practitioners are not just there to record what the stakeholder needs and pass it on to development; instead, they have to really understand the stakeholder’s requests and be able to determine if that ask contributes to the value the stakeholders want out of the project^[1]. To be fair, well written, quality requirements do make it easier to meet the project’s business objectives and thus necessitate some attention, but requirements quality is not the most important thing for us to focus on.

Some researchers do acknowledge this type of project value measurement. In Al Davis’ keynote at RE2010, he said we fall into a trap that “the process to achieve a goal becomes more important than the goal itself.” We do this because it’s easier to follow a script and do what it says, rather than figure out the real goal and how to best get there. In 2010, Davis challenged researchers to make sure their future requirements work relates to revenue growth or increasing market share^[2]. Another example is a researcher who says we can only measure the results of our requirements success by looking at the customer results we drive^[3].

IT Executive Interests

When we set out to define relevant metrics to determine project success, instead of focusing on what we knew we could measure, we sought to understand what CIOs care about and work backwards from there. For example, we could measure how many requirements are testable, but we wanted to learn about the perspective of the people paying for projects.

Guess what? Nowhere in our research findings do we see that CIOs care about requirements. Remember, requirements are only a means to an end. The Society of Information Management does a CIO study every year, and in 2013 the top IT executive concerns included business and IT alignment, business agility, productivity, cutting costs, and time to market. Based on the SIM results, CIO surveys, and our own experiences with IT executives, we focused on what we think are the top three most common IT executive goals:

• Increase business alignment
• Shift budget to the things that add value
• Decrease the time to market

For requirements practitioners, this means that writing “good” requirements and having “good” requirements practices mean nothing in a vacuum. Rather, our requirements practices need to help achieve these three goals; and they can by helping achieve specific business objectives.

Business Objectives: Define and Measure Business Value

Business value can be measured through business objectives — the measurable benefit that an organization expects to receive from a project. Business objectives should be written in a quantitative and measurable way, and almost always relate back to money by either increasing revenues and profit or decreasing costs.

Business objectives can be used to prioritize an organization’s project portfolio. If some measurable value can’t be identified, the project should be cut. Then within each project, the business objectives help to determine what features are in or out of scope. The business objectives are the single most important factor to consider when making scope decisions – not someone’s gut feel or personal desires, but actual value that will be delivered.

Using business objectives to cut projects and features are cyclic processes – you should constantly reassess which projects to start or cancel, and always manage the prioritization of features within projects. This is a major area of opportunity for requirements practitioners to have an impact; recommendations for project prioritization or de-scoping features backed by data will make a difference to a company’s bottom line.

Metrics that Add Value

IT executives care about increasing business alignment, shifting budget to the things that add value, and decreasing the time to market. Increasing business alignment is fundamentally about developing software that delivers intended business objectives. To determine if projects that add the most value are being selected, consider measuring:

• How many projects should be cut before they even begin?
• How many projects have measurable business objectives?
• Of those projects with measureable business objectives, how many met the stated business objectives?

Within projects, to determine whether or not features that add the most value to the project are being included, and those that do not are being cut from scope, consider measuring:

• How many features on already deployed projects are rarely or never used?
• What is the end user adoption rate for a project?
• How many features are typically cut on any given project?
• Are features mapped back to applicable business objectives?

Understanding there is a limit to spending and resourcing capacity, when shifting budget to things that matter, IT must deliver the most value possible within their given time, money and resourcing constraints. Some metrics to consider include:

• What is the business value return on development hours (business value realized/development hours)?
• What portion of the expected business value is delivered by IT?

Finally, getting products to market faster is fundamentally about reducing unnecessary work. Some metrics that can be used to monitor this include:

• How often are there releases to actual users?
• How much money is spent on rework?
• How many defects are there?
• How many defects are there which directly relate to missed requirements?

There are some requirements-related metrics that are great interim indicators of project success. These metrics are typically easier to measure but, while important, they are not sufficient to truly measure the quality of requirements. These include:

• Number of missed requirements relative to project size
• Number of requirements models used relative to project size
• Percentage of requirements mapped to models
• Number of requirements changed after development relative to project size
• Number of open issues (tracked throughout the life of the project)
• Number of requirements modified per day per employee (a proxy measurement for productivity)

Seilevel Project Evaluations and Survey Results

The following research results are from customer projects where an attempt was made to measure: the value of the requirements work, requests for assessments of customers’ past projects, and a survey used to measure alignment of business and IT. These are preliminary results based on first rounds of research; studies will continue to be conducted to further refine our understanding of measuring delivered value on projects.

Project Analysis Results

The following tables represent data from two customers where an assessment of past projects was executed in various dimensions. Specifically, there was an evaluation of the use of business objectives. These results are consistent with our experiences in other organizations — some have written business objectives, some have none, and very few measure whether they were met.

Criteria	Customer 1: Out of 6 projects	Customer 2: Out of 1000s of projects
Have business objectives	6	0
Measurable	4	0
Business objectives are related to money	2	0
Measured whether business objectives were met	0	0

Below are results from a sampling of six projects where we tracked how many features were cut from scope.

Projects	Initial Features	Final Features	% Cut
Project 1	66	56	15%
Project 2	50	41	18%
Project 3	15	9	40%
Project 4	32	9	72%
Project 5	68	15	78%
Project 6	308	16	95%

The first two projects were analyzed after they had been completed. In these cases, teams did a fair job of cutting some scope (15, and 18 percent respectively), but, since there is a reasonable chance that up to 65% of features are rarely or never used^[4], they probably could have cut another 35-40% of features. The other four projects show results ranging from 40% to even 95% of features being cut. This is primarily a result of using requirements models to help cut scope. The last project listed is a case where the entire focus was to put an acquired customer’s functionality into an existing application. The customer wanted to expend as little money as possible, so the minimum-viable product was derived for actual deployment.

The final project analysis counted how many models were used on projects. These are all projects we actually did work on and we have a very heavy models-focused approach, so we expected good results here.

	Project 1	Project 2	Project 3	Project 4	Project 5
Project Size (User Stories or Requirements)	419	109 business, 2500 functional	1490	308	88 epics, 484 stories, 972 acc crit
Models	102	149	345	59	51
Process Flows	62	88	74	37	24
Ratio (models and process flows as % of reqs / stories	24% 14%	6% 3.5%	23% 5%	19% 12%	10.5% 5%

The first row indicates the size of the project in terms of requirements. Know that this is not a perfect measure – a project could have more or less requirements depending on the diligence put into requirements.

After identifying total project size, the total number of models were counted. Note that process flows were counted as a separate category of models, as this is the one most frequently used on all Seilevel projects. We also included a ratio of both models to total requirements and process flows to total requirements to relatively compare the projects together. While we don’t currently have enough data to validate what ratio of models to requirements a “good” project should have, our initial recommendation is between 10 and 20%.

Business and IT Stakeholder Survey Results

Seilevel conducted surveys of actual IT and business stakeholders which focused on measuring the level of trust between these stakeholders. We received an approximate 33% response rate from IT stakeholders and a 70% response rate from business stakeholders (albeit with a smaller sample size). Our assumption was that part of the misalignment between business and IT stems from a lack of trust. We asked each group three questions about their confidence in what other teams were doing or capable of doing. Generally speaking, the results tell us that the business side has more confidence in IT capabilities than the IT side has in business’ involvement on IT projects.

Conclusions and Moving Forward

When you try to start measuring value there are likely to be challenges, so be ready for them. First, you can’t measure the real value a project delivers until after it’s over. So, do things as you go to help determine if you are on the right track. For example, are you cutting unnecessary features early on? One of the issues is that sometimes those teams disperse immediately after a project ends, so you may never be able to figure it out if you don’t plan ahead on how to handle this situation. The data usually doesn’t exist today, so it’s not going to be easy to measure this stuff. Measuring takes extra time, so plan for that time. Also, plan to build features to do measurements if you have to. People don’t want to be held accountable for actual results – so you will have to politically maneuver through this carefully.

We threw a bunch of research, survey results, and project analysis at you, so what can you take away to start implementing on your own projects?

• Remember first and foremost that people are motivated by what they are measured against. Bad metrics can lead to bad behavior. For example, if you set a metric to bonus based on numbers of defects, people stop logging the defects. Trying to rally the entire organization to measure business value increases the likelihood the value will be delivered. This can be exciting for them if they feel like they are actually helping to improve the company’s bottom line.
• Select practices that will help you measure value — like focusing on business objectives. For example, ensure the entire team – the business stakeholders, the developers, and the testers understand the business objectives. Then, once you have business objectives, cut projects that have minimal value added and tie your requirements or features back to those business objectives.
• Select a few interim metrics that apply to your project, but don’t try to do too many (3-5). Start keeping track of those chosen metrics and working on incremental improvement or continue analyzing to determine a trend.

But remember, at the end of the day, requirements are just a means to an end. Requirements practitioners can have a huge impact on their projects and make a difference in their organizations. To do this, they need to focus requirements efforts on the tough decisions that lead to the ultimate measure of project and requirements success: achieved business value.

References

• [1] http://www.ireb.org/fileadmin/IREB/Download/Homepage%20Downloads/IREB_CPRE_Glossary_16.pdf ) 2014.
• [2] Al Davis at RE2010 "Requirements: A Textbook Example of Goals Displacement“ keynote.
• [3] http://varokas.files.wordpress.com/2010/02/measuring-requirements-engineering-success.pdf
• [4] The Standish Group. 2009. “CHAOS Summary 2009.“ West Yarmouth, MA: The Standish Group International, Inc.