By search term

By author
  • Practice
  • Studies and Research

Project Value Delivered


Many organizations wonder how they know if their requirements are “good,” and search for metrics they can use to measure the quality of their requirements. Even in our own organization, we continually assess how we can or should measure the quality of our deliverables. Understanding how to measure requirements quality seems important because it indicates that requirements practitioners want to do good work and show their value. This could be the symptom of often feeling our role is undervalued or subordinate to other positions. Quantifying whether we as practitioners are doing a “good job” helps us feel good about what we are doing, and justifies our presence on projects.

So how do we measure requirements quality? Most researchers and practitioners would point to some variation of this list: well-written, complete, atomic, testable, unambiguous, validated, verifiable, clear glossary, unique identifiers on every requirement, and document consistency. These things are important, but we think deciding how to measure the quality of the requirements is an uninteresting problem to solve. In fact, asking how to measure the quality of the requirements is asking the wrong question.

A Better Question

Requirements are only a means to an end. Perfect requirements do not guarantee project success. If we could deliver a successful project every time without any requirements, we would. If we could do it with sloppy, poorly written requirements, we would. We spend money on getting requirements right because they tend to help deliver successful projects. However, it’s not requirements that help attain desired goals on projects – it’s business objectives.

With that in mind, a better question to ask, and a more interesting problem to solve related to the quality of our requirements, is “Are we going to achieve the business value?” We want to measure whether we have the right requirements that will ultimately meet a project’s business objectives because requirements practitioners are not just there to record what the stakeholder needs and pass it on to development; instead, they have to really understand the stakeholder’s requests and be able to determine if that ask contributes to the value the stakeholders want out of the project[1]. To be fair, well written, quality requirements do make it easier to meet the project’s business objectives and thus necessitate some attention, but requirements quality is not the most important thing for us to focus on.

Some researchers do acknowledge this type of project value measurement. In Al Davis’ keynote at RE2010, he said we fall into a trap that “the process to achieve a goal becomes more important than the goal itself.” We do this because it’s easier to follow a script and do what it says, rather than figure out the real goal and how to best get there. In 2010, Davis challenged researchers to make sure their future requirements work relates to revenue growth or increasing market share[2]. Another example is a researcher who says we can only measure the results of our requirements success by looking at the customer results we drive[3].

IT Executive Interests

When we set out to define relevant metrics to determine project success, instead of focusing on what we knew we could measure, we sought to understand what CIOs care about and work backwards from there. For example, we could measure how many requirements are testable, but we wanted to learn about the perspective of the people paying for projects.

Guess what? Nowhere in our research findings do we see that CIOs care about requirements. Remember, requirements are only a means to an end. The Society of Information Management does a CIO study every year, and in 2013 the top IT executive concerns included business and IT alignment, business agility, productivity, cutting costs, and time to market. Based on the SIM results, CIO surveys, and our own experiences with IT executives, we focused on what we think are the top three most common IT executive goals:

  • Increase business alignment
  • Shift budget to the things that add value
  • Decrease the time to market

For requirements practitioners, this means that writing “good” requirements and having “good” requirements practices mean nothing in a vacuum. Rather, our requirements practices need to help achieve these three goals; and they can by helping achieve specific business objectives.

Business Objectives: Define and Measure Business Value

Business value can be measured through business objectives — the measurable benefit that an organization expects to receive from a project. Business objectives should be written in a quantitative and measurable way, and almost always relate back to money by either increasing revenues and profit or decreasing costs.

Business objectives can be used to prioritize an organization’s project portfolio. If some measurable value can’t be identified, the project should be cut. Then within each project, the business objectives help to determine what features are in or out of scope. The business objectives are the single most important factor to consider when making scope decisions – not someone’s gut feel or personal desires, but actual value that will be delivered.

Using business objectives to cut projects and features are cyclic processes – you should constantly reassess which projects to start or cancel, and always manage the prioritization of features within projects. This is a major area of opportunity for requirements practitioners to have an impact; recommendations for project prioritization or de-scoping features backed by data will make a difference to a company’s bottom line.

Metrics that Add Value

IT executives care about increasing business alignment, shifting budget to the things that add value, and decreasing the time to market. Increasing business alignment is fundamentally about developing software that delivers intended business objectives. To determine if projects that add the most value are being selected, consider measuring:

  • How many projects should be cut before they even begin?
  • How many projects have measurable business objectives?
  • Of those projects with measureable business objectives, how many met the stated business objectives?

Within projects, to determine whether or not features that add the most value to the project are being included, and those that do not are being cut from scope, consider measuring:

  • How many features on already deployed projects are rarely or never used?
  • What is the end user adoption rate for a project?
  • How many features are typically cut on any given project?
  • Are features mapped back to applicable business objectives?

Understanding there is a limit to spending and resourcing capacity, when shifting budget to things that matter, IT must deliver the most value possible within their given time, money and resourcing constraints. Some metrics to consider include:

  • What is the business value return on development hours (business value realized/development hours)?
  • What portion of the expected business value is delivered by IT?

Finally, getting products to market faster is fundamentally about reducing unnecessary work. Some metrics that can be used to monitor this include:

  • How often are there releases to actual users?
  • How much money is spent on rework?
  • How many defects are there?
  • How many defects are there which directly relate to missed requirements?

There are some requirements-related metrics that are great interim indicators of project success. These metrics are typically easier to measure but, while important, they are not sufficient to truly measure the quality of requirements. These include:

  • Number of missed requirements relative to project size
  • Number of requirements models used relative to project size
  • Percentage of requirements mapped to models
  • Number of requirements changed after development relative to project size
  • Number of open issues (tracked throughout the life of the project)
  • Number of requirements modified per day per employee (a proxy measurement for productivity)

Seilevel Project Evaluations and Survey Results

The following research results are from customer projects where an attempt was made to measure: the value of the requirements work, requests for assessments of customers’ past projects, and a survey used to measure alignment of business and IT. These are preliminary results based on first rounds of research; studies will continue to be conducted to further refine our understanding of measuring delivered value on projects.

Project Analysis Results

The following tables represent data from two customers where an assessment of past projects was executed in various dimensions. Specifically, there was an evaluation of the use of business objectives. These results are consistent with our experiences in other organizations — some have written business objectives, some have none, and very few measure whether they were met.

Criteria
Customer 1: Out of 6 projects
Customer 2: Out of 1000s of projects
Have business objectives
6
0
Measurable
4
0
Business objectives are related to money
2
0
Measured whether business objectives were met
0
0

Below are results from a sampling of six projects where we tracked how many features were cut from scope.

Projects
Initial Features
Final Features
% Cut
Project 1
66
56
15%
Project 2
50
41
18%
Project 3
15
9
40%
Project 4
32
9
72%
Project 5
68
15
78%
Project 6
308
16
95%

The first two projects were analyzed after they had been completed. In these cases, teams did a fair job of cutting some scope (15, and 18 percent respectively), but, since there is a reasonable chance that up to 65% of features are rarely or never used[4], they probably could have cut another 35-40% of features. The other four projects show results ranging from 40% to even 95% of features being cut. This is primarily a result of using requirements models to help cut scope. The last project listed is a case where the entire focus was to put an acquired customer’s functionality into an existing application. The customer wanted to expend as little money as possible, so the minimum-viable product was derived for actual deployment.

The final project analysis counted how many models were used on projects. These are all projects we actually did work on and we have a very heavy models-focused approach, so we expected good results here.

Project
1
Project
2
Project
3
Project
4
Project
5
Project Size (User Stories or Requirements)
419
109 business, 2500 functional
1490
308
88 epics, 484 stories, 972 acc crit
Models
102
149
345
59
51
Process Flows
62
88
74
37
24
Ratio (models and process flows as % of reqs / stories
24%
14%
6%
3.5%
23%
5%
19%
12%
10.5%
5%

The first row indicates the size of the project in terms of requirements. Know that this is not a perfect measure – a project could have more or less requirements depending on the diligence put into requirements.

After identifying total project size, the total number of models were counted. Note that process flows were counted as a separate category of models, as this is the one most frequently used on all Seilevel projects. We also included a ratio of both models to total requirements and process flows to total requirements to relatively compare the projects together. While we don’t currently have enough data to validate what ratio of models to requirements a “good” project should have, our initial recommendation is between 10 and 20%.

Business and IT Stakeholder Survey Results

Seilevel conducted surveys of actual IT and business stakeholders which focused on measuring the level of trust between these stakeholders. We received an approximate 33% response rate from IT stakeholders and a 70% response rate from business stakeholders (albeit with a smaller sample size). Our assumption was that part of the misalignment between business and IT stems from a lack of trust. We asked each group three questions about their confidence in what other teams were doing or capable of doing. Generally speaking, the results tell us that the business side has more confidence in IT capabilities than the IT side has in business’ involvement on IT projects.

Conclusions and Moving Forward

When you try to start measuring value there are likely to be challenges, so be ready for them. First, you can’t measure the real value a project delivers until after it’s over. So, do things as you go to help determine if you are on the right track. For example, are you cutting unnecessary features early on? One of the issues is that sometimes those teams disperse immediately after a project ends, so you may never be able to figure it out if you don’t plan ahead on how to handle this situation. The data usually doesn’t exist today, so it’s not going to be easy to measure this stuff. Measuring takes extra time, so plan for that time. Also, plan to build features to do measurements if you have to. People don’t want to be held accountable for actual results – so you will have to politically maneuver through this carefully.

We threw a bunch of research, survey results, and project analysis at you, so what can you take away to start implementing on your own projects?

  • Remember first and foremost that people are motivated by what they are measured against. Bad metrics can lead to bad behavior. For example, if you set a metric to bonus based on numbers of defects, people stop logging the defects. Trying to rally the entire organization to measure business value increases the likelihood the value will be delivered. This can be exciting for them if they feel like they are actually helping to improve the company’s bottom line.
  • Select practices that will help you measure value — like focusing on business objectives. For example, ensure the entire team – the business stakeholders, the developers, and the testers understand the business objectives. Then, once you have business objectives, cut projects that have minimal value added and tie your requirements or features back to those business objectives.
  • Select a few interim metrics that apply to your project, but don’t try to do too many (3-5). Start keeping track of those chosen metrics and working on incremental improvement or continue analyzing to determine a trend.

But remember, at the end of the day, requirements are just a means to an end. Requirements practitioners can have a huge impact on their projects and make a difference in their organizations. To do this, they need to focus requirements efforts on the tough decisions that lead to the ultimate measure of project and requirements success: achieved business value.

References

Comments (4)

  • From: Marc Nadeau
  • Date: 21. July 2016

Thank you!

I lead a BA organization and have been circling this issue over the last few weeks after it occurred to me out project teams (and stakeholders) do not really understand the business objectives of the project much less have any measures for them. This article has given me a good framework to have the conversation to advocate a change in our approach.

  • From: René
  • Date: 25. August 2014

Hi,

in chapter "Project Analysis Results" you are talking about requirements models. Can you explain that a little bit? What models are this, can you give an example please?

Did I understand right, that you can question the requirements with those models and eventually reduce the number of user stories / req?

Thanks for you feedback.

cheers
René

Hi Rene,
For requirements models here, we are talking about any model used to help find requirements. We were basing our analysis on the RML models found in Visual Models for Software Requirements. There are 22 RML models (seilevel.com/ba-resources/rml-requirements-visual-models), but some of the more common ones used for this kind of analysis are Process Flows, Feature Trees and Business Objectives Models.

And yes, with the models, you can question or analyze the requirements- this generally leads to finding missing requirements but can reduce scope by prioritizing features and requirements according to the business objectives of the project. Hope that helps!
Candase

  • From: José
  • Date: 12. August 2014

I guess the principal idea is ok, rather than focus on a document specification, focus on added value delivering, and avoid wasting time in useless projects and features.

Just that it's not enough... I don't know what kind of industry do you work for, I work for software, I know requirements engineering apply to a very wide of projects, and in a software project environment, software requirements quality attributes matter.
Imagine that you are developing the next killer app that will raise your profit according with your XYZ predictions based on your XYZ estimations. That is really important, I agree, it's better to focus on what really will generate profit. But, what happens if you are not able to transfer the business/strategic knowledge to a document and/or people minds, so you are not able to elicit, analyse and validate the requirements… so I guess it's a knowledge transfer problem.

I also think that when building metrics, two things could be measured, the process and the product. As you write, there are a lot of metrics about requirements specifications, but there are few for the process. Define what project/feature should be developed sound more a decision making task, and I guess that not all RE analysts are able to make VNA or a ROI predictions. But, is really interesting.

Talking about chaos reports, they are being heavily criticized because of their methodology. 68% projects with unused features... but no one says what kind of projects? finished projects? finished on time? finished with overruns? not finished? Jørgensen & Østvold (2005) and Holtsnider et al (2010) made a good review on chaos reports, some times it's very dangerous make predictions based on a report whose methodology is unclear.

Finally, about the article redaction, I guess it's ok for an online magazine, but a I prefer to label each table/image and include a reference on a paragraph, so they doesn't looks like unchained or out of context.

Author's profile
Related Articles
RE for Testers

Why Testers should have a closer look into Requirements Engineering

Read Article
Product Owner in Scrum

State of the discussion: Requirements Engineering and Product Owner in Scrum

Read Article
Toward Better RE

The Main Thing is Keeping the Main Thing
the Main Thing

Read Article