Will RE Go the Way of the Dinosaur?
Will RE/BA become extinct? I first took note of this question in 2013, when 2 large companies in different business domains and countries (the energy sector in Scandinavia and telecommunications in Canada) approached me on the verge of a planned enterprise-wide transition to agile development. They wanted to know whether they would still need their requirements engineers/business analysts after the transition. They were concerned because agile consultants had been advising them that those roles would no longer be needed since developers would be having direct conversations with users. Today, this question remains a major source of anxiety for those in the RE profession.
The Elephant in the Room
Let’s begin by addressing the elephant in the room. The truth is that there are many startups, companies in the social media space and disruptive innovators that do not include Requirements Engineers or BAs on their teams - despite the fact that they often include a broad spectrum of other roles, such as data analysts, data scientists, cloud experts, engineers and engineering managers. Most of the popular agile frameworks, like Scrum, XP and SAFe don’t specify an RE or BA role either. It’s no wonder, then, that many in the RE and BA community are worried.
There are a number of reasons, though, not to take the above concerns at face value. While it’s true that in the early stages of a company and new product development, agile teams seem to do well without a focus on requirements engineering and planning, once the product and the business mature, those competencies become critical to agility and to overall performance. The reason for this has to do with complexity. During the early period in the development of a novel product, the need for adaptability is high and the product itself is relatively simple from a technical and functional standpoint. These factors favour minimal planning and preparation. However as the business problems addressed by the product become more complex and its technical architecture follows suit, more competencies are needed than can fit on a small agile team. These factors – along with the use of components and component teams – introduce dependencies which, in turn, increase the importance of preparation and planning in order to align teams. As a result, startups that have previously placed little emphasis on RE/BA find that they need to strengthen those competencies as they mature. As for the established businesses and organizations who already have had REs/BAs on staff prior to an agile transition - my own experience is that the vast majority have retained them afterwards. They’ve done so, they tell me, because they find they add value. Research backs this up.
The RE Value-Added
Extensive research on nimbleness shows that agile organizations do better when they include RE/BA practitioners on their teams, especially when they are given leading roles. This conclusion aligns with my own experience working with companies across a broad range of sectors including telecommunications, finance and Insurance, healthcare, retail and government.
The following figures summarize results from IIBA’s nimble initiative (of which I am an advising member) – where nimble is defined “as the ability of an organization to navigate unpredictable business environments and operationalize innovation through a scalable capability to sense and respond to change with precision and speed”[1]. Nimble practices are defined in the study as abilities that support nimbleness, such as the ability to enable fast decision-making. Figure 1 illustrates a strong correlation between an organization’s nimble practices and BA capability - the degree to which business analysis and the business analyst is in a leader/driver role. For example, it found that organizations who exceed [others] or are best in class in the nimble practice “the ability to visualize, identify and assess opportunities for improvement and translate these in terms of what is both viable and valuable for the business and/or customer”[2] had a high BA capability (60.2) whereas those that lagged or were behind their peers in this nimble practice had extremely low BA capability (7.3). Similarly, the difference in BA capabilities between those who excel in the “ability to bring together business, customer, and technology to drive value”[3] and those who lag in this area is a staggering 813% (61 to 7.5). It’s clear that businesses are much more likely to excel in all of the 8 nimble capabilities measured by the study when BA is in a leading role.
Click for detailed view of figure 1
Research also indicates that organizations are more likely to adopt agile methods when they have a strong BA capability – regardless of the agile framework. For example, figure 2 indicates that adoption of adaptive/learning organization approaches is 71% when BA is in an enabler or lead role but only 49% when in a limited/support role.
A similar correlation holds true when we look at organizational performance as a whole. Figure 3 indicates that an organization is more likely to excel in each of the following dimensions of performance when BA is in an enabler/lead role:
- Exceeding the speed of change demanded by market conditions
- Achieving excellence in operational efficiency
- Creating and operationalizing innovation
- Delivering superior customer value or experience
The research also gives us an indication about the reason for this impact. As figure 3 shows, a key nimble practice impacted by BA/RE is the ability to align and facilitate the team’s activity to the purpose of the business. Simply put, with a strong RE competency in the mix, teams spend more of their time working on the things that really matter. We’ll examine how it does that, but first let’s look at what RE – and agile development in general – cannot do.
It's Not About More Work in Less Time
Despite initial claims to the contrary, agile development has not been shown to deliver outsize productivity benefits, when measured by output, e.g., lines of code. Though there have been modest improvements (e.g., a “16% increase in productivity” quoted by QSM Associates[7]), these are a far cry from the “several 100 percents higher” promised by early agile advocates.[8] In fact, if you do a feature-by-feature comparison, agile development often costs almost as much as waterfall because of the rework resulting from its trial-and-error approach. The promise of agile development is not to get more work for less effort but to improve outcomes. In that sense agile has succeeded. Adoption of agile development has been shown to result in products that are 37% faster to market without the increase in defect counts that would be expected by the increased speed.[9] A strong, agile RE competency helps the organization realize these benefits by focusing the team on the work of the highest value.
Prioritizing Value Across Backlog
When RE/BA practitioners guide the business to evaluate work items (e.g., stories and features) in order to prioritize them, they do so across all of the dimensions of value. As the figure 4 indicates, these include user value, business value, learning value, risk reduction and opportunity enablement (RR & OE) and time criticality. (Readers may recognize these as components of Cost of Delay in the SAFe/WSJF approach.)
User Value
User value is what teams are typically focused on: features or changes that provide new or enhanced capabilities. User value is the source of all other value: if users don’t like the product, nothing else matters. But it is not the only element to consider with respect to prioritization. In fact, it could be argued that while user value is a prerequisite for all other values - business value is the reason for creating the product.
Business Value
Business value is a benefit to the company offering the product or service, e.g., increased efficiencies, monetization of data and improved forecasting. Business value may or may not align with user value. For example, a feature to apply new transaction charges benefits the business but most customers would prefer that it not be included. Business value may be strategic. For example, a team working on a social aggregator application was guided to prioritize integration with a 3rd party social network. The work was prioritized not because users were clamouring for it, but because of a strategic decision to pursue a partnership with the 3rd party. (It was thought that developing these integration features would demonstrate the seriousness of their long-term commitment to the other company.)
Learning value
An example of learning value is a backlog item to test whether users will want to buy a proposed product by offering it to them before it is ready. These kinds of learning experiments are particularly important in the context of novel product development where there is no historical data. For example, an insurance company I worked with was considering developing Usage Based Insurance (UBI) – an insurance product whereby benefits are tied to behaviours. Because there was so much uncertainty about whether customers would overcome their privacy concerns about the product, the team tested the market first by offering immediate benefits to anyone who signed up for a trial. They did so before the product was developed so that they could learn whether it was even worth developing.
Risk Reduction and Opportunity Enablement (RR & OE)
Examples of Risk Reduction (RR) include prototypes and proofs-of-concept to test technical and marketing assumptions. Opportunity Enablement (OE) includes technical improvements to enable future business opportunities, such as improvements in scalability, security and automated testing.
Time Criticality
Time criticality takes into consideration the value lost if the work is not done quickly. For example, a development organization I worked with was endeavouring to comply with a set of reporting requirements by a specified deadline in order to qualify for government consulting contracts. The lead RE on the team helped stakeholders distinguish between features that were required for compliance vs. others that were ‘nice-to-have’. It turned out that many of the requested features were those stakeholders had seen in the market and liked – but there was no need to implement them quickly (or perhaps at all), since they were not needed for compliance. By focusing on the smaller set of time-critical features, the organization was better able to meet its deadline – thereby opening up a critical business opportunity that would otherwise have been lost.
The inclusion of an RE or BA in an enabler/lead role within an agile organization helps ensure that all of these dimensions of value are considered when prioritization decisions are made.
Defining ‘agile RE’
As the preceding discussion shows, we know that RE/BA is important to an agile organization. But what does the practice look like? To understand what it means to practice RE in an agile context, we must begin with RE itself. RE is “The systematic and disciplined approach to the specification and management of requirements with the goal of understanding the stakeholders’ desires and needs and minimizing the risk of delivering a system that does not meet these desires and needs.”[10] The practice of RE in agile context - what I’ll refer to as “agile RE” – is roughly analogous to agile business analysis and planning which I’ve defined as follows:
“the examination of a business or any aspect of it (culture, organizational structure, process, products, and more) to learn what needs to change and when in order to achieve a desired outcome in a context that places a high premium on adaptability, resilience, and continuous
innovation and value delivery. It includes analyzing who the product is for (the stakeholders), defining their requirements, determining when the capabilities will be delivered, and estimating costs and resources.”[11]
Note the specification of a context where adaptability and innovation are highly valued. The context is important because it profoundly affects the way the RE competency is practiced. For example, while it may make sense to perform an extensive requirements analysis upfront in a context where requirements are knowable at the start, it makes much less sense to do so if the product is novel and there is extreme uncertainty about customer needs.
The Premium Paid When RE is Not Adequately Addressed
Agile organizations that don’t pay enough attention to developing an RE competency tend to suffer from the following symptoms:
- Product-wide initiatives neglected: Big changes sit on the shelf because feature teams are rarely available at the same time for broad, product-wide changes.
- Fractured user experience: Teams lack a common vision, leading to a fractured experience as users navigate different areas of the product.
- Unnecessary errors and rework: Requirements are insufficiently analyzed before development begins, leading to unnecessary errors and rework.
- Soft skills deficit: The soft skills that RE and BA professionals would have brought to the table are often inadequately covered by other team members. These include conflict negotiation, problem solving, requirements elicitation and change management.
- Delays: Features and user stories are inadequately analyzed and understood at the time they are estimated, resulting in delayed deployment when the work turns out to take longer than expected.
- Real value still not delivered any quicker to customer: The key sign that RE is under-developed is that despite following an agile framework and best practices, like story-writing, teams are not delivering value to the customer any faster than they were when using a waterfall approach.
How Does Agile RE Achieve Its Benefits?
Agile RE uses a number of tools and techniques to focus the team on highest-value work at all times. Some of the most effective of these are described below.
Lean Startup MVP
One of the primary ways REs reduce waste in the development process is by practicing the Lean Startup approach to product development. The central idea behind LeanStartup is to use low-cost experiments to drive development investment. These exploratory versions are referred to as Minimum Viable Products (MVPs). Agile REs specify and plan MVPs to test unproven hypotheses that are critical to a product’s success – then use the results to help direct investment towards the most promising ideas. I’ve seen this approach used effectively in countless ways. The previously mentioned experiment to gauge consumer interest in UBI is one such example. In another example, a telecommunications company was considering a novel feature to allow customers to customize their own mobile phone plan. Before spending a lot to develop a self-service portal they weren’t sure people would use, the company developed a ‘sandbox’ MVP, unconnected with backend systems. To customers, it looked like the real thing, while, behind the scenes, internal staff manually implemented the customized plans.[12]
Minimum Marketable Product (MMP), Minimum Marketable Feature (MMF)
Agile REs and BAs shorten time-to-market by focusing development effort on the minimum capabilities needed for a new product or feature to be viable in the marketplace. These versions are known respectively as the Minimum Marketable Product (MMP) and Minimum Marketable Feature (MMF). MVP experiments (described in the previous section) are used to determine the initial MMP and MMFs. As development proceeds, requirements are incrementally added and dropped based on user feedback and metrics.
Story-Splitting
A key competency for enabling the continuous delivery of value is “story splitting” - the decomposition of large requests into smaller requirements items (usually referred to as user stories), each of which delivers value when used. It’s a competency that many teams I’ve worked with find difficult; yet without it, it’s often impossible to deliver valuable improvements within the time frames required by many companies and by agile frameworks like Scrum. RE practitioners use techniques like the SAFe/Lawrence patterns to recognize and address commonly occurring scenarios. For example, in one pattern, features are split based on their Acceptance Criteria (AC) – with each resulting story implementing a small subset of AC. Since each criterion represents a complete scenario, each story is guaranteed to deliver value to the user.[13]
Feature Preparation (aka Backlog Refinement)
Requirements engineers avoid unnecessary delays by preparing backlog items so that the team will be able to hit the ground running when those items come up for implementation. The lean guideline they follow is to wait to the Last Responsible Moment (LRM) to do so – the moment after which the cost of further delays will be unacceptable. By doing so, teams avoid the waste associated with stale requirements and are able to make do with less written documentation. Typically this guidance means that large features are prepared starting about half-way into the previous quarter or release cycle, while smaller requirements units (user stories) are analyzed about half-way into the previous sprint (a planning period or about 2 weeks). This preparation, also known as backlog refinement, means that items are understood well enough to be effectively estimated for planning purposes. In addition, with proper preparation, teams know enough about the requirements to avoid unnecessary errors. Activities involved in feature preparation include:
- Specification and planning: specifying, negotiating, estimating, prioritizing, and scheduling user stories and features
- Decomposition: story splitting (as discussed earlier)
- Stakeholder analysis: analyzing the impact on stakeholders and their needs
- Value stream mapping/process analysis: analyzing processes and discovering ways to improve outcomes (e.g., turnaround time)
- Business rules analysis: determining the business rules that must be supported by the solution
- Prepare wireframes: developing sketches of the user interface
- Prepare architectural runway: preparing the technical environment
You may recognize some of these activities (e.g., business rules analysis) as ‘classic RE’. They are, but the timing differs from the way they are carried out in waterfall. In agile development the activities are performed in an incremental, rolling manner as items come up in the backlog rather than all at once during a planning/preparation phase.
Triad
RE practitioners use Triad meetings to help business and development focus on the highest-value work throughout the development process. With the Triad approach, representatives of the business, development and testing get together whenever requirements (user stories) are discussed. The RE/BA facilitates negotiations between these parties to find the highest value requirements that can be delivered with the least amount of effort.
Acceptance Test-Driven Development/ Behaviour-Driven Development (ATDD/BDD)
Acceptance Test-Driven Development/ Behaviour-Driven Development (ATDD/BDD) is a set of practices for incrementally defining acceptance criteria before implementation begins. The specifications consist of specific scenarios described in a natural language tied to automated tests. RE or BA practitioners often play lead roles with respect to this key agile practice. For example, they own feature files and write the high-level feature and scenario descriptions contained within them.
Conclusions
Research and anecdotal evidence indicate that agile RE practitioners (whatever their title) improve outcomes by directing development towards work that will deliver the highest value to the business and customer throughout the product’s lifecycle. While it’s difficult to make a reliable prediction about the future of RE and BA as a formal role, we can say with confidence that the RE/BA competency has been shown to be vital to the success of agile organizations and is likely to remain so for the foreseeable future.
Footnotes
- [1] IIBA Global Research, Being Nimble: The Scalable Capability for an Organization to Sense and Respond to Change (IIBA, 2022), 5.
- [2] IIBA Global Research, 24.
- [3] IIBA Global Research, 24.
- [4] IIBA Global Research, 24.
- [5] IIBA Global Research, 21.
- [6] IIBA Global Research, 25.
- [7] The Agile Impact Report, Proven Performance Metrics from the Agile Enterprise (QSM Associates, Nov. 14, 2015), 5. rallydev.com/sites/default/files/Agile_Impact_Report.pdf
- [8] 17. Ken Schwaber and Mike Beedle, Agile Software Development with Scrum (Upper Saddle River, NJ: Prentice Hall, 2002), viii.
- [9] The Agile Impact Report, Proven Performance Metrics from the Agile Enterprise (QSM Associates, Nov. 14, 2015), 6. rallydev.com/sites/default/files/Agile_Impact_Report.pdf
- [10] Martin Glinz, Hans van Loenhoud, Stefan Staal and Stan Bühne, Handbook for the CPRE Foundation Level according to the IREB Standard - Version 1.0.0, (IREB, Nov, 2020), 9.
- [11] Podeswa, Howard, The Agile Guide to Business Analysis and Planning: From Strategic Plan to Continuous Value Delivery, (Addison-Wesley, 2021), 14.
- [12] For a comprehensive guide to MVP types, see, Podeswa, Howard, The Agile Guide to Business Analysis and Planning: From Strategic Plan to Continuous Value Delivery, (Addison-Wesley, 2021), 355-365.
- [13] For a comprehensive guide to story splitting patterns, see, Podeswa, Howard, The Agile Guide to Business Analysis and Planning: From Strategic Plan to Continuous Value Delivery, (Addison-Wesley, 2021), 422-432.
Howard Podeswa is a thought leader in the intersection of agile and business analysis. For over twenty years, he has been helping large organizations optimize their analysis and planning practices for agile software development approaches. He has authored a number of books that have become staple references for practitioners, most recently: The Agile Guide to Business Analysis and Planning: From Strategic Plan to Continuous Value Delivery - How Product Owners and Business Analysts (BAs) maximize the value of the product developed by the team, by integrating BA competencies with agile methodologies (Addison-Wesley Professional; 1st edition, 2021). Other works include UML for the IT Business Analyst (2009), and The Business Analyst’s Handbook (2008).
At Noble Inc., he has provided agile and business analysis services to clients worldwide, including the International Standards Organization (ISO), Moody’s, the Mayo Clinic, TELUS, TD Bank, LabCorp, the U.S. Food and Drug Administration (FDA), Mawer Investment Management Ltd., Bell Nexia, and REI Coop. He is currently advising IIBA on its Nimble initiative.
To contact Howard for in-house and public training and coaching services, please email howardpodeswa@nobleinc.ca or find him on LinkedIn @howardpodeswa