Options:

Rana Siadati Paul Wernick Vito Veneziano

Learning from history: The case of Software Requirements Engineering

‘A large elephant is in the room but we are not able or brave or willing enough to point at it’


This article describes initial conclusions of our research into how and why the requirements engineer’s role has changed over the years, with the longer-term goal of developing sound conceptual and interpretative tools and models to support their practice. To this end we address the general question of why and how to focus on RE, introduce politics and its close companion power as key conceptual viewpoints, and give a critical and comparative history of RE, considering a number of relevant methodologies. We then consider how to define requirements engineering, profile the requirements engineering practitioner and discuss other non-technical aspects of RE, all in the context of politics. We conclude by outlining our plans for progressing our research.

1 Introduction

In this paper, we identify politics and power as crucial components of requirements engineering (RE) and argue that the role it plays, especially when applied to the software industry, needs to be given greater attention than is currently the case. We highlight how, for a number of reasons and along the years, practitioners and researchers in RE have left politics substantially overlooked. Whilst many of the sources we use here are not recent, we contend that the underlying issue is still in need of urgent attention.

This article is an initial public airing of a broader ongoing research project aimed at understanding how and why the role of the requirements engineer has changed over the years. Our intention in this work is to improve the understanding of professionals and academics of the present state of that role, and to face (if not to drive) any future evolution of the discipline by providing sounder conceptual and interpretative tools and models than are currently available.

In order to better understand how and why RE is where it is, we posed four objectives/questions to guide our investigation.

Q1) (Re-)writing a critical and comparative history of RE: where is RE from?
Q2) Identifying a solid and yet flexible definition of RE: what is RE really about?
Q3) Profiling the professional practitioner: who does the job?
Q4) How RE affects and is affected by which socio-technological factors?

The structure of this paper follows the above questions. After some initial thoughts, Section 2 addresses the general question of why and how to focus on RE; it gives a critical and comparative history (by means of using the term in a non-technical sense to review the past in the hope of understanding the present and hopefully improving our future) of RE and introduces politics as a key conceptual focus. Identifying and consolidating such a conceptual key has been made possible by considering a number of relevant methodologies, which we also introduce and review in Section 2. The three following sections respectively address the remaining questions, in the light of this conceptual key, viz. politics. The following section will show how politics relates to other non-technical components in RE. A final section includes a conclusion and our plans for progressing our research.

2 Why and how to focus on RE: politics as the conceptual key and a necessary paradigm shift

Our initial research focused on existing logical and technological concepts and achievements in the broader practice of software engineering and how these affected (and still affect) the requirements engineer’s role, starting from the observation that increasingly more sophisticated and integrated requirements management tools and smarter approaches to development have played an important part in improving practice in recent decades.
However, we soon realised that the “technical” component, together with other factors such as time and memory loss within organisations, financial constraints, etc., are not the only source of change and that a quieter, deeper force seems to be driving the underlying evolution of RE practice over time.

After a long interview with an internationally-known British academic and practitioner, once the conceptual key of politics had been explicitly identified as the crucial factor to be considered, we felt that certain distracting information from our first literature survey was somehow preventing us from gaining a full appreciation of how insightful and even revolutionary a full appreciation of politics in RE could be.

Reading through the history of RE and noticing the continuity of mistakes, a new perspective emerged, in the light of which all of the information we had accumulated immediately became somehow clearer: a crucial and identifiable paradigm shift (Kuhn, 1970) had occurred in our research and our reading. Once this shift had occurred and we had managed to break out of our previous mindset, we were able to identify new approaches to investigate, and see an opportunity to model (perhaps mathematically; see Maiden et al., 2012), how politics could be controlled and used by practitioners as the basis for our subsequent work.

It is important to highlight here that we have adopted as a starting point in our research the traditional engineering-based definition of Requirements Engineering: that is, the first work phase/step of any software development project, on the assumption that defining goals and expected outcomes is essential for project success (e.g., Finkelstein, 1994; Maccari, 1999; Gebauer at al., 2008).

According to this vision, the main actor in RE is the requirements engineer, who is expected to learn about clients, their organisation and their needs, and to map, to analyse, to validate and to specify these against the client’s goals and a number of constraints such as finance, technical feasibility and deadlines. Such an exercise has two components; it demands both high technical skills and competence and an ability to negotiate across relevant parties: failing in either the technical or diplomatic tasks increases the risk of project failure. A traditional response in the past to issues in RE has been to increase the efforts in controlling and managing engineering-related activities. However, while we are all aware that these activities are necessary, we now realise that, by themselves, are not sufficient for project success (see Figure 1).

Figure 1: RE is where we invest more efforts in fixing problems

Over the years, many approaches and methods have emerged to give clients and practitioners easy (and sometimes cheap) answers to complex problems. Whilst it would sound ungenerous to classify agile methods as one of those answers, we observe that a focus on short interactions and user involvement is now sometimes adopted as the best approach (and whatever comes closest to a silver bullet) of modern software development practice.

Despite some progress, software practitioners are still some distance from fully mastering the art of eliciting, analysing and validating requirements in such a way that all parties would find it satisfactory. Research by the Standish Group (1995; see Figure 2), highlights in detail the main known problem causes affecting the overall quality of RE outcomes.

Figure 2: Technical VS non-technical problem causes in RE

As can been seen, the “technical” component (which we identify in both the “Lack of IT Management” and “Technology Illiteracy” categories) amounts for only about the 10% of identified problems. Even if we consider dealing with “Incomplete requirements” as a “technical” issue and include all of the “Other” category, we have only accounted for approximately one third of all problem causes.

The remaining two thirds of RE problems are non-technical in nature. This situation calls for self-critical reflection: in concentrating on technical issues, are practitioners in RE overlooking something crucial? In a way, what made it difficult for us to understand how far we were from a full appreciation of the “non-technical component” of the “dual” technical/interpersonal nature of RE was the fact that we seem always to only “talk” (and no doubt we do so quite extensively) about this almost mythical non-technical nature. RE has developed into a discipline where it is now commonly accepted that human, organisational and non-technical factors play a crucial role, but progress in that area is still slow.

It is our perception that, since 1995, both practitioners and academics have not done enough to address non-technical issues, and/or that some crucial factors that might possibly improve RE practice have not yet been effectively addressed.
But why has the focus of RE developments been mostly on the technical component? Is it because of a biased self-perception of the role we requirements engineers have to play (or which we are conventionally expected to play)? Do we see the role as more technical than managerial, based on an assumption (or fear?) that what is non-technical is per se uncontrollable and therefore of no interest to an engineer’s mind, even if by doing so we end up defining ourselves as the musicians playing on a sinking Titanic?[a] Attempting to solve this dilemma from a higher (even philosophical or cultural?) perspective goes beyond the scope of this paper: but helping us in reconstituting the professional identity of RE practitioners and researchers, and trying to improve it, is not.

We argue that a pragmatic answer to this issue is to be found in the conceptual key of “politics” (or the political dimension) in RE. This term can have several forms and meanings, but all relate to the roles people and their interrelationships play within any software development project from its earliest stages until the system is finally retired.

Alexander (2005) has said that we are aware that ‘projects need to know the roles involved, and the viewpoints of the stakeholders playing those roles’. But we think it is fair to underline how the acquisition and dissemination of this knowledge has been somehow obstructed by the dominant literature and ‘mindset’.

We report two cases here, which we consider symptomatic of an overly technophile approach to RE. The first is described by Sommerville et al (1997), who seem to consider it to be a problem to identify too many viewpoints when they write that requirements engineers ‘must balance the advantages of wider coverage offered by additional viewpoints against the costs of analysis and the problems of information management’. Given the problems stated in Figure 2 above, we would rather compare costs with costs, namely those costs necessary to gain and manage additional viewpoints with those which are a consequence of not doing so: failing to make sense and value different viewpoints would seriously undermine the key goal of the requirements engineer’s job, which is, in the final analysis, to understand the purpose of a system and the activities it will be used for.

Our second case is given by Easterbrook (2004), who acknowledges the intrinsic difficulty of such a job and still somehow suggesting it could only be a matter of applying a number of “techniques” when he writes that ‘such human activities may be complex, because they generally involve many different types of people, with conflicting interests. In such situations it can be hard to decide exactly which problem should be tackled, and it can be hard to reach agreement among the stakeholders. Requirements Engineering techniques offer ways of dealing with complexity, by systematically breaking down complex problems into simpler ones, so that we can understand them better’. We believe Easterbrook did not mean any “reductionism”[b] here, from politics to engineering: but his words might have offered in the past an easy way of avoiding seeing the non-technical aspects through to the bitter end.

Notwithstanding the above and other similar examples, the need to take people aspects into account emerged as the key conceptual finding of our research. In considering this, and because we are mainly talking about people roles and activities, we then identified two terms which properly represent the human dimension from different but related perspectives: politics (in a conventional sense) and power. This new focus on RE as an activity conducted by, between and for people means we can no longer avoid the burden of considering and understanding politics and power and how they affect both RE practice and self-awareness in practitioners.

It is noticeable how non-technical aspects other than politics and power, such as commercial awareness and viability, finance and project management, have been better received by both practitioners and academics in software RE. We suggest that this is either because they are considered to be more easily translatable into non-functional requirements, or just because it is nowadays accepted and expected that a requirements engineer should have both business knowledge and strong soft skills in these areas together with sound IT-related technical skills (Gallivan et al., 2004) in order to be able (and enabled) to negotiate with all stakeholders, even with the ones at the top of the pyramid (and negotiation is obviously part of the “politics” we are referring to). However, we contend that over-simplified views and considerations of such aspects have become predominant in how we train requirements engineers: such views may well have contributed to a selective blindness for power dynamics and how they do not always propagate linearly, from top to bottom, but rather follow more complex patterns.

We also feel that the adoption across the field of notations and technical language(s) from engineering (e.g., organigrams and UML) with limited ability to express, for example, ambiguity, to represent complex phenomena like organisations, can result in models that only capture a static, structural view, as if complex, changing webs of personal relationships in an organisation can be the object of just another engineering blueprint. This has in our opinion led to an implicit decision to ignore or abstract away how organisations become permeated by political relationships in a fluid, dynamic and sometimes unpredictable way.

If software requirements engineers and analysts are to be able to take advantage of any insight they could gain into politics and power relations within organisations – for example, who de facto decides which priorities come first when a conflict arises between two departments in the same company – a more appropriate (but unfortunately less easy) approach would demand more effort to understand relevant political dynamics and their source(s) behind the official organisational structure.

The approach we are proposing could be translated into a number of “non-technical” tasks, such as:

  1. Mapping organisational politics, i.e. the set of power relations occurring across (and despite) the official organisation chart;
  2. Mapping the political dimension of finance, i.e. who and what drives decisions behind the identification and allocation of time, budget, skills and resources;
  3. Mapping the politics of the technological dimension, by enquiring who makes decisions about what to develop, by using which development technologies, and why.

However, politics as actually occurring in an organisation should be seen as a fourth, more fundamental dimension informing the results of the above three tasks, one which sometimes subverts the ‘official’ power structures as might be documented as above. The informal chains of power must be understood as well as the formal when, for example, trading off requirements. Achieving this demands a conceptual change in how requirements engineers see themselves, in order to go beyond what Sommerville (2007) and most of us with him teach and preach to generations of old and new practitioners: ‘Engineers make things work. They apply theories, methods and tools where these are appropriate’.

3 Existing research into Politics, Power and RE

Bubenko, who has investigated in detail the non-technical component of the engineer’s job, seems reluctant to mention explicitly the political dimension. He states (1995) that ‘RE includes managerial, organisational, economic, social, as well as technical issues and problems’, and later that ‘Existing methods are not adequate for explicit capturing, and representing, in a structured way, “business and organisational knowledge” to be subsequently used to drive the information system development phases’, which is possibly what we mean by organisational politics. But an explicit use of the term ‘politics’ as a key word is still missing, and another opportunity to consider this seems to have been missed.

As another example, Nuseibeh (2000) says that ‘RE needs to be sensitive to how people perceive and understand the world around them, how they interact, and how the sociology of the workplace affects their actions’, a statement that we believe provides support for our contention that the elephant of human-human interaction – politics – is there in the room, but somehow, for some reason, we as a community are not able or brave or willing enough to point at it.

The effects of power relationships must also be taken into account. Dahl (1957) writes that ‘if so many people at so many different times have felt the need to attach the label power, or something like it, to some Thing they believe they observed, one is tempted to suppose that the Thing must exist; and not only exist, but exist in the form capable of being studied more or less systematically. […] That some people have more power than others is one of the most palpable facts of human existence [...] power is a relation among people [...] The main problem, however, is not to determine the existence of power but to make comparisons. Doubtless we are all agreed that Stalin was more powerful than Roosevelt in a great many ways, that McCarthy was less powerful after his censure by the Senate than before, etc. But what, precisely, do we mean? Evidently we need to define the concepts “more power than”, “less power than”, and “equal power”’. We tend to agree with the definition of power as an influence and politics as a process of demonstration of power, as provided by Milne et al. (2012), who believe that ‘if power is defined in terms of a potential force, then politics can be regarded simply as the study of power in action or alternatively in terms of a “process of bargaining and negotiation that is used to overcome conflicts and differences of opinion”. Power and politics have not been adequately considered’.

In fact, we strongly believe that they are currently far away from being adequately considered: even when the importance of politics and power is rightly stated, the temptation to reduce their effects to an engineering problem or to over-simplify things is strong, resulting in an incorrect and incomplete political analysis. Nelson, for example, in ‘The Politics of Agile’ (2008) explains politics as follows: ‘Politics is the process by which groups of people make decisions. In business, the process for building products requires hundreds of decisions every week. The challenge lies in who is involved, when do the decisions need to be made, how are decisions made, and who has the final authority. What is the real problem? On the surface, the problem seems to be that everyone is trying to wrestle control from the other.
Stop!
The real point of all of this is to build products people want to buy. Ultimately, we’re on the same side. Ultimately, we all want successful products that help us all make lots of money’. This is quite clearly a case of naivety, and possibly of weak political analysis, as it is assumed that everyone has the same agenda. A few authors have explicitly addressed the issue with greater subtlety. Maiden, whilst pointing to the topic of politics, warns (2012) that ‘the concept of ‘Politics’ is a phenomenon that we perceive to affect our work. However, our failure to define it puts us at a disadvantage, because it keeps us from developing strategies to respond to it in our requirements work’. Maiden continues that ‘modelling power relations between stakeholders should become a norm’: this is a statement to which we would subscribe, but bearing in mind that talking about politics in any organisation can be quite a frightening exercise and RE practitioners are currently likely not to have been prepared by their professional training with the necessary skills and tools to do so appropriately and efficiently. Milne et al (2012) cited Kentar (1979) who noted that ‘it is easier to talk about money – and much easier to talk about sex – than it is to talk about power’. This kind of taboo has affected and seems evident in the RE literature, where Milne et al. (2012) writes that ‘although notable work has been undertaken on the importance of social factors in RE, there has been relatively little direct consideration of the influence of power and politics’.

3.1 A need for better definitions?

Whilst we hope that the discussion so far has given enough evidence for the need to grant suitable consideration to power and politics in RE, one major question still demands attention: how could this consideration of power and politics be transformed into conceptual tools, approaches and techniques, so that practitioners can become familiar with them and start using them whilst pursuing overall good practice in their work?

Milne et al (2012) contribute to a working definition for both terms that goes in the direction of a possible answer of some use to engineers and other “techies”: ‘power can be viewed as a potential possessed by an actor, A, who can only be properly understood in terms of A’s relationships with others. So, given a relationship between A and B, if A has power over B then one or more of the following are possible:

Simply put, one-dimensional power involves A telling B what to do and B complying. Two-dimensional power is where A is able to define an agenda which B will act in accordance with without needing to be asked. For example, within an organisation this could be seen in a department where A has the power to determine both who attends decision-making meetings, and also to set the agenda for such meetings.’ Lukes S. (2005) added a further dimension to this understanding of power. This third dimension can be characterised as the ability to determine values, norms and ideologies.

Experts in software engineering would immediately recognise the above as easily applicable to their field. Almost twenty years ago, such a scenario seemed quite evident to Andriole (1998) too, who was one of the first to claim that “requirements management is a political process, not a technical one” as he highlighted the risk for naïve programming teams to think that, by writing good-quality code, they were satisfying customer requirements and not instead providing their employer with political cover, promptly available alibis and a more general ability to control expectations, should the inevitable happen and a need arises to focus or diffuse blame. And whilst there might be good reasons to pursue a political analysis, for example when ‘with the increasing complexity and uncertainty in requirements engineering (RE), the impact of power relationships between stakeholders become critical to the success of requirements engineering process, especially in requirements negotiation to resolve conflicting requirements’ (Yang et al., 2013). Krishna (2012) cites (and agrees with) Steve Yegge (2012) that other reasons might also be to be taken into account:

‘1) Software engineering has its own political axis, ranging from conservative to liberal. [..]
2) The notions of “conservative” and “liberal” on this political axis are specialised to software engineering. But they exhibit some strong similarities to their counterparts in real world politics.
3) Everyone in the software industry who does stuff related to programming computers falls somewhere fairly precise on this political spectrum, whether they realise it or not….. ’

4 What is RE really about? Politics and power underlying human activities

The history of software requirements engineering has always been a consistent concern for researchers attempting to define the field, starting from the very beginning of the so-called “Computer Era”. Nievergelt J. (1968) claimed that ‘it is necessary to look into the past because we have not yet had time to assess it and draw final conclusions from past experience. Yesterday’s ideas, out of fashion today, may well come back to prominence tomorrow’.

Boehm B. (2006) thought that ‘George Santayana’s statement, ‘those who cannot remember the past are condemned to repeat it,” is only half true. The past also includes successful histories. If you haven’t been made aware of them, you’re often condemned not to repeat their successes. Extensive studies of many software projects such as the Standish Reports offer convincing evidence that many projects fail to repeat the past successes’.

However, it is interesting to underline how difficult it is to define “the past” and its trends, no matter how recent these might be, and, even if we were able to track benchmark episodes, how easy it is then to fall down into an over-simplifying, naïve interpretation of facts. We see a typical example of this in an internal report written by the first author, in which she assumed that after the first electronic computer, ENIAC, was developed in 1946 by the United States Army for calculating artillery firing tables, computing and computers have since moved towards society and a commercial computer industry started worldwide in the 1950s[c] without wondering why this happened, as if it were an inevitable process. On reconsidering that report when preparing this article, it suddenly appeared clear to us how not just naïve but genuinely misleading some underlying assumptions we were making could have been. Was computing moving towards society, or was maybe a general shift happening in society, which somehow affected and determined how both the army and the computing industry became more open to society? In other words, were there not political factors that could be overwhelming the accidental occurrence of progress in science and technology? And this is just the surface: were we able to dig further, what kind of “politics” happened in practice? Who started moving the mechanism? Who exercised their power to let this happen the way it happened?

The same retrospective critical evaluation could be applied to academic thinking. When Wirth (2008) spoke of the “computer era”, might not his argument[d] also be based upon a set of assumptions that failed to take into account how politics and power have been the amongst the most influential drivers in the shift occurring in computing and IT, from academic labs and researchers to the commercial world? And what role would the political arena and actual politics and power (including the micro-decisions people in charge at different levels made) have played in dictating how, in the last instance, software would have needed to be understood, designed and managed the way it was outside the academic sphere?
Let us underline here that our suggestion that those overall political dynamics exist does not imply any kind of “conspiracy theory”. We have not assumed (nor to our knowledge has any RE academic) that someone in particular (or some group of people) behaved as a “big brother” who plotted and controlled whatever historical trends we can identify in software and system engineering: likely, these would have been the unpredictable result of a huge, complex political game, in which all relevant parties (from major players to large numbers of individual uses, each coming with their own agendas and their large or small amount of power) eventually contributed to the success or the failure of this project, to the rising or the decline of that process model, to the influence to be acknowledged to that approach, to the usage or the dismissal of that technology. But when this political game occurs at a lower level and within the smaller environments and organisations, as it usually is assumed to be the case for “ordinary” requirements engineering jobs and projects, we suggest that practitioners should pay attention and try to understand its dynamics, in order to ensure that any functional (and non-functional) software requirement would in last instance be truly “functional” (or at least not dysfunctional) to that organisation.

4.1 The emergence of RE as a recognisable discipline

The difficulty of developing and deploying software-based systems successfully is a longstanding issue, known since before the term “software crisis” was used during the first NATO Software Engineering Conference held at Garmisch in 1968[e]. Even then, achievement and deployment of suitable IT solutions to increasingly complex problems was becoming more and more difficult to ensure, by relying mostly on ever more powerful computers and despite acknowledging that organisations and socio-technical structures were constantly evolving. To quote Britton et al. (2006), ‘software systems were typically late, unreliable, difficult to maintain and did not do what was required of them’. Haigh (2010) reports that attendees at Garmisch, ‘having never met before, were shocked to realise that identical troubles plagued many different kinds of software and agreed that a ‘software crisis’ was raging’.

Wirth states that during the Garmisch conference ‘the difficulties and pitfalls of designing complex systems were frankly discussed’ (2008). Maybe this is the case, but our study of its proceedings failed to identify power and politics as items of the highest priority[f]. This did not prevent the NATO conferences (in 1968, and at Rome in 1969) and their proceedings from playing a crucial role in the development of software engineering as a discipline, but might this also come with a measure of responsibility for the neglect of politics and power in current RE practice? Whether or not this is the case, some authors reported unsolved issues with RE from its very beginning. Lamsweerde (2000), for example, considering what had happened since 1975, wrote that ‘software requirements have been repeatedly recognised during the past 25 years to be a real problem’. Possibly, this implied the need for RE to be recognised as a specific body of knowledge and practice, which needed to have its own dedicated spaces for discussion and reflection.

Mead (2013) confirms that ‘in the early 1990s it was recognised that requirements engineering was becoming a distinct discipline. The initial idea for a requirements engineering conference occurred within the software engineering community’. In the meantime, Alexander (1997) had noticed that ‘several major trends in technology have driven this progress in understanding the place of requirements in development:

In 1993 a first RE conference (the International Symposium on RE) took place in California, and one (the International Conference on RE, or ICRE) in Sorrento, Italy, followed by the establishment of the Requirements Engineering Journal in 1996. The merging of the Symposium and the ICRE announced in 2000, was finalised in 2002 with the International Requirements Engineering Conference (RE), which was then rebranded as the IEEE International Requirements Engineering Conference (RE’xx) in 2003: a series of conferences that continues successfully.

Although the term ‘Requirements Engineering’ continues in common usage and we have therefore employed it in this article, Easterbrook thought (2004) that it sounds a little awkward: ‘Requirements suggests that there is someone out there doing the ‘requiring’ – a specific customer who knows what she wants where in practice, there is rarely a single customer, but rather a diverse set of people who will be affected in one way or another by the system. ‘Engineering’ suggests that RE is an engineering discipline in its own right […]. The term ‘engineering’ also suggests that the outputs of an RE process need to be carefully engineered.’

By questioning whether the current discipline could be actually considered as “engineering” (and not rather a hybrid field with a strong “political” component), we share Mahoney’s (2004) doubts: ‘is it about the engineering of software? If so, by what criteria or model of engineering?’, but tend to disagree with him when he cites Shaw (1990) in the same paper suggesting that ‘Software engineering is not yet a true engineering discipline, but it has the potential to become one.’ Here again we see the temptation of assuming that the political dimension could somehow be absorbed or neutralised by the engineering one, as supported by those who think that ‘Engineers make things work. They apply theories, methods and tools where these are appropriate’ (Sommerville, 2007). We find Boehm’s approach to the field (in an attempt to redefine engineering in order to include a non-technical dimension[g]) a more suitable starting point for discussion, and agree with Nuseibeh (2000) that ‘the term engineering in RE serves as a reminder that RE is an important part of an engineering process, being the part concerned with anchoring development activities to a real-world problem, so that the appropriateness and cost-effectiveness of the solution can then be analysed. It also refers to the idea that specifications themselves need to be engineered, and RE represents a series of engineering decisions that lead from recognition of a problem to be solved to a detailed specification of that problem.’

We strongly believe that it is now time for practitioners and academics not only to realise that politics and power need to be considered and properly addressed in RE, if we are to make full sense of what Easterbrook (2004) claimed, when he wrote that ‘the real goal of an engineering process is to improve human activities in some way, rather than to build some technological artefact’ (and, by saying so, contradicting Sommerville), but also for us to argue whether we should not completely redefine RE itself as a primarily human practice rather than a mainly technical engineering process.

4.2 Politics and power in action: the subtle art of negotiation

Many authors have attempted a definition of requirements engineering or at least of its field of application. Cheng et al. (2007) believe that ‘requirements engineering is about defining precisely the problem that the software is to solve (i.e., defining what the software is to do), whereas other SE activities are about defining and refining a proposed software solution’.

Whilst most of us would substantially agree with the above definition, we argue here that, by doing so, we also agree to include within the semantics of the word “engineering” something which has nothing to do with the technical component we usually associate to it: because the problem that software is to solve within organisations and complex environments is in the final analysis not (only) an engineering problem, at least not in the same way in which the same organisations would likely look at other assets they might want to “engineer”: buildings, products, money and any other “physical” entity they might like to obtain and/or accumulate.

By means of software systems, organisations are trying to set up not just “physical” repositories of information and knowledge[h]: by channelling information by means of software systems, organisations (and individual people) try to establish and/or control networks of power, to alter the balances of forces, to ensure that knowledge and information can be easily translated into political currency for better managing dynamics between internal and external stakeholders.

We agree with Berry et al (1998), quoting Brooks (1995), that ‘the hardest single part of building a software system is deciding precisely what to build’ and continue with the definition by Boehm (1981), as cited by Sutcliffe (2002), that requirements are ‘designing the right thing’ as opposed to software engineering, which is ‘designing the thing right’. Sutcliffe in the same book explains nicely how requirements are “inseparable from us”, whenever we design software systems, across the four traditional phases into which we usually divide RE: requirements discovery, requirements analysis, requirements negotiation and requirements specification.

Wiegers (2012) is one of the few authors who claims that ‘requirements engineering is primarily a communication, not technical, activity. Communication problems can begin early on if project participants have different ideas of exactly what “requirements” are’.
An interesting and slightly different viewpoint is offered by Fricker et al. (2015), who focus on success measurements for RE according to quality, with criteria including explicit references to what we understand as politics and power: strategy, ability and willingness to make changes, willingness to defend solutions, relationships to users and between stakeholders.

Quality of RE service Quality of RE products
Business-technical alignment: fit with strategy, ability and willingness to make business changes, and management support. Quality of cost-benefits analysis: completeness and coverage of cost-benefit analysis, new benefits created by the new solution, and sufficient accuracy of cost estimates.
Stakeholder acceptance: awareness of business changes, extent of consensus, willingness to defend solution, and relationship to users. Argumentation of impact: diagnosis of existing solution, traceability of supported processes to problem to be solved and to system goals, and traceability of strengths and weaknesses of new solution to replaced solution.
Table 1: Success measurements for RE including and/or referring to politics (derived from Fricker et al, 2015)

The indirect definition they provide of RE is derived by ‘the quality of RE service, [which] refers to the effects a requirements engineer wants to achieve, and the quality of requirements engineering products, [which] refers to the work results delivered by the requirements engineer’.

Including users in any definition of, or related to, RE is always tricky, because of the uncertainty and ambiguity they usually bring with them. Complaining about users would be as meaningless for a requirements engineer as for doctors to blame their patients for making their job so challenging[i]. But whilst doctors can nowadays rely on objective clinical and bio-chemical tests and scans to better diagnose their patients' conditions, requirements engineers still have to rely on more subjective information – what users and stakeholders tell them supplemented by their own RE and domain knowledge. And, to be fair, whilst patients try to refer to what they “have” or are experiencing in terms of symptoms (which supposedly could be quite a straightforward task), users and/or customers have a more difficult and abstract goal to achieve in telling what they “would want”.

We think it is appropriate to restate here some of our earlier comments, in which we underlined that requirements engineers had for a long time worked on the basis of “capturing” requirements before a general acceptance emerged that requirements are elicited, not captured. As Easterbrook (2004) explains, ‘‘requirements’ suggests that there is someone out there doing the ‘requiring’ – a specific customer who knows what she wants. In practice, there is rarely a single customer, but rather a diverse set of people who will be affected in one way or another by the system. These people may have varied and conflicting goals. Their goals may not be explicit or may be hard to articulate. They may not know what they want or what is possible. Under these circumstances, asking them what they ‘require’ is not likely to be fruitful’.

The idea of having to manage conflicting goals is critical to RE. In the past, as confirmed by the report published as part of Software Engineering problems in the first NATO conference, this was rather interpreted (if not simplified) as a lack of understanding of System requirements on the part of customers and designers. Managing conflicts and the uncertainty resulting from them seems to be a familiar problem which we are still unable to face in RE, if things have not improved since Sutcliffe (2002) stated that ‘we have not improved our practice much since 1910 and mistakes in RE still cause problems’.

The extent of such problems has been clear for some time. Sommerville et al (1997) cited Gao (1979), Gibbs (1994) and Barlas (1996) when they stated that ‘it has been recognised for many years that problems with specifications are probably the principle reason for project failure where systems are delivered late, do not meet the real needs of their users, and perform in an unsatisfactory way’.

Easterbrook (2004) also observes that requirements (and programs!) ‘fail to work in the way we expect, they are unreliable, and sometimes dangerous, and they may create more problems than they solve’.

We have found no evidence that, notwithstanding many technophile temptations, any author has deliberately and explicitly understated the importance of stakeholders and the role they play in RE. It has to be acknowledged that such a shift in the focus from the product to the user, however difficult to accept for many who put “engineering” first, has found many supporters among well-known practitioners in this field, ready to pay extra attention to the human-centered design, and the challenges coming with it. Bubenko (1995), for example, believed that a crucial methodological challenge to address in RE is ‘to improve user-developer communication, much more than to expand the use of formal methods. Current models are not communicative, and descriptions cannot be fully understood by users and stakeholders. Current modelling methods are borrowed from SE, and do not document the reasoning and the rationale behind solutions suggested. Users often sign-off requirements specifications without fully understanding them’.

Sommerville (op cit.), who has been one of the strongest musketeers of the “engineering” approach, himself admitted that ‘there is no such thing as a typical user’, and then listed a number of them (e.g., doctors, nurses, patients, administrators, maintenance engineers). We cannot, however, agree with the solution he proposed to this problem, to limit the number of stakeholders to be taken into account: ‘if too many viewpoints are identified, it is difficult to manage the large amount of information generated and prioritise the requirements’.

But why should we disagree with it? Would it not be right for us to limit the viewpoints if we want to eventually obtain our final engineering product and the technology, which would eventually support it? Easterbrook (2004) gives us the answer: ‘the purpose of a banking system is not to be found in the technology used to build such systems, but in the business activities of banks and day-to-day needs of their customers. The purpose of an online flight reservation system is not to be found in the details of the web technology used to implement it, but in the desire by passengers for more convenient ways of booking their travel, and the desire of airlines to provide competitive and profitable services’. In other words, we cannot avoid the challenge of understanding in detail all stakeholders (or at least enough of them to get a satisfactory result), their field/domain and their needs if we want to produce a working system that is of any value to them.

On this ground, we further agree with Easterbrook that ‘software on its own cannot be said to be of ‘high quality’ or of ‘low quality’, because quality is an attribute of the relationship between software and the purpose for which it is used. This means that requirements engineering plays a critical role in understanding and assessment of software quality’.

Bergman et al. (2002) observed that ‘two key assumptions frame traditional RE. One is that requirements exist ‘out there’ in the minds of stakeholders (user, customers, clients), and they can be elicited through various mechanisms and refined into complete and consistent specifications. The second is that the key stakeholders operate in a state of goal congruence, in which there is widespread and coherent agreement on the goals of the organisation’.

The second assumption above would in a way seem to justify those requirements engineers who feel they are forced to guess (or “induce”?) users’ needs and to produce some imaginary requirements so that the developed system can eventually be perfectly designed against a complete set of requirements, as the original ones provided by users or based on our knowledge of the domain can be incomplete: but even if and when all stakeholders agree that the set of requirements is sufficiently complete there is always a possibility that some important requirements will be missed and cause problems later on during development or after deployment. For this reason, getting to know users and their needs has now been recognised as a very significant and essential step in RE practice, and to “imagine” them is now always seen as a poor choice in the same way as it would be a poor choice “not to imagine” them as parts of the system we are going to develop. The RE needs to know their systems’ users the same way as they are expected to know the technologies they are going to adopt and exploit to eventually create a system for those users. And it is a good starting point to acknowledge that, in most cases, we don’t really know who the actual clients/users are. Alexander’s “Stakeholders – Who is Your System For?” (2003) provides food for thought: ‘there isn't one person who you can unequivocally call 'the client'. Engineering projects like the West Coast Main Line upgrade [a major UK rail infrastructure project] nowadays run into billions of pounds, take many years to complete, and involve many people who could be considered to be 'clients', 'users', or 'customers'. In fact, since public money is involved, aren't we all customers in some sense? The question is, therefore, who can legitimately claim to hold a stake in a project's territory; or, to put it the other way round, whom should engineers seeking to define the requirements for a project consult about their viewpoints?’

This leaves us with two options to consider: to decide that, with regards to the example above, it is not possible to find all the users and try to negotiate with them and we must therefore accept the risk that this failure to capture complete information could result in serious issues arising as a result of missed requirements, or to increase efforts to identify all of the significant users and stakeholders involved. Leaning towards the first option is Sillitti (2005), who believes that ‘in agile method, the problem of multiple stakeholders is solved by reducing their numbers to one, a single person that represents all the stakeholders involved in the project’. This seems to us to be a naïve approach because it denies or simply removes the main issue a requirements engineer needs to address, i.e. how to negotiate the conflict of goals: does the presence of one individual really address all the political issues in an organisation? Who is this person? What is his/her agenda? Who selected them as the user representative and why did they do this? And what are the agendas of other stakeholders involved in the selection? The implications of this approach demand an investigation which we intend to undertake in future stages of our research.

We suggest that one primary purpose of (although a major problem in) requirements elicitation is the negotiation which results from the identification and resolution of conflicts among stakeholders, rather than to deny or abstract them. As one of the authors said to another at a recent meeting, ‘you can negotiate once you have a party and if you don’t have a party then you can’t negotiate!’

Although organisations evolve and different types of hierarchy can coexist within the same organisation, the use of primarily hierarchical models to describe organisations are still in use (see Schwarz, 2006) and a person or a group of people located towards the top of this multi-dimensional pyramid typically has/have more power than those located nearer the bottom. Within these hierarchical models, though, does power always inform organisations through executive boards, chiefs, line managers, supervisors, team leaders, etc., with a direction that clearly mimics the force of gravity, from the top to the bottom? Whilst this kind of structure could be seen as a sort of “survival of the fittest” feature and an advantage in any organisation as everyone is clear about whom to get commands from, does it always function linearly from top to bottom?

We argue that in reality power in an organisation is not always representable as simply as this, and in the resulting confusing situation, there are many negotiation paths to be considered, besides the obvious difference between those who pay the money and those who actually do the job. In this situation, an understanding by the RE of the art of negotiation is clearly needed and practiced. Finkelstein (1994) confirms that assessing organisational settings is part of ‘the necessary preconditions for effective Requirements Engineering […]. The process of stakeholder analysis involves identifying those individuals or roles who should have a voice[j] in the requirements engineering process. These may be clients, users and other beneficiaries, they may also be people involved in subsequent design, implementation, maintenance of the system. Stakeholder analysis involves understanding their responsibilities, capacities and the organisational relations between them. This analysis serves as a map for subsequent information gathering and a means of interpreting the information provided and its status’. It is a pity they did not produce a notation to set out this knowledge and to capture political power relationships, a task we are already undertaking in the next phase of our research.

Returning to the need for negotiation, even the technophile Sommerville (2004) accepts that ‘stakeholders negotiate to agree on a set of requirements for the system. Generally, there are a greater number of desirable requirements than can be implemented so decisions have to be made at this stage about leaving out requirements and modifying requirements to result in a lower-cost system.’

Fricker et al. (2015) seems to have spotted the elephant in the room when they stated that ‘management support is critical to align the software with the strategic goals of the organisation’, and therefore appreciated that, in order to be able to find the right person or group of people to negotiate with, we have to get to know the client organisation as well as other stakeholders. In the table below, we set out the list of requirements negotiation techniques the previously mentioned authors have identified: it is useful to note how they refer explicitly to “power analysis”.

Technique Description
Conflict management Discovering and resolving conflicts among stakeholders and between stakeholders and development team
Handshaking The review and discussion of implementation proposals to align the planned implementation of the software system with stated and unstated stakeholder needs
Negotiation analysis Analysing possible negotiation outcomes and selecting a value-creating, fair agreement
Power analysis Analysing power and influence of stakeholders and planning how to interact with them
Prioritising Ranking the requirements to obtain an order of how they shall be addressed by the project work
Strategy alignment Aligning requirements with company strategy, for example through explicit traceability
Variant analysis Analysing and selecting alternative features or ways of solving a problem
Win-win negotiation Structured, possibly tool-supported approach to identification of options for agreement and selection of the appropriate option
Table 2: List of requirements negotiation techniques (Fricker et al., 2015)

5 Who does the RE job? The politically-vested analyst and engineer

So, then, who is this semi-mythical practitioner we consider to be the well-rounded requirements engineer? Until a few years ago, it was supposed (or expected) to be ‘an individual […] capable of utilising knowledge to synthesise effective solutions’ (Davis and Hickey, 2002). And knowledge, according to the same authors, meant (1) knowledge of the problem domain, (2) knowledge of existing solutions, and (3) knowledge of processes, methods and tools of the practice of requirements engineering. ‘Only recently a fourth area of knowledge has been identified: knowledge of how to decide which processes, methods and tools make most sense as a function of certain aspects of the problem domain’.

In the same paper, Davis and Hickey state that practitioners tend to reproduce a standardised code of practice, which in our opinion poorly addresses the deeper and more problematic aspects of the above fourth area of knowledge. More generally, practitioners tend to be quite conservative in pursuing innovative practice and recommendations as they become available, and not coherent in their quest for quality and insight: the title of a paper Davis wrote in 1996 perfectly summarises the critical issues we have highlighted so far: “Practitioner, heal thyself”!
And whilst an outside might legitimately wonder whether “we practice what we preach” in RE (Davis and Hickey, 2002), our provocative answer would be: yes, unfortunately we do.

We have grounded this powerful but depressing critique of the current state of RE on the criteria adopted by two major professional bodies RE in Western Europe, the International Requirements Engineering Board (IREB) and the British Computer Society (BCS). Each of these currently accords a recognised status to requirements engineers and adopts training handbooks to instruct certification programmes for business analysts.
These textbooks demand that candidates study modules such as “Requirements Engineering” to achieve their certification awards. IREB uses “Requirements Engineering Fundamentals” by Klaus Pohl and Chris Rupp, 2nd Edition 2015 (from now on referred to as REF), whilst in the UK the BCS trains and assesses practitioners on the basis of “Business Analysis” by Debra Paul, James Cadle and Donald Yeats, Third Edition 2014 (which we hereafter refer to as BA).

According to REF, ‘the requirements engineer as a project role is often at the centre of attention. She is usually the only one who has direct contact with the stakeholders and has both the ability and the responsibility to become as familiar as possible with the domain and to understand it as well as possible. […] To be able to fulfil all of her tasks, the requirements engineer needs much more than process knowledge’. The skills which the authors consider necessary for a qualified RE include not only analytic thinking, empathy, communication skills, moderation skills, etc., but also “conflict resolution skills”.

Whilst all other skills (and the process knowledge, obviously) are the object of training and education courses, it is unclear whether resolving conflicts is learned (and if so, how) or if it is assumed that a requirements engineer needs to be born with it.

On the other hand the BA, whilst emphasising “commercial awareness”, seems to emphasise the “selling” of a resolution rather than the actual process of resolving underlying issues.

Both books, in our opinion, do not give sufficient weight to politics and power, and if requirements engineers had been trained only from those books, they would have hardly heard of politics and power as issues to be considered alongside their practice, and might even after qualification be unaware, and possibly unable, to cope with the associated challenges. This is a cultural and mental boundary that we believe needs to be broken down: an imprisonment in Plato’s cave which must be escaped.

6 How a politically aware RE relates to other non-technical factors

How do we intend to transform what might seem more like a mission than a research programme into a set of achievable research tasks that will meet our goal of helping REs in industry with their practice? We already anticipate that this will not be an easy journey, because, as can been seen from the authors we have mentioned above, politics has been addressed by very distinguished academics and practitioners, who were however left in doubt on how to transform and implement it into RE practice.

Maiden (2012) states that he and Milne were not the first to talk about power and politics and to attempt a formalisation of their influence: to confirm that statement, he cites sources from the 1980s. In fact, the paper from Dahl (1957) which we have already mentioned above is much older and includes a proposed mathematical notation to represent power relationships between people.

Still, and quite correctly, Maiden expresses his surprise about ‘the absence of concrete requirements engineering guidelines and techniques with which to understand the structure of relationships between project stakeholders (Power) and the decision-making process informed by these relationships (Politics)’.

Despite technical advances in the intervening decades since Dahl wrote, it is still becoming further evident that, since other non-technical aspects (finance, human factors, cultural backgrounds, etc.) can be causally correlated to political and/or power factors, any proposed model and its underlying notation would need to be sufficiently sophisticated to ‘reflect the social characteristics of [a] complex system’ (Milne and Maiden, 2012).
We will also need to take into account Bergman et al. (2002), who according to Milne and Maiden were the only ones who presented ‘a comprehensive argument for reconceptualising RE as a political process’. As a result of such expected complexity, we would not be surprised if the model Maiden was hinting at, and which we will try to develop in the next stages of our research, would be less engineering-like and more of a political nature.

6.1 Industry and Academia

There is another non-technical factor in which politics plays a role, and this is the traditionally difficult relationship between industry and academia, and the big gap all parties are mutually complaining about.

Although Boehm (2007) suggests that ‘keeping courses and courseware continually refreshed and up-to-date’ would be of help, the reality is somehow less easy to frame and handle.

For example, Berry et al. (1998), when focus on the research-practice gap, say that ‘slowly, we are bridging the gap between requirements engineering research and practice. The gap is still large, but we have a few more practice-validated methods and tools in our pockets, and the bridge building continues’.

On the other hand, Fricker (2015) states that ‘many of these techniques did not become industrial because they were not practicable or ineffective when used in real-world projects’: overall, it seems typical of cases in which some power relationships have consolidated into ideological positions that have their supporters and are grounded on arguments of a political nature.

A more nuanced view of the theory/practice gap is taken by Sadraei et al. (2007), who cite Kaindl et al. (2002) to highlight that ‘researchers have continued to develop new techniques and methods, but practitioners have experienced difficulty in applying them … the challenge is thus to fill the gap between the research … and … practitioners.’

Closer to Flicker was Bubenko’s (1995) criticism, when he wrote that ‘many of our academic researchers in SE and RE continue to tackle “interesting and researchable” RE problems, often without knowledge of relevant issues and problems in practical life. Our academic education in SE and RE is, in many places, not quite adequate for practical work in the field. There is a need to look at software and requirements engineering education, indeed, as a question of producing engineers instead of computer scientists[k]’.

We tend to agree, this time, with Wirth (2008) when he states that ‘we teach, learn, and perform only what is immediately profitable, what is requested by students. In plain words: we focus on what sells’.

However, we feel that Wirth’s analysis is incomplete, as is that of Shaw (2000) when she suggests that ‘education for software developers should prepare students differently for different roles, infuse a stronger engineering attitude in curricula, help students stay current in the face of rapid change, and establish credentials that accurately reflect ability’; neither explicitly mentions the role of politics as one of the main drivers for producing “different roles”, “rapid change”, “credentials”.

Finally, to improve RE in the real world any changes we propose to the discipline with new notations, processes etc. must be usable by practitioners with a minimum of training and conceptual readjustment on their part. One of the criteria of success of our research will be observable changes in RE as an activity.

6.2 Professional Bodies

Another political issue to be addressed in our investigation is that of the role of professional bodies in RE, and the ‘professionalisation’ of the practitioner. Shaw (op cit.) argues for the professionalization of software engineering (and by extension of software RE) by licensing its practitioners; a trend towards explicit qualification for a role which is increasingly happening in other fields.

Any answer given to this issue is in itself a political statement, depending on where the emphasis has been placed in determining the relevant body of knowledge to be tested in candidates to ‘qualify’ them as practitioners. As we observed above when comparing the two certifications from IREB and BCS, training professionals within a certain given perspective brings with it an implicit political agenda arising from membership of a particular professional body (cf. Kuhn, 1970). Given that all parties to a negotiation come with their own explicit and/or implicit beliefs about political and power relationships, background and training will in part determine how these qualified practitioners will then interact with stakeholder groups.

Bergman et al (2002) state that ‘the idea that requirements engineering is a political process is already well established, although it is often overlooked in the prescriptive literature’. They also suggest that failing to acknowledge the political dimension would lead to overall failure, especially in those ‘large-scale development [projects that have] remained a high risk proposition despite huge advances in computing and telecommunications technologies’, therefore highlighting how the technological component is not the main problem.

From our reading, it is evident that RE professional bodies as “political” entities are not including politics in their training agendas. The questions asked by Shaw (1998) about the need for professionalisation of the software engineer (‘is there a need? yes; are we ready? no’) are in our view equally applicable to the political aspects, and here they demand different answers: “is there a need for politics in RE? yes; are we ready? not yet!”
It is important to recognise that the requirements engineering job is not “just a technical job”. We agree with Yang (2013), as cited by Hoh et al. (2001), when Yang said that ‘improving requirements modelling methods, supporting techniques and tools is not enough to deal with the increasing complexity in the RE process, especially in requirements negotiation’.

It is time to acknowledge that a specification document is not only a technical specification, that a representation is not only the outcome of a modelling exercise, and –above all – that any agreement, including the agreement of a set of software requirements, comes with a social and political dimension that must reflect the real human world, whatever the underlying technical achievement.

7 Conclusion and future plans

We have now started research into identifying elements of a workable notation for documenting political and power relationships within a typical RE project, and through which the necessary negotiation processes might be contextualised and understood.

When a first version of this notation is ready, we will adopt the Impact Evaluation methodology, based on the retrospective counterfactual analysis of what difference an intervention would have made in outcomes: this methodology has been mostly developed within the policy-making sector and became a main stream methodology after the World Bank adopted it extensively (Gertler et al., 2011). In our case, we aim at inviting experienced practitioners to use our approach to analyse their previous projects and consider whether this would have helped them in their work, particularly in identifying and resolving political and power-related issues which they had to address. We have already identified a number of volunteers to help us in this work; if you would like to join us, please contact the corresponding author via email (r.siadati@herts.ac.uk).

In the meantime, we will be seeking more information and feedback from experts in both industry and academia. There is still a steep mountain for us to climb but we believe we have already identified an important mountain, and what has made it so challenging to climb in the past. Hopefully, once we have reached the peak, the view will be an important contribution worthy to be shared with the RE community.

Footnotes

References