Options:

Daniel Méndez Xavier Franch Andreas Vogelsang

What is the Relevance of Requirements Engineering Research to Practitioners?

Preliminary Results from an Ongoing Study


**Introduction** Requirements Engineering (RE) has been the subject of promising research for the last four decades, supported by a strong and vibrant research community that contributes to the development of new requirements tools, methods, and approaches. Yet, providing the research that industry needs remains an ongoing challenge [1]. There is little clarity concerning the research that is needed. Meanwhile, academia is undertaking a broad spectrum of research on various topics, often without knowing what research is of interest to or needed by practitioners. This information is needed in order to further align the selection of research areas with their practical relevance. Because of this concern, we initiated the RE-Pract Project [2], an international collaboration of ten researchers. The project is a research activity of the International Requirements Engineering Board (IREB) Research Working Group and has a goal to conduct an empirical study to assess the relevance of RE research as perceived in industry.

To assess the practical relevance of available RE research, we first analysed the literature from RE conference venues and summarised each work published between 2010 and 2016. We then asked practitioners, via survey research, to assess samples of those summarised publications with respect to the relevance of the research to their daily work and, further, which general research topics they think would be worthwhile pursuing. In the course of the survey, our respondents assessed 435 RE research papers published between 2010 and 2016 at the major RE research conferences and we received 154 responses from all over the world that provided a total of 2,164 ratings for individual research papers.

In this article, we summarise initial results, focusing on three important questions:

  1. How do practitioners rate different approaches of carrying out research concerning RE, in terms of their relevance to industry practice?

  2. Which research areas do practitioners rate most highly?

  3. What research areas do practitioners believe are most important for the RE research
    community to address?

This research remains a work in progress. Our hope is that in the not too-distant future, we will be able to distil general trends concerning the views of RE practitioners concerning RE research topics and provide important conclusions for both researchers and educators to guide RE research based on its perceived relevance in practice.

Overview

Respondents from different backgrounds and with different profiles participated in the survey. They work in a variety of industry sectors, as shown in Figure 1.

Figure 1: Industry backgrounds of the respondents.

Each respondent was provided a sample of 15 randomly selected summaries of RE research contributions. Respondents were then requested to rate the relevance of each contribution relative to their particular needs utilizing the categories: (1) Essential, (2) Worthwhile, (3) Unimportant, and (4) Unwise.

The results of the ratings are provided in Figure 2. In general, practitioners considered a large share of RE research worthwhile or even essential. However, in 25% of the cases, respondents rated the research as unimportant, and in less than 5% of the cases, unwise.

Figure 2: Overall ratings, regardless source of research contributions.

It is interesting to note in Figure 3, below, that the distribution of ratings is quite similar in all of the academic conference venues in which the research has been published and presented. In other words, the respondents had a similar impression regardless of the venue. However, the type of track in which the work is presented, i.e. fundamental research (“research tracks”) or those concerned more with more practically relevant topics and applied research (“industry track”) seemed to have made a difference.

Figure 3: Ratings of research contributions in different venues.

Without being advised concerning the track in which the paper was presented, practitioners rated those papers being presented in the industry track of the respective conferences higher than those being presented in the research track (Figure 4). Notably, the rating is not affected at all by the type of institutions the authors were affiliated to. That is to say, regardless of whether the authors were academics or practitioners themselves, their work received a more positive rating when it was presented in an industry track of the conference.

Figure 4: Ratings by affiliation [academic, industry, mixed] and track ["yes" indicates industry track, "no" indicates another track such as the research track].

In summary, the track in which the papers are presented seems to be more indicative of the perceived practical relevance than the actual affiliation of the authors.

How do practitioners rate the different approaches of carrying out research in RE?

We analysed the research results from different perspectives – regardless of the actual topics addressed (such as whether a paper addresses a new requirements elicitation technique). We were particularly interested in two things: (1) which research methods received a higher preference in the ratings, if at all, and (2), in the case of experimental research, how are the types of subjects in the studies rated. The latter inquiry emerges from an ongoing debate in the research community on whether students are suitable subjects in experimental settings [3].

Without going into detail concerning all of the different ratings that were provided by the respondents, we summarise those finding that seem most relevant to us. When assessing the relevance of the different types of research, i.e. the way the research is carried out, solution proposals received the highest rating (with roughly 40% of the contributions being assessed as worthwhile), closely followed by observational studies (30% worthwhile). In contrast, controlled experiments received the lowest ratings (with roughly 10% being assessed as worthwhile, but nearly the same amount of ratings indicate that papers in this category are considered unwise). In other words, practitioners seem to value experimental research carried out in realistic real-life settings and they seem to think, at the same time, that artificial settings, e.g. in classrooms, are not only rather unimportant, but even irrelevant.

The ratings of the type of subjects show a similar picture, as shown in Figure 5. We distinguished between laypeople (students and others) and professionals (practitioners and academics). Not surprisingly, studies carried out with practitioners received the highest ratings. However, contrary to our intuitive expectation, professional academics rather than students received the worst ratings.

Figure 5: Rating of types of subjects involved in experimental settings.

Which research ideas do practitioners rate most highly?

Concerning the perceived relevance of the contributions, we analysed various aspects in the papers. In particular, we distinguish the following aspects:

Figure 6 summarises the results of the ratings and structures them from top to bottom according to these three aspects considered in the papers.

Regarding the documentation category, we observed that most topics received a rather balanced distribution in the ratings: Although some topics received a higher rating, such as topics related to the specification of scenarios and user stories, most are perceived as rather relevant; with three exceptions. Topics related to use case diagrams, goal models, and process had mixed ratings. Use case models and goal models received the highest number of negative ratings. This lukewarm reception was rather surprising to us given their dominant presence in the RE research community.

Figure 6: Rating of the topics addressed in RE research contributions, including the ratings of documentation-related aspects [top], process-related aspects [middle], and quality-related aspects [bottom].

Process-related aspects received a balanced distribution of the ratings. It was surprising to us that contributions related to formalising requirements were rated positively. One possible explanation for this is that a relatively large share of the respondents were in the automotive and avionic sectors where formal topics, such as formal verification, are part of the respondents’ daily work. This might also explain the rather high rating of the quality attribute verifiability.

Which research problems do practitioners think are most important for the RE research community to address?

We asked our respondents to indicate research concerns that, in their opinions, are most important for the RE research community to address. They provided their answers in free text form which we analysed manually.

Overall, we received a very diverse picture of recommendations and statements, for example the following:

“I think the community is too focused on producing short term results for scientific publications […] I currently see the RE community as a very closed family […] same authors on same topics year after year, with very little chance for real experiences to be presented.”
— Business Analyst, 15 years of experience, Ecuador

In summary, when analysing the responses, we can come up with two principal key take-aways:

  1. The community should concentrate on discussing down-to-earth approaches in tune with real-life project environments (considering, e.g., existing tool environments).
  2. The community should include three principal topic areas in their solution approaches:
    a. Requirements Management aspects (such as traceability)
    b. Agile Requirements Engineering
    c. Human Factors in Requirements Engineering

Space does not permit providing a detailed analysis and discussion concerning the various recommendations provided. However, the exemplary statement provided above reflects a general attitude – and a key take-away – that the RE research community should focus more on real-world experiences and provide an open avenue for new solutions to realistic problems.

Summary and Conclusions

In this article, we reported on the results from an ongoing study supported by the IREB Research Working Group with the goal of analysing the perception practitioners have on the practical relevance of available RE research contributions and what they believe would be worthwhile pursuing as a research community. We found that 30% of RE research is considered either unimportant or unwise to pursue from a practitioners’ perspective. A more detailed analysis showed that this especially holds for studies conducted with students or – in the view of practitioners even worse – with academics as subjects (e.g., in context of controlled experiments) rather than for studies relying on academia-industry collaborations with practitioners (e.g., in context of an industry case study). On the other hand, work that tends to come with prototypical implementations or that is otherwise directly applicable to the practitioners’ context tends to be favoured. Recommendations we received on topics to pursue focus on considering more practical, down-to-earth approaches while including human-centered, agile aspects.

While we are still analysing the data in more detail, key take-aways are that, researchers should be sensitive to the finding that 30% of ongoing research concerning RE is considered rather unimportant, based on the research results. Specifically, researchers might choose to perform research more directly focused on the perceived research needs of the Requirements Engineering community of practitioners. Furthermore, while every topic published in RE is suitable to find its champion in practice, it seems rather important how the research is actually being carried out. As a community, we should focus on realistic research settings rather than employing our developed techniques in artificial environments which abstract from real-life scenarios. Needless to say, this is often not as easy to implement in research-centric laboratories as it is to demand from an external perspective. Yet, the results permit further reflection concerning the manner in which RE research is performed.

With this article, we provided early insights into our currently ongoing data analysis in the hope to further foster necessary discussions in the RE community. A full data analysis will be reported soon with the aim to allow for more detailed discussions and recommendations.

One future perspective, depending on the feedback from the community concerning our work, is to establish this research format as a continuous survey under the umbrella of IREB research. This could yield a “trend radar” that continuously provides general trends in practitioners’ perceptions allowing for an improvement of the practical relevance of our research, teaching, and training curriculums so that our outcomes better fit the particularities and needs of practical environments.

Acronyms

ACM – Association of Computing Machinery
CrowdRE - International Workshop on Crowd-Based Requirements Engineering
ESEC/FSE - ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
ESEM – International Symposium on Empirical Software Engineering and Measurement
GI – Gesellschaft für Informatik (German Informatics Society)
ICSE – International Conference on Software Engineering
IEEE - Institute of Electrical and Electronics Engineers
IREB - International Requirements Engineering Board
ISERN - International Software Engineering Research Network
JIT-RE – International Workshop on Just-in-Time Requirements Engineering
NLP4RE – International Workshop for Natural Language Professing for Requirements Engineering
REFSQ – Working Conference on Requirements Engineering as a Foundation for Software Quality

References