To assess the practical relevance of available RE research, we first analysed the literature from RE conference venues and summarised each work published between 2010 and 2016. We then asked practitioners, via survey research, to assess samples of those summarised publications with respect to the relevance of the research to their daily work and, further, which general research topics they think would be worthwhile pursuing. In the course of the survey, our respondents assessed 435 RE research papers published between 2010 and 2016 at the major RE research conferences and we received 154 responses from all over the world that provided a total of 2,164 ratings for individual research papers.
In this article, we summarise initial results, focusing on three important questions:
-
How do practitioners rate different approaches of carrying out research concerning RE, in terms of their relevance to industry practice?
-
Which research areas do practitioners rate most highly?
-
What research areas do practitioners believe are most important for the RE research
community to address?
This research remains a work in progress. Our hope is that in the not too-distant future, we will be able to distil general trends concerning the views of RE practitioners concerning RE research topics and provide important conclusions for both researchers and educators to guide RE research based on its perceived relevance in practice.
Overview
Respondents from different backgrounds and with different profiles participated in the survey. They work in a variety of industry sectors, as shown in Figure 1.
Each respondent was provided a sample of 15 randomly selected summaries of RE research contributions. Respondents were then requested to rate the relevance of each contribution relative to their particular needs utilizing the categories: (1) Essential, (2) Worthwhile, (3) Unimportant, and (4) Unwise.
The results of the ratings are provided in Figure 2. In general, practitioners considered a large share of RE research worthwhile or even essential. However, in 25% of the cases, respondents rated the research as unimportant, and in less than 5% of the cases, unwise.
It is interesting to note in Figure 3, below, that the distribution of ratings is quite similar in all of the academic conference venues in which the research has been published and presented. In other words, the respondents had a similar impression regardless of the venue. However, the type of track in which the work is presented, i.e. fundamental research (“research tracks”) or those concerned more with more practically relevant topics and applied research (“industry track”) seemed to have made a difference.
Without being advised concerning the track in which the paper was presented, practitioners rated those papers being presented in the industry track of the respective conferences higher than those being presented in the research track (Figure 4). Notably, the rating is not affected at all by the type of institutions the authors were affiliated to. That is to say, regardless of whether the authors were academics or practitioners themselves, their work received a more positive rating when it was presented in an industry track of the conference.
In summary, the track in which the papers are presented seems to be more indicative of the perceived practical relevance than the actual affiliation of the authors.
How do practitioners rate the different approaches of carrying out research in RE?
We analysed the research results from different perspectives – regardless of the actual topics addressed (such as whether a paper addresses a new requirements elicitation technique). We were particularly interested in two things: (1) which research methods received a higher preference in the ratings, if at all, and (2), in the case of experimental research, how are the types of subjects in the studies rated. The latter inquiry emerges from an ongoing debate in the research community on whether students are suitable subjects in experimental settings [3].
Without going into detail concerning all of the different ratings that were provided by the respondents, we summarise those finding that seem most relevant to us. When assessing the relevance of the different types of research, i.e. the way the research is carried out, solution proposals received the highest rating (with roughly 40% of the contributions being assessed as worthwhile), closely followed by observational studies (30% worthwhile). In contrast, controlled experiments received the lowest ratings (with roughly 10% being assessed as worthwhile, but nearly the same amount of ratings indicate that papers in this category are considered unwise). In other words, practitioners seem to value experimental research carried out in realistic real-life settings and they seem to think, at the same time, that artificial settings, e.g. in classrooms, are not only rather unimportant, but even irrelevant.
The ratings of the type of subjects show a similar picture, as shown in Figure 5. We distinguished between laypeople (students and others) and professionals (practitioners and academics). Not surprisingly, studies carried out with practitioners received the highest ratings. However, contrary to our intuitive expectation, professional academics rather than students received the worst ratings.
Which research ideas do practitioners rate most highly?
Concerning the perceived relevance of the contributions, we analysed various aspects in the papers. In particular, we distinguish the following aspects:
- Documentation: artefacts and/or models relevant to RE (e.g. topics related to business models or user stories).
- Process: process-related aspect of RE (e.g. topics related to a series of actions in a procedure).
- Quality: quality attribute in scope of the respective papers.
Figure 6 summarises the results of the ratings and structures them from top to bottom according to these three aspects considered in the papers.
Regarding the documentation category, we observed that most topics received a rather balanced distribution in the ratings: Although some topics received a higher rating, such as topics related to the specification of scenarios and user stories, most are perceived as rather relevant; with three exceptions. Topics related to use case diagrams, goal models, and process had mixed ratings. Use case models and goal models received the highest number of negative ratings. This lukewarm reception was rather surprising to us given their dominant presence in the RE research community.
Process-related aspects received a balanced distribution of the ratings. It was surprising to us that contributions related to formalising requirements were rated positively. One possible explanation for this is that a relatively large share of the respondents were in the automotive and avionic sectors where formal topics, such as formal verification, are part of the respondents’ daily work. This might also explain the rather high rating of the quality attribute verifiability.
Which research problems do practitioners think are most important for the RE research community to address?
We asked our respondents to indicate research concerns that, in their opinions, are most important for the RE research community to address. They provided their answers in free text form which we analysed manually.
Overall, we received a very diverse picture of recommendations and statements, for example the following:
— Business Analyst, 15 years of experience, Ecuador
In summary, when analysing the responses, we can come up with two principal key take-aways:
- The community should concentrate on discussing down-to-earth approaches in tune with real-life project environments (considering, e.g., existing tool environments).
- The community should include three principal topic areas in their solution approaches:
a. Requirements Management aspects (such as traceability)
b. Agile Requirements Engineering
c. Human Factors in Requirements Engineering
Space does not permit providing a detailed analysis and discussion concerning the various recommendations provided. However, the exemplary statement provided above reflects a general attitude – and a key take-away – that the RE research community should focus more on real-world experiences and provide an open avenue for new solutions to realistic problems.
Summary and Conclusions
In this article, we reported on the results from an ongoing study supported by the IREB Research Working Group with the goal of analysing the perception practitioners have on the practical relevance of available RE research contributions and what they believe would be worthwhile pursuing as a research community. We found that 30% of RE research is considered either unimportant or unwise to pursue from a practitioners’ perspective. A more detailed analysis showed that this especially holds for studies conducted with students or – in the view of practitioners even worse – with academics as subjects (e.g., in context of controlled experiments) rather than for studies relying on academia-industry collaborations with practitioners (e.g., in context of an industry case study). On the other hand, work that tends to come with prototypical implementations or that is otherwise directly applicable to the practitioners’ context tends to be favoured. Recommendations we received on topics to pursue focus on considering more practical, down-to-earth approaches while including human-centered, agile aspects.
While we are still analysing the data in more detail, key take-aways are that, researchers should be sensitive to the finding that 30% of ongoing research concerning RE is considered rather unimportant, based on the research results. Specifically, researchers might choose to perform research more directly focused on the perceived research needs of the Requirements Engineering community of practitioners. Furthermore, while every topic published in RE is suitable to find its champion in practice, it seems rather important how the research is actually being carried out. As a community, we should focus on realistic research settings rather than employing our developed techniques in artificial environments which abstract from real-life scenarios. Needless to say, this is often not as easy to implement in research-centric laboratories as it is to demand from an external perspective. Yet, the results permit further reflection concerning the manner in which RE research is performed.
With this article, we provided early insights into our currently ongoing data analysis in the hope to further foster necessary discussions in the RE community. A full data analysis will be reported soon with the aim to allow for more detailed discussions and recommendations.
One future perspective, depending on the feedback from the community concerning our work, is to establish this research format as a continuous survey under the umbrella of IREB research. This could yield a “trend radar” that continuously provides general trends in practitioners’ perceptions allowing for an improvement of the practical relevance of our research, teaching, and training curriculums so that our outcomes better fit the particularities and needs of practical environments.
Acronyms
ACM – Association of Computing Machinery
CrowdRE - International Workshop on Crowd-Based Requirements Engineering
ESEC/FSE - ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
ESEM – International Symposium on Empirical Software Engineering and Measurement
GI – Gesellschaft für Informatik (German Informatics Society)
ICSE – International Conference on Software Engineering
IEEE - Institute of Electrical and Electronics Engineers
IREB - International Requirements Engineering Board
ISERN - International Software Engineering Research Network
JIT-RE – International Workshop on Just-in-Time Requirements Engineering
NLP4RE – International Workshop for Natural Language Professing for Requirements Engineering
REFSQ – Working Conference on Requirements Engineering as a Foundation for Software Quality
References
- [1] D. Mendez Fernandez. Supporting Requirements Engineering Research that Industry Needs: The Naming the Pain in Requirements Engineering Initiative. In: IEEE Software, 2018.
- [2] X. Franch, D. Mendez, M. Oriol, A. Vogelsang, R. Heldal, E. Knauss, G. H. Travassos, J. C. Carver, O. Dieste, and T. Zimmermann, “How do practitioners perceive the relevance of requirements engineering research? an ongoing study,” in 25th IEEE International Requirements Engineering Conference (RE), 2017.
- [3] D. Falessi, N. Juristo, C. Wohlin, B. Turhan, J. Münch, A. and Jedlitschka, M. Oivo. „Empirical software engineering experts on the use of students and professionals in experiments.” In: Empirical Software Engineering Journal, 2018.
Daniel Méndez is an Associate Professor for Software Engineering at the Blekinge Institute of Technology, Sweden and Senior Scientist at fortiss, the research institute of the Free State of Bavaria for software-intensive systems and services. His research is on Empirical Software Engineering with a particular focus on interdisciplinary, qualitative research in Requirements Engineering and its quality improvement. He regularly publishes in various software engineering publications and has occupied several key positions in venues of the empirical software engineering community. He is a member of the ACM, the IEEE Computer Society, and the German association of university professors and lecturers, and he serves as the representative to ISERN, the International Empirical Software Engineering Research Network. Further information is provided at www.mendezfe.org.
Xavier Franch is Professor at the Universitat Politècnica de Barcelona (UPC-BarcelonaTech), Spain. He is Council Member and Full Member of the IREB association. He has published over 200 peer-reviewed papers in conferences and journals, many of them related to requirements engineering. He was Program Co-Chair of the RE’16 and REFSQ’11 conferences, and he belongs to the Editorial Board of the Requirements Engineering Journal (Springer) and Information Software and Technology (IST) journals, among others. He is coordinator of the Q-Rapids project and participates in the OpenReq project, both in the H2020 programme. He organizes workshops concerning requirements engineering for NLP4RE, CrowdRE, JIT-RE and others.
Andreas Vogelsang is a post-doc researcher in the software & systems engineering group at the Technical University of Munich. He works on model-based requirements engineering and participated in several research collaborations with industrial partners especially from the automotive industry. His research published was published in relevant international conferences like RE, REFSQ, or ICSE.