Software engineering is an applied research field. Applied research should have practical relevance. Why is this often not the case? We synthesize what has been expressed in the community and share our views. We need to be more relevant!

This paper was neither part of a bigger publication plan nor part of any ongoing research project. But since the topic is fundamentally important, and close to my heart, I couldn’t resist joining the study when Vahid Garousi invited me. I was active on the topic during 2018, as co-chair of the Software Engineering Research and Industrial Practice workshop and I was a panelist on “When are Software Testing Research Contributions, Real Contributions?” at ICST2018 in Västerås. While the panel discussions didn’t quite go in the directions I hoped for, this (very long!) paper surely covers most of that.

Defining practical relevance

Before we could even start working on this, we of course had to clarify what we mean by relevance. This is surely not a simple concept! One of the major contributions of this paper is really our attempt to define both relevance and other closely related terms. We did this by looking beyond the SE community, as the topic the debate is more mature in neighboring research areas such as information systems and management science. Below follows a summary of useful definitions found in the literature.

Terms	Definition/Interpretation in SE Research	Source
Utility	Fitness for some purpose or worth to some end.	Merriam-Webster Dictionary
Impact	How does your research change the world?	Zeller (2019)
Rigor	The correct use of any method for its intended purpose.	Wieringa and Heerkens (2006)
Relevance	Significant and demonstrable bearing on the matter at hand.	Merriam-Webster Dictionary
Academic relevance	The value provided for the specific academic field.	Rautiainen et al. (2017)
Practical relevance	How does your research influence SE practice, i.e., lead to change, modification, or confirmation of how engineers think, talk or act?	Kieser et al. (2015)

Practical relevance is multi-faceted and a difficult construct to deal with. We like an approach proposed by Benbesat and Zmud (1999) in the information systems community. First, the authors state that “research in applied fields has to be responsive to the needs of business and industry to make it useful and practicable for them” (we agree with that!). Second, they continue by presenting four dimensions of relevance in research.

• [PROBLEMS] Does the research address the real problems or challenges that are of concern to professionals?
• [IMPLEMENTABLE] Does the research produce knowledge and offer solutions that can be utilized by practitioners?
• [CURRENT] Does the research focus on current technologies and business issues?
• [ACCESIBLE] Is the research paper easily readable and understandable by professionals?

Another reference, this one from management science, that we strongly agree with is Toffel (2016), stating that “I define relevant research papers as those whose research questions address problems found (or potentially found) in practice and whose hypotheses connect independent variables within the control of practitioners to outcomes they care about using logic they view as feasible”.

Our definition of practical relevance

Based on our literature study, and in particular Benbesat and Zmud (1999) and Toffel (2016), we present our view on dimensions of practical relevance in software engineering.

Dimension	Description
Focusing on real-world SE problems	Research topic and research question(s) should improve SE practices and/or address problems found (or potentially-found) in practice (presently or in the future).
Applicable (Implementable)	Hypotheses should connect independent variables within the control of practitioners to outcomes they care about using approaches they view as feasible. Based on 1) Realistic inputs / constraints 2) Controllable independent variables 3) Outcomes practitioners care about 4) Feasible and cost-effective approaches
Actionable implications	The research provides implications that would encourage practitioners to act based on the findings.

Many researchers conduct curiosity-driven research of discovery nature. This is important research and can lead to breakthroughs in directions that industry would not pursue on their own. Such research comes with high risk and many ideas never work in practice. Research has an important role in shaping the future, but we argue that research targeting plausible problems of the future still benefits from extrapolating needs from current SE practice. The figure below puts it all in context. The main message is that higher potential relevance increases the chances of impactful research.

A multi-vocal literature review on SE relevance

The goal of this study was to collect everything meaningful that has been written about the relevance of software engineering research. The topic has of course not only been discussed in peer-reviewed publications, thus we conducted a multi-vocal literature review to also include other sources. We selected 54 sources and synthesized what the SE community had discussed related to the topic.

The focus of the paper is root causes, i.e., why is software engineering research not relevant? and improvement suggestions, i.e., what can we do to address the low relevance? As the figure shows, we focus improvement suggests on the conceptual ideas (as this is primarily discussed in the sources) but report some hands-on improvement mechanisms where applicable.

What did we find from the synthesis of 54 sources? We boiled it down to the 16 causes in the figure below, organized in causes involving 1) individual researchers and 2) collaborations. We also identified causes that originate in the academic system, the SE research community, and the funding system. The causes are sorted by the number of sources that highlight them, suggesting their order of importance. The top-3 items are:

• Researchers start with simplistic views (and often, wrong assumptions) about SE in practice and thus formulate inappropriate research problems.
• The lack of connection with industry causes low relevance. SE researchers have the choice of maintaining active connections with industry practice or not. This is largely a matter of preference.
• Wrong identification of research problems that industry eagerly needs to be solved. Studying problems that are poorly connected with practice causes low relevance.

Using the same approach to synthesis, we also report improvement suggestions from the community. Most of them are related to choices by individual researchers. The top-3 items are:

• Researchers must accept the need for a paradigm shift in research approaches, from purely theoretical to experimental by more usage of approaches such as action research, case study research, and field studies. In-vivo evaluations are needed to understand how things work in practice.
• Choosing relevant (practical) research problems. The higher the relevance of the research questions, the higher likelihood that the end result will be relevant.
• Many researchers and practitioners are advocating for more industry-academia collaboration. Although such collaborations are non-trivial, there are best practices – and if successful, the relevance will be supported.

Implications for industry

• Help your local SE researchers. By working together, much more relevant research will be conducted – less resource waste.
• Support “industry track” initiatives at SE conferences. Join forces with academics and submit relevant papers. Volunteer to review papers to bring the industry perspective.
• Check what is going on in academic SE research from time to time. Encourage researchers that pursue relevant research – tell them, they would love to hear it!

Implications for research

• Research relevance in SE has been discussed since the 1980s. This paper aggregates the discussion.
• The SE community should conduct and report more relevant research. We report the main reasons why there are low relevance contributions and how to improve the situation.
• Keep the debating the topic, and compare with discussions in other fields such as information systems, management – and even nursing.

Vahid Garousi, Markus Borg, and Markku Oivo. Practical Relevance of Software Engineering Research: Synthesizing the Community's Voice, Empirical Software Engineering, 25, pp.  1687–1754, 2020. (open access)

Abstract

Software engineering (SE) research should be relevant to industrial practice. There have been regular discussions in the SE community on this issue since the 1980's, led by pioneers such as Robert Glass. As we recently passed the milestone of "50 years of software engineering", some recent positive efforts have been made in this direction, e.g., establishing "industrial" tracks in several SE conferences. However, many researchers and practitioners believe that we, as a community, are still struggling with research relevance and utility. The goal of this paper is to synthesize the evidence and experience-based opinions shared on this topic so far in the SE community, and to encourage the community to further reflect and act on the research relevance. For this purpose, we have conducted a Multi-vocal Literature Review (MLR) of 54 systematically-selected sources (papers and non peer-reviewed articles). Instead of relying on and considering the individual opinions on research relevance, mentioned in each of the sources, the MLR aims to synthesize and provide the "holistic" view on the topic. The highlights of our MLR findings are as follows. The top three root causes of low relevance, discussed in the community, are: (1) Researchers having simplistic views (or wrong assumptions) about SE in practice; (2) Lack of connection with industry; and (3) Wrong identification of research problems. The top three suggestions for improving research relevance are: (1) Using appropriate research approaches such as action-research; (2) Choosing relevant (practical) research problems; and (3) Collaborating with industry. By synthesizing all the discussions on this important topic so far, this paper aims to encourage further discussions and actions in the community to increase our collective efforts to improve the research relevance. Furthermore, we raise the need for empirically-grounded and rigorous studies on the relevance problem in SE research, as carried out in other fields such as management science.