Requirements engineers and testers – communicate!

Requirements engineering (RE) and V&V are cornerstones in software engineering. But to really get the most benefits the two activities must be aligned – simply claiming to be agile is not enough. We report challenges and practices involved in RE-V&V alignment from a large industrial case study.

This paper is referred to as the “grand publication” among the researchers involved in the EASE Theme D research project. As can be seen on the paper, a lot of people were involved in the study. The data collection had already begun when I joined the project, and I immediately started with conducting interviews. The data analysis was a challenge, finding consensus among all involved researchers was obviously not easy in such a large qualitative endeavor – but we did our best in a series on monthly working meetings.

In total, we interviewed 30 engineers in six companies, capturing the perspectives of different roles in software engineering. The interviews were truly deep with many open questions, leading to hours of audio recordings to transcribe. We based the discussions on the conceptual model presented on the top of this blog post, showing how RE and V&V are related across the three dimensions: 1) abstraction level, 2) time, and 3) product lines.

an index of the Interviews – in a visual graph

We used quite a complex coding scheme to bring structure to the qualitative data. Not only were the data coded into different categories, we also coded relations represented by ‘contributes to’ and ‘prerequisite for’ as well as their negated counterparts ‘not’. The coding process was validated to ensure consistency among the researchers. All this work lead to information in a graph structure. To enable an overview of the data, stored in MS Excel, I wrote a script to convert it to GraphML – a format that we then visualized in the yEd graph editor. Quite practical actually, we used it as an index of the interviews and interacting with the graph was the first step in several analyses.

Empirically found challenges and Practices

We report 16 alignment challenges, including poor requirements specifications, inadequate test processes, and insufficient traceability. Furthermore, we report 27 practices that support alignment organized in 10 categories, e.g., requirements engineering processes and structured change management. Finally, we report which challenges are addressed by which practices. All findings are based on empirical data from the interviews, and the study is to a large extent the foundation of the RET workshop series.

We conclude the paper by sharing four high-level observations. First, the human side is central, especially communication and coordination between requirements engineers and testers. Second, quality and accuracy of the requirements is a important starting point for testers. Third, the size of the company is a key variation factor for both challenges and practices of alignment, i.e., small organizations have a set of problems and solutions that differs from the really large ones. Fourth, safety-critical projects have external motivation to align requirements engineering and testing (traceability needed to certify the products!), while other projects typically have only internal motivation to align RE and V&V.

Implications for Research

• The first empirical study on alignment of requirements engineering and testing, in practice defining the field.
• An example of a very large case study in software engineering, presented as an illustrative example in the textbook by Runeson et al. (2012).
• A novel approach to get a graphical overview of large amounts of interview data.

Implications for Practice

• There are many challenges in aligning RE and V&V, but also many established practices – use our study as a roadmap for process improvement.
• A simple but really imperative challenge is lack of communication – make sure requirements engineers and testers frequently talk to each other.
• The main variation factor for which challenges an organization might face is its size.

Elizabeth Bjarnasson, Per Runeson, Markus Borg, Michael Unterkalmsteiner, Emelie Engström, Björn Regnell, Giedre Sabaliauskaite, Annabella Loconsole, Tony Gorschek, and Robert Feldt, Empirical Software Engineering, 19(6), pp. 1565-1616, 2014. (link, preprint)

Abstract

Weak alignment of requirements engineering (RE) with verification and validation (VV) may lead to problems in delivering the required products in time with the right quality. For example, weak communication of requirements changes to testers may result in lack of verification of new requirements and incorrect verification of old invalid requirements, leading to software quality problems, wasted effort and delays. However, despite the serious implications of weak alignment research and practice both tend to focus on one or the other of RE or VV rather than on the alignment of the two. We have performed a multi-unit case study to gain insight into issues around aligning RE and VV by interviewing 30 practitioners from 6 software developing companies, involving 10 researchers in a flexible research process for case studies. The results describe current industry challenges and practices in aligning RE with VV, ranging from quality of the individual RE and VV activities, through tracing and tools, to change control and sharing a common understanding at strategy, goal and design level. The study identified that human aspects are central, i.e. cooperation and communication, and that requirements engineering practices are a critical basis for alignment. Further, the size of an organisation and its motivation for applying alignment practices, e.g. external enforcement of traceability, are variation factors that play a key role in achieving alignment. Our results provide a strategic roadmap for practitioners improvement work to address alignment challenges. Furthermore, the study provides a foundation for continued research to improve the alignment of RE with VV.