Measure the process, not just the output!

Requirements engineering (RE) metrics are mostly artifact-oriented, i.e., they tend to target the actual requirements specifications. However, also the RE process that leads to the artifacts is possible to measure – and relevant process metrics facilitate improvements. We present the results from an interview study on RE process metrics used in industry, and practitioners’ proposals of new metrics.

Requirements engineering (RE) is hard. And in agile development contexts RE happens all the time, a.k.a. integrated RE. Without appropriate metrics it is difficult to assess the quality of the process, thus all improvement efforts have to rely on gut feeling. But there are metrics to understand the RE process – some are already used in industry, while others are ready to be implemented! We present a case study on RE process metrics in a large telecom company, focusing on requirements scoping.

This paper had a long road to publication. Full paper versions rejected at REFSQ, ESEM, EASE and PROFES – but PROFES at least accepted it as a short paper. The paper has its merits, it always had, but now the presentation format is finally suitable for publication. My guess is there will be an extension of this work into a full paper sometime soon.

Requirements scoping metrics

Large companies selling products on a consumer market must continuously manage new feature requests. Cleverly deciding which features to implement is critical in such market-driven contexts, i.e., requirements scoping. To increase the flexibility of the development process, many companies allow scope fluctuations – features are moved back and forth between the scope and the backlog. However, previous work shows that there is limited support available to support the requirements scoping.

In this study, we explored what kind of metrics are currently used to evaluate the scoping process. We collected a large set of empirical data through interviews with 22 engineers. We also asked the interviewees to suggest new metrics that could be useful. In total, we report 26 RE process metrics – only 5 of them are currently used. Our findings suggest that metrics for the scoping process is underdeveloped. The metrics requested by most interviewees were: 1) how often features are moved between software releases, 2) how often features are moved backwards in the decision model (backward transitions in the figure on top of the page), and 3) how much time is spent on analyzing features that never get implemented.

Implications for Research

• Most requirements engineering metrics target artifacts, but also the process is highly relevant.
• Practitioners suggested numerous novel process metrics, suggesting we identified an underdeveloped area of requirements engineering.

Implications for Practice

• The requirements scoping process in market-driven development context might occupy a lot of time – measure it to understand how and why.
• There might already be several ideas for scoping process metrics among your co-workers – investigate if there are any low-hanging fruits ready to be implemented.

Krzysztof Wnuk, Markus Borg, and Sardar Muhammad Sulaman. An Industrial Case Study on Measuring the Quality of the Requirements Scoping Process, In Proc. of the 17th International Conference on Product-Focused Software Process Improvement, 2016. (link, preprint)

Abstract

Requirements management is an important part of the software development process. Decision making and requirements scoping occupy central roles in helping to develop products that are demanded by the customers and ensuring company strategies are accurately realized in product scope. Many companies experience continuous and frequent scope changes and fluctuations but struggle to measure the phenomena and correlate the measurement to the quality of the requirements process. We present the results from an exploratory interview study among 22 participants working with requirements management processes at a large company that develops embedded systems for a global market. Our respondents shared their opinions about the current set of requirements management process metrics as well as what additional metrics they envisioned as useful. We present a set of process metrics that describe the quality of the requirements scoping process. The findings provide practical insights that can be used as input when introducing new measurement programs for requirements management and decision making.