Private or public? Testing quality aspects is difficult…

This experience report is a follow-up on our paper presented at the 1st Int’l Workshop on Requirements Engineering and Testing (RET): Revisiting the Challenges in Aligning RE and V&V: Experiences from the Public Sector. Back in 2014, we reported experiences based on the comprehensive list of RE <-> V&V challenges listed in our earlier paper Bjarnason et al. (2014) – Basically we went through the entire list and commented to what extent we see it as a challenge in a specific project in the public sector in Sweden. For this experience report, we returned to the same case project, and focused on one of the major challenges: testing quality requirements.

Both the experience reports from 2014 and 2016 we wrote with the RET workshop in mind. As I’ve been co-organizing the RET events, I knew contributions from industry would be highly valuable at the workshop. Twice Jacob Larsson from Capgemini has been kind enough to share his experiences from industry, and to help writing it up in a paper. A win-win I believe, good for us as researchers, and a rewarding experience for Jacob.

This paper goes beyond simply reporting problems in industry. I consider this work to be a fairly solid workshop contribution, as we complement the experiences from the first author by a document analysis by the second and third authors, i.e., Thomas and I looked for evidence of the claims expressed by Jacob by investigating the project documentation. Furthermore, we reviewed the body of research literature to identify potential solutions that could be used by the development organization under study to mitigate the identified challenges. Thus, instead of inventing new solutions, we considered work already published by other researchers – probably something the community should do more often!

Implications for RESEARCH:

• We highlight that highly complex systems-of-(information) systems are developed in the Swedish public sector – with huge monetary stakes!
• We show an example of how the rigor of an experience report can be improved by an independent artifact analysis.
• Instead of inventing new solutions, we present a matchmaking of experienced challenges and already published solutions by other researchers.

Implications for Practice:

• Consider whether the following five challenges to testing Quality Requirements (QR) apply to your case:
  1. QRs keep evolving although test activities are ongoing
  2. Test managers must know the business models well
  3. QRs are not quantified
  4. QRs are not prioritized
  5. Some operational states are hard to simulate in a test environment
• Based on a review of literature, we suggest addressing the five challenges by:
  1. Integrated Requirements Engineering (Sommerville, 2005)
  2. Twin Peaks model (Cleland-Huang et al., 2013)
  3. QUPER model (Regnell et al., 2008)
  4. Architecturally significant requirements (Chen et al., 2012)
  5. Virtual plumblines (Cleland-Huang, 2008)

Jacob Larsson, Markus Borg, and Thomas Olsson. Testing Quality Requirements of a System-of-Systems in the Public Sector - Challenges and Potential Remedies, In Proc. of the 3rd International Workshop on Requirements Engineering and Testing, 2016. (preprint, slides)

Abstract

Quality requirements is a difficult concept in software projects, and testing software qualities is a well-known challenge. Without proper management of quality requirements, there is an increased risk that the software product under development will not meet the expectations of its future users. In this paper, we share experiences from testing quality requirements when developing a large system-of-systems in the public sector in Sweden. We complement the experience reporting by analyzing documents from the case under study. As a final step, we match the identified challenges with solution proposals from the literature. We report five main challenges covering inadequate requirements engineering and disconnected test managers. Finally, we match the challenges to solutions proposed in the scientific literature, including integrated requirements engineering, the twin peaks model, virtual plumblines, the QUPER model, and architecturally significant requirements. Our experiences are valuable to other large development projects struggling with testing of quality requirements. Furthermore, the report could be used by as input to process improvement activities in the case under study.