Impact analysis – Multifaceted and beyond code

Change impact analysis (CIA) is fundamental when developing a safety-critical software system. Still, few empirical studies exist. We present a case study on CIA in an evolving automation system. We report how much time it takes, the main challenges involved, and what developers actually think about doing CIA.

Developers adhering to safety standards such as IEC 61508 need to be on top of the evolving software. This is far from trivial, as foreseeing how changes to the code affects the rest of the system is really tough in complex systems – side effects often appear! It is not enough to simply modify the code and run the regression tests, the safety standards require developers to analyze the impact before committing changes. CIA is not restricted to source code, any development artifact might be impacted by changes, e.g., the safety requirements, the test code, the architecture, and the user manual… As well as quality aspects such a performance, security, or the memory footprint.

A major part of my PhD studies was connected to developing ImpRec, a recommendation system for change impact analysis. As part of this work, I interviewed developers and managers in Sweden and India about both ImpRec and CIA in general – in total 14 interviews. This is the first exploratory study on the interview data, mainly based on qualitative methods. What actually triggered this publication right now are the reviewers’ comments on the main ImpRec publication – including “do developers see CIA as something good or a tax that must be paid?” and “how do you measure that your CIA tool support actually works?”. We address three research questions in the paper.

RQ1 – How much effort is spent on CIA?

In the first RQ, we explore both i) how often engineers do CIA, ii) how much time is spent on an individual CIA task, and iii) what the greatest challenges are. Our results suggest that developers in the case company spend roughly 50-100 hours on CIA per year, corresponding to one CIA per week with 1-2 hours effort each. The figure below shows the interviewees’ estimates, expressed as minimum, average, and maximum effort needed. A senior manager shared his rule of thumb: “CIA takes roughly 10% of the total time to resolve an issue”.

We report several CIA challenges. Two major challenges are i) communicating to developers that CIAs are expensive, but necessary and beneficial, and ii) navigating the large document space accompanying the source code, especially the requirements documentation. These two challenges have not received that much attention in the research literature before… But they need some careful thought in a safety-critical development context. Make sure the developers understand the necessity of CIA, and try to make them get some personal development benefits from doing it well – our approach in ImpRec is to let developers reuse historical CIAs in a recommendation system. Other CIA challenges include: remembering details of past CIAs, building trust in your own CIAs, and making sure the developers follow CIA guidelines.

RQ2 – Impression and perceived importance

The second research questions investigates the engineers’ engineers’ connotation to “change impact analysis”, i.e., the emotional association carried by CIA (in contrast to the denotation, i.e., theexplicit or literal meaning). We map the connotation to the perceived importance of the CIA task. Bluntly, do engineers like the CIA work they have to do, and do they feel it carries meaning in their development project? Could it even be experienced as an awful task “just for show”?

The figure on top of this post shows an overview of the results. We note that all perspectives are covered by the interviewees, although there is a tendency to consider CIA important – this is obviously an encouraging result. While this is a qualitative study, we note that our interviewees tend to consider the CIA to have either a neutral or positive connotation. Comments include: “CIAs are a healthy sign”, “it shows that we do complex software engineering”, “CIAs are just part of the job”, and the more negative: “CIAs are a too heavy construct in our organization” and “CIAs could be done in a better way”.

RQ3 – Supporting CIA

We finally asked the engineers what kind of CIA support they would like to see. It comes as no surprise that the most common reply was “tool support” – we asked interviews after all! Several interviewees would like to see an increased level of automation, including identifying impact beyond the source code level. One engineer explicitly questioned the unstructured input format of the CIA report (free text), envisioning what a format better adapted machine-readability could enable. Other suggested improvements included: better training for new employees, improved guidelines, and a stricter CIA process including additional reviews.

If you plan to introduce CIA support, you also need to think about how to evaluate it. Do your tools really add value to the organization? There is not much research available on how to measure the utility of assisted CIA in the research literature, especially not beyond accuracy metrics on the source code level. Prior to the interviews, we invented two metrics and calculated them for the recent CIAs conducted by the interviewees – then we asked them for comments. We two metrics were:

• TIME = time between a developer is assigned an issue and the first CIA report is submitted. TIME thus targets the effort required to do a CIA.
• MODS = the number of modifications on a CIA report after its first submission. MODS was suggested as a proxy of the difficulty of a specific CIA task.

The engineers clearly explained that the metrics are too naïve, CIA is a complex activity and the number of confounding factors are just too many. Six interviewees directly rebutted TIME, as it really depends on workload, number of parallel tasks, priorities etc. While the opinions on MODS were more balanced, the interviews show that is no a reliable metric for assessing difficulty – the tool used to enter CIA reports is really bad, and many later edits are just tiny spelling corrections and indenting. Our work clearly shows that further work on how to measure CIA is needed before we can quantitatively evaluate CIA tools. The CIA activity is intermixed with general development activities, thus it is hard to isolate CIA to perform reliable measurements. At the moment we appear to suffice with qualitative evaluations – but hey, that’s maybe not so bad?

Implications for Research

• Change impact analysis (CIA) is not only about pointing out what code is impacted – a major challenge is to understand impact on various documents.
• Engineers particularly consider impact on requirements to be difficult – more research is needed.
• Measuring utility of CIA tools is non-trivial, as CIA is hard to isolate from general development work – quantitative measurement must be done with care.

Recommendations for practice

• Acknowledge that change impact analysis takes time – 10% of the total time to resolve an issue appears to be a reasonable first estimate.
• Make sure the CIA outcome is stored in a structured format in adequate tools – avoid frustrated engineers and enable future data analysis.
• Communicate the importance and usefulness of CIA – try to make the developers see a personal benefit from doing it well.

Markus Borg, José Luis de la Vara, and Krzysztof Wnuk. Practitioners' Perspectives on Change Impact Analysis for Safety-Critical Software - A Preliminary Analysis, In Proc. of the 5th International Workshop on Next Generation of System Assurance Approaches for Safety-Critical Systems, 2016. (link, preprint)

Abstract

Safety standards prescribe change impact analysis (CIA) during evolution of safety-critical software systems. Although CIA is a fundamental activity, there is a lack of empirical studies about how it is performed in practice. We present a case study on CIA in the context of an evolving automation system, based on 14 interviews in Sweden and India. Our analysis suggests that engineers on average spend 50-100 hours on CIA per year, but the effort varies considerably with the phases of projects. Also, the respondents presented different connotations to CIA and perceived the importance of CIA differently. We report the most pressing CIA challenges, and several ideas on how to support future CIA. However, we show that measuring the effect of such improvement solutions is non-trivial, as CIA is intertwined with other development activities. While this paper only reports preliminary results, our work contributes empirical insights into practical CIA.