Search, browse or trace?

Change impact analysis (CIA) requires engineers to find critical pieces of information. Code, requirements, models, and test cases – all these artifacts must be easily findable for CIA to be successful. So how do engineers seek information? Also, as traceability is often motivated as a CIA enabler – do engineers actually benefit from the trace links?

Finally I got the chance to finish this paper! I have a big interest in software engineering findability and enterprise search, but it took a while before I found the opportunity to publish these results. I collected the data analyzed in this paper already before I deployed and evaluated ImpRec – the data waited for me a long time! I conducted the interviews already in 2013/2014 but I did the analysis first in the end of 2016. On the other hand, the interviews cover an ever-present challenge and the general findings remain relevant. By the way, this paper is also closely related to the study we did on engineers’ perception of CIA in the same case company.

The information landscape of a big software engineering project is nasty. It’s big, heterogeneous in terms of artifact types, and everything keeps changing. Still, engineers need to stay on top of what is going on. In safety-critical engineering this gets even more important, as Change Impact Analysis (CIA) is a mandated formal activity, i.e., engineers need to specify what is going to be affected when the system is modified. Traceability researchers often motivate costly trace maintenance by arguing that it improves and supports CIA, for example De Lucia et al. (2008) and Cuddeback et al. (2010). But to what extent do engineers value the existence of trace links when conducting CIA? And how do they practically seek the information needed to conduct CIA?

We studied a safety-critical software engineering context with a rigorous CIA process. We conducted interviews with 14 engineers in Sweden and India to explore their thoughts about traceability and CIA, as well as their general information seeking strategies. The interview guide was carefully designed, interviews recorded and transcribed, and the analysis involved a validation step. We initiated this study with three main questions: 1) How is traceability used in CIA?, 2) How do engineers seek information when doing CIA?, and 3) How do engineers decide when the CIA is complete? (We also investigated what kind of CIA support was available… But we didn’t find much beyond process guidelines and training, in line with our previous survey on CIA in industry.)

Traceability and CIA

Very early during the interviews we asked the engineers how useful they considered traceability to be when conducting CIA. We wanted to get an answer to this question before biasing them too much. The interviewees had very different opinions as shown below.

We then continued the interviews by asking them to describe the different trace links that were used in CIA… And the engineers mentioned many different links. The figure below shows an aggregation of all types of trace links reported. There are obviously a lot of traceability in the case company, but still it is apparently not useful to everyone. Some engineers make use of it, and some find it not useful. Explanations include “too small system components to bother about trace links” and “trace links cause so much extra work”.

Information seeking behavior in CIA

To understand how engineers seek information in CIA, we let the interviewees comment on a fixed list of seeking alternatives. First of all, we notice the big variations in seeking preferences. No engineers report the same approaches to information seeking. Engineers do a mix of searching, browsing, and tracing. Also, only one engineer (C) uses all the alternatives we mentioned. We also notice that interviewee C is the only one that browses the document management system (DMS) – what a wasted seeking opportunity!

Finally, we were curious to know how engineers decide that a CIA is completed. Kind of a sensitive question to ask in a safety-critical context, of course you can always investigate further! Holbrook et al. (2013) referred to a similar idea as “satisfaction assessment” when concluding a traceability matrix – I like the term and we use it too. And what did we find regarding satisfaction assessment in CIA? Almost nothing. It is really done in an ad hoc fashion and only based on experience and gut feeling. I think this could motivate a follow-up study in itself. What do safety engineers and external assessors say about this? And maybe this is a promising use case for recommendation systems such as ImpRec, i.e., highlighting if an engineer missed reporting some impact that others frequently specified? Overall, this study was fun to, from data collection to reporting, and I look forward to presenting it in Buenos Aires.

Implications for Research

• Do not take the usefulness of trace links for granted. All engineers do not value traceability as much as you might think!
• Engineers appear to prefer less rigid types of support more than formal approaches and tools. Flexibility is considered critical.
• Satisfaction assessment for change impact analysis would be a sensitive future research topic – but very important.

Implications for Practice

• Engineers have different seeking preferences. Make sure the tools and databases provide good support for both searching and browsing – as well as tracing using stored links.
• Engineers are inclined to oral communication and intertwine looking for documents and people. Make also co-workers easily findable!
• Make sure the engineers understand the information organization, policies, and metadata – consider the end-user perspective and make information findable.

Markus Borg, Emil Alégroth, and Per Runeson. Software Engineers' Information Seeking Behavior in Change Impact Analysis – An Interview Study, In Proc. of the 25th IEEE International Conference on Program Comprehension, pp. 12-22, 2017. (link, preprint)

Abstract

Software engineers working in large projects must navigate complex information landscapes. Change Impact Analysis (CIA) is a task that relies on engineers' successful information seeking in databases storing, e.g., source code, requirements, design descriptions, and test case specifications. Several previous approaches to support information seeking are task-specific, thus understanding engineers' seeking behavior in specific tasks is fundamental. We present an industrial case study on how
engineers seek information in CIA, with a particular focus on traceability and development artifacts that are not source code. We show that engineers have different information seeking behavior, and that some do not consider traceability particularly useful when conducting CIA. Furthermore, we observe a tendency for engineers to prefer less rigid types of support rather than formal approaches, i.e., engineers value support that allows flexibility in how to practically conduct CIA. Finally, due to diverse
information seeking behavior, we argue that future CIA support should embrace individual preferences to identify change impact by empowering several seeking alternatives, including searching, browsing, and tracing.