Make or buy? A survey.

Suppose you need to extend your system with a new component. Should you develop it in-house? Or is outsourcing better? Could you buy if from some software vendor? Or can you adapt a piece of open source software? We present a thorough analysis of empirical results from a survey of 188 industry practitioners.

This paper is one of the foundational empirical studies of the Orion project – a large research endeavor on decision-making in component-based software engineering. Before I joined the project, the team decided to conduct a series of empirical studies to understand the phenomeon of decision making for sourcing options in depth. A literature review, a case survey, a quetionnarire-based survey, and some case studies were planned. When I joined, I agreed to take the lead in the survey. It took quite some time to finish the study and publish the results, but the activity resulted in this IST publication, a distinguished paper at Euromicro SEAA in 2018 and an extended version published in Software Quality Journal.

Choosing between component sourcing options

The main objective of the Orion research project was to understand how practitioners choose between Component Sourcing Options (CSO). If you were to add or replace a component in an evolving system, should you 1) develop it yourself, 2) outsource the development, 3) integrate an open source component, or 4) purchase something commercial-of-the-shelf (COTS)? The alternatives are not as clear cut in reality, but the simplification makes it possible to study. In the Orion project, we have explored this from several perspectives.

In this paper, we share our findings from a survey of 188 industry practitioners with experience from CSO decisions. The questionnaire used to collect data was long and complex – we purchased a customized solution from a third party based in the US – thus we were quite happy with the 188 responses we finally obtained.

The figure below shows the CSOs typically considered by the respondents. 87% consider in-house development and 60% consider integrating open source software. We also note that 73% typically consider two or more options, thus actively making CSO decisions. This confirms the relevance of the Orion objective.

We took the chance to ask several questions related to the nature of the CSO decision process. A majority of the respondents answer that CSO decisions are based on expert judgment – no surprise there. On the other hand, almost half of the respondents consider the decision process to be based on collected data. This appears contradictory at first. We believe that expert judgment dominates CSO decisions in industry, but the experts inform themselves based on data collected within the organizations, i.e., the typical CSO decision process is based on a data-driven approach to expert judgment. Finally, our data suggest that decision processes vary in terms of transparency, how systematic they are, and whether they are democratic or authoritarian.

Qualities that matter

Another inquiry we had was about the most important quality attributes as defined by ISO/IEC 25010. As shown in the figure below, functional suitability was considered the most important by the respondents, followed by reliability and maintainability. Portability stands out as the least important quality attribute in the study.

In relation to the previous questions, we asked the respondents how they estimate the level of quality corresponding to the CSO decision. In line with previous results, we find that expert judgment is the most common method. Considerably less frequent estimations methods are asking source providers, asking component users, and external expert judgment.

No one could argue that this study found anything revolutionizing. But it does bring novel empirical evidence to the table. Also, to reduce the publication bias in software engineering, we certainly should publish papers that simply confirm what is considered to be true. In light of this goal, I think this paper is a fine contribution.

Implications for practice

• Most organizations consider multiple CSOs during software evolution. You probably should as well.
• CSO decisions in industry are dominated by expert judgment, but they are still influenced by data.
• How do you approach CSO decisions? Use the findings from this survey as a benchmark for comparison to others.

Implications for research

• Make-or-buy decisions in SE are more complex than in other engineering fields.
• By understanding the state-of-practice CSO decisions, we can develop better interventions to support the activity.
• In CSO decisions, practitioners appear to use functional suitability as an initial filter, then reliability is the most important software quality.

Markus Borg, Panagiota Chatzipetrou, Krzysztof Wnuk, Emil Alégroth, Tony Gorschek, Efi Papatheocharous, Syed Muhammad Ali Shah, and Jakob Axelsson. Selecting Component Sourcing Options: A Survey of Software Engineering's Broader Make-or-Buy Decisions, Information and Software Technology, Volume 112, pp. 18-34, 2019. (link)

Abstract

Context: Component-based software engineering (CBSE) is a common approach to develop and evolve contemporary software systems. When evolving a system based on components, make-or-buy decisions are frequent, i.e., whether to develop components internally or to acquire them from external sources. In CBSE, several dierent sourcing options are available: 1) developing software in-house, 2) outsourcing development, 3) buying commercial-o-the-shelf software, and 4) integrating open source software components. Objective: Unfortunately, there is little available research on how organizations select component sourcing options (CSO) in industry practice. In this work, we seek to contribute empirical evidence to CSO selection. Method: We conduct a cross-domain survey on CSO selection in industry, implemented as an online questionnaire. Results: Based on 188 responses, we nd that most organizations consider multiple CSOs during software evolution, and that the CSO decisions in industry are dominated by expert judgment. When choosing between candidate components, functional suitability acts as an initial lter, then reliability is the most
important quality. Conclusion: We stress that future solution-oriented work on decision support has to account for the dominance of expert judgment in industry. Moreover, we identify considerable variation in CSO decision processes in industry. Finally, we encourage software development organizations to reflect on their decision processes when choosing whether to make or buy components, and we recommend using our survey for a first benchmarking.