Plan that Feedback!

DevOps promises synergies between software development and operations engineers. However, all data collected during Ops cannot mindlessly be shoveled to Dev. We argue that the feedback process must be carefully designed to enable actionable insights.

This paper resulted from discussions in one of the breakout groups at the GI Dagstuhl Seminar Software Performance Engineering in the DevOps World. The workshop was organized so that each participant got to discuss in two separate breakout groups. Both groups I joined had really good discussions, and both resulted in peer-reviewed publications! This work is from the group “Uncertainty in a Performance-Aware DevOps Context”.

DevOps and feedback

Software engineering is certainly not only about developing software and then releasing it. In practice, deploying software to production systems often requires considerable skills and know-how. The DevOps movement highlights the cross-functional synergies between software development and operations activities. There are many ways to bring Dev and Ops together – in this paper we focus on various types of feedback from operations.

Feedback from operations can be critical to successful decision making during development (and evolution!). Feedback can inform both decisions considering the product and the entire development process. For example, performance engineers can study latency metrics to identify slow execution. Furthermore, organizations that apply the practice of continuous deployment can act swiftly based on the feedback.

Different organizations, different challenges

However, organizations face some challenges when designing feedback channels between operations and development. In the paper, we argue that there the intrinsic challenges depend on four key variation points: 1) organizational size, 2) nature of the business, 3) presentation of feedback, and 4) case-specific technical challenges.

We motivate the variation points through a discussion of three fictive case companies. First, a small startup with a B2C online platform that handles most of its transactions over its website that is operated in a public cloud. Second, an SME that offers logistics support system for retailers with software either deployed on a private cloud or – for enterprise clients – on another data center. Third, a large corporation providing factory automation solutions with software primarily deployed on the client’s plant.

Several questions regarding feedback from operations to development arise in these cross-company settings. Who owns the operations data? How should it be made available to product development? Should solution architects, that act as mediators, have access to the same feedback? In the figure on top of the page, we model a feedback process that captures an overview feedback process that can be used to depict all three cases. One aspect that we stress in the paper is the need for feedback governance, i.e., control over which kind of operations feedback is available to which kind of stakeholder.

Characterizing feedback

In the paper, we present a high level discussion on how to design a feedback process from operations to development. First, we describe some fundamental steps of the environment and tooling, e.g., using a feedback control mechanism to make sure the right data is stored in a feedback datastore. The importance of actionable data analytics and how the presentation of results must be carefully engineered to support the recipients, e.g., pop-ups in the IDE, e-mail notifications or dashboards.

Feedback from operations might come in a multitude of flavors. To make it actionable, it is important to specify when it is needed – and by whom. The table below shows examples of useful metrics from operations, created by either observing the running system or by analyzing produced log messages. The takeaway message in the paper is “to let development benefit from operations data, the organization must carefully design the feedback process”.

Implications for Research

• There is a need for additional research on how to design feedback processes.
• The feedback processes will inevitably be highly context-dependent – more research is needed to validate the key variation points we proposed.
• Guidelines for how decision-support tools based on operations data should deliver feedback is needed – what feedback to whom as well as when and how?

Implications for Practice

• Carefully design the DevOps feedback process – too little data might be a wasted opportunity, too much data might lead to information overload.
• Feedback governance will be needed to manage data sharing across organizations.

J. Cito, J. Wettinger, L. Lwakatare, M. Borg, and F. Li, In Proc. of DEVOPS 2018 1st International Workshop on Software Engineering Aspects of Continuous Development and New paradigms of Software Production and Deployment, pp. 184-195, 2019. (link, preprint)

Abstract

DevOps achieves synergy between software development and operations engineers. This synergy can only happen if the right culture is in place to foster communication between these roles. We investigate the relationship between runtime data generated during production and how this data can be used as feedback in the software development process. For that, we want to discuss case study organizations that have different needs on their operations-to-development feedback pipeline, from which we abstract and propose a more general, higher-level feedback process. Given such a process, we discuss a technical environment required to support this process. We sketch out different scenarios in which feedback is useful in different phases of the software development life-cycle.