(Blogged by Saket S.)

The DevOps approach (or mindset, philosophy, toolchain… whatever it is), has emerged as a crucial component to bridge the gap between has made life easy for software development and operation teams and has emerged as a crucial component in bridging the gap between these two groups. However, it comes with a host of new challenges when applied to the process of performance evaluation. It entails the developers to be aware of the application runtime characteristics and deployment settings. At the operational stage too, several uncertainties like resource availability and workload fluctuations may affect the performance analysis.

Therefore the DevOps teams have to have clarity about the uncertainties in the entire DevOps lifecycle so that the interpretation of models, data, and results can be made accordingly. We believe that performance-aware DevOps can benefit immensely if assistance in this regard is provided in the following areas.

1. Identification of sources of uncertainty in a performance-aware DevOps situation.
2. Explanation about how these uncertainties represent themselves in design models, input data, and operational results.
3. Execution of a sensitivity evaluation to quantify the impact that these sources of uncertainties have on the interpretation of results.

In this paperblog, we shall try to help the DevOps teams in identifying the sources of uncertainty and applying a set of methodologies that control these uncertainties to detect the violations of performance requirements and bridge the operational performance issues.

The sources of uncertainty must be identified first if one were to minimize it. For the identification purpose, we can use the following two strategies.

1. The bottom-up strategy: It is based on the information of the possible sources of uncertainty in a model.
2. The top-down strategy: This strategy is based upon the complete lack of information about the possible uncertainties.

Some of the primary sources of uncertainty are-:

• Deployment Infrastructure: Within the same release and workload, the changing deployment infrastructure can have negligible to a considerable impact on the performance of the system. It is essential to carry out user acceptance testing or keep an eye on the performance of the canary application before a full-scale infrastructure change to limit such uncertainties.

• Software versions and code changes: It is essential to evaluate the impact of performance on various configuration parameters for every release. However, performance testing reduction methods should be applied to explore the system configuration space due to the extensive combination of rapid changes taking place in DevOps and configuration space.

• Configuration Parameters: The percentage of common configurations is quite low among various hardware platforms, workloads, and software versions. It proves how critical configuration parameters are for system performance.

• Monitoring and sensor accuracy: The measurement noise can have a significant impact on the validity of performance results. However, its overall effect is usually small. Nonetheless, it will do no harm to cross-check the measurement data to check how valid they are.

Modeling and controlling uncertainties

Upon the identification of uncertainty sources, their impact can be minimized by modeling and analyzing the variation in performance caused by those uncertainties and then devising methods to put a check on the impact that such uncertainties have.

Uncertainty can be minimized through three main techniques. The first technique is the widely used method of repeated measurement. Discrepancies in performance measurement can lead to uncertainties in the evaluation of performance results. Calculating a confidence interval for consistent performance measurements during performance evaluation can eliminate these discrepancies the result interpretation could be made more reliable. If the variation in results is already known, uncertainty could be minimized through rigorous statistical techniques.

Secondly, uncertainty could also be minimized if one were to have more knowledge about the nature of the system. This knowledge could be gained through extensive simulation and modeling. With models, it would be easy to stipulate the uncertainty in parameters values through probability distribution functions. Thus, samples of parameter values could be extracted to minimize the uncertainty. A model-based approach developed by Goldsby and Cheng can generate a system model to replicate the behavior in complex environments. Such a model can be used by developers to understand the uncertainty in similar settings adequately.

The third way of minimizing uncertainty is by providing additional information about the subject system. This way system monitoring could be enriched by attaching more runtime information.

C. Trubiani, P. Jamshidi, J. Cito, W. Shang, Z. Jiang, M Borg. Performance issues? Hey DevOps, mind the uncertainty! IEEE Software, 36(2), pp. 110-117, 2019. (link, preprint)

Abstract

DevOps is a novel trend that aims to bridge the gap between software development and operation teams. This article presents an experience report that better identifies performance uncertainties through a case study and provides a step-by-step guide to practitioners for controlling system uncertainties.