The Evolving Landscape of Software Performance Engineering

TL;DR

This paper reviews the challenges of applying Software Performance Engineering in CI/CD environments, highlighting the need for continuous evaluation, broader skills, and process improvements based on five years of practical experience.

Contribution

It proposes enhancements to SPE processes tailored for CI/CD, based on lessons learned from long-term industry practice.

Findings

SPE has become mainly a post hoc activity using production data

Frequent re-evaluation of priorities is essential for successful SPE

Engineers need broader skill sets for effective SPE in modern environments

Abstract

Satisfactory software performance is essential for the adoption and the success of a product. In organizations that follow traditional software development models (e.g., waterfall), Software Performance Engineering (SPE) involves time-consuming experimental modeling and performance testing outside the actual production environment. Such existing SPE methods, however, are not optimized for environments utilizing Continuous Integration (CI) and Continuous Delivery (CD) that result in high frequency and high volume of code changes. We present a summary of lessons learned and propose improvements to the SPE process in the context of CI/CD. Our findings are based on SPE work on products A and B conducted over 5 years at an online services company X. We find that (a) SPE has mainly become a post hoc activity based on data from the production environment, (b) successful application of SPE techniques require frequent re-evaluation of priorities, and (c) engineers working on SPE require a broader skill set than one traditionally possessed by engineers working on performance.