Rapid Regression Detection in Software Deployments through Sequential Testing

TL;DR

This paper introduces a statistical framework for quickly detecting regressions in software deployments using sequential testing, improving quality control during continuous delivery.

Contribution

It proposes a novel sequential testing method based on stochastic order and distribution equality to monitor canary tests in real-time.

Findings

Effective regression detection demonstrated at Netflix

Strict false detection probability control achieved

Rapid identification of regressions in production environments

Abstract

The practice of continuous deployment has enabled companies to reduce time-to-market by increasing the rate at which software can be deployed. However, deploying more frequently bears the risk that occasionally defective changes are released. For Internet companies, this has the potential to degrade the user experience and increase user abandonment. Therefore, quality control gates are an important component of the software delivery process. These are used to build confidence in the reliability of a release or change. Towards this end, a common approach is to perform a canary test to evaluate new software under production workloads. Detecting defects as early as possible is necessary to reduce exposure and to provide immediate feedback to the developer. We present a statistical framework for rapidly detecting regressions in software deployments. Our approach is based on sequential tests of stochastic order and of equality in distribution. This enables canary tests to be continuously monitored, permitting regressions to be rapidly detected while strictly controlling the false detection probability throughout. The utility of this approach is demonstrated based on two case studies at Netflix.