Formalizing Regression Testing for Agile and Continuous Integration Environments

TL;DR

This paper formalizes the process of continuous regression testing in agile and CI environments, modeling successive builds and testing windows to better understand and validate testing strategies.

Contribution

It introduces a formal model for continuous regression testing, capturing build sequences and testing windows, and validates it against existing agile testing algorithms.

Findings

Model degenerates to retest-all for two builds

Formalization is validated through soundness and completeness proofs

Represents state-of-the-art algorithms without extra assumptions

Abstract

Software developed using modern agile practices delivers a stream of software versions that require continuous regression testing rather than testing once close to the delivery or maintenance phase, as assumed by classical regression-testing theory. In this work, we formalize the phenomenon of continuous or near-continuous regression testing using successive builds as a time-ordered chain, where each build contains the program, requirements, and the accompanying tests. We also formalize the regression test window between any two builds, which captures the limited time budget available for regression testing. As the time limit is set to infinity and the chain is closed to two builds, the model degenerates to retest-all, thereby preserving semantics for the classical two-version case. The formalization is validated by directly representing two state-of-the-art agile regression testing algorithms in terms of build-tuple operations without requiring auxiliary assumptions, followed by proof of the soundness and completeness of our formalization.