Time-Space Efficient Regression Testing for Configurable Systems

TL;DR

EvoSPLat is a novel regression testing technique for configurable systems that efficiently reduces testing time and configuration space while maintaining bug detection accuracy.

Contribution

It introduces EvoSPLat, a scalable and sound regression testing approach tailored for configurable systems, improving over existing methods in efficiency and bug detection.

Findings

Reduced testing time by 22% in Java programs

Decreased configuration space by 45%

Detected all bugs faster than sampling techniques

Abstract

Configurable systems are those that can be adapted from a set of options. They are prevalent and testing them is important and challenging. Existing approaches for testing configurable systems are either unsound (i.e., they can miss fault-revealing configurations) or do not scale. This paper proposes EvoSPLat, a regression testing technique for configurable systems. EvoSPLat builds on our previously-developed technique, SPLat, which explores all dynamically reachable configurations from a test. EvoSPLat is tuned for two scenarios of use in regression testing: Regression Configuration Selection (RCS) and Regression Test Selection (RTS). EvoSPLat for RCS prunes configurations (not tests) that are not impacted by changes whereas EvoSPLat for RTS prunes tests (not configurations) which are not impacted by changes. Handling both scenarios in the context of evolution is important. Experimental results show that EvoSPLat is promising. We observed a substantial reduction in time (22%) and in the number of configurations (45%) for configurable Java programs. In a case study on a large real-world configurable system (GCC), EvoSPLat reduced 35% of the running time. Comparing EvoSPLat with sampling techniques, 2-wise was the most efficient technique, but it missed two bugs whereas EvoSPLat detected all bugs four times faster than 6-wise, on average.