How Low Can We Go? Minimizing Interaction Samples for Configurable Systems

TL;DR

This paper introduces a new theoretical framework and algorithm for minimizing interaction samples in configurable systems, achieving smaller samples and certifiable optimality in most cases, thus improving testing efficiency.

Contribution

The paper presents a duality-based framework and the SampLNS algorithm, providing provable bounds and optimal solutions for interaction sampling in configurable systems.

Findings

SampLNS finds smaller samples in 85% of cases

Proves optimality for 63% of instances

Reduces testing resources significantly

Abstract

Modern software systems are typically configurable, a fundamental prerequisite for wide applicability and reusability. This flexibility poses an extraordinary challenge for quality assurance, as the enormous number of possible configurations makes it impractical to test each of them separately. This is where t-wise interaction sampling can be used to systematically cover the configuration space and detect unknown feature interactions. Over the last two decades, numerous algorithms for computing small interaction samples have been studied, providing improvements for a range of heuristic results; nevertheless, it has remained unclear how much these results can still be improved. We present a significant breakthrough: a fundamental framework, based on the mathematical principle of duality, for combining near-optimal solutions with provable lower bounds on the required sample size. This implies that we no longer need to work on heuristics with marginal or no improvement, but can certify the solution quality by establishing a limit on the remaining gap; in many cases, we can even prove optimality of achieved solutions. This theoretical contribution also provides extensive practical improvements: Our algorithm SampLNS was tested on 47 small and medium-sized configurable systems from the existing literature. SampLNS can reliably find samples of smaller size than previous methods in 85% of the cases; moreover, we can achieve and prove optimality of solutions for 63% of all instances. This makes it possible to avoid cumbersome efforts of minimizing samples by researchers as well as practitioners, and substantially save testing resources for most configurable systems.