Detecting Performance-Relevant Changes in Configurable Software Systems

TL;DR

ConfFLARE is a novel approach that efficiently predicts performance regressions in configurable software by identifying key features and reducing the number of configurations needed for testing, saving significant time and resources.

Contribution

It introduces ConfFLARE, a method that estimates performance impact and selects relevant configurations using data-flow analysis, improving testing efficiency in configurable systems.

Findings

ConfFLARE correctly detects almost all performance regressions.

It identifies relevant features in nearly all cases.

Reduces configuration testing by up to 79%.

Abstract

Performance is a volatile property of a software system and frequent performance profiling is required to keep the knowledge about a software system's performance behavior up to date. Repeating all performance measurements after every revision is a cost-intensive task, especially in the presence of configurability, where one has to measure multiple configurations to obtain a comprehensive picture. Configuration sampling is a common approach to control the measurement cost. However, it cannot guarantee completeness and might miss performance regressions, especially if they only affect few configurations. As an alternative to solve the cost reduction problem, we present ConfFLARE: ConfFLARE estimates whether a change potentially impacts performance by identifying data-flow interactions with performance-relevant code and extracts which software features participate in such interactions. Based on these features, we can select a subset of relevant configurations to focus performance profiling efforts on. In a study conducted on both, synthetic and real-world software systems, ConfFLARE correctly detects performance regressions in almost all cases and identifies relevant features in all but two cases, reducing the number of configurations to be tested on average by $79\%$ for synthetic and by $70\%$ for real-world regression scenarios saving hours of performance testing time.