Variability Fault Localization: A Benchmark

TL;DR

This paper introduces a benchmark dataset for variability fault localization in Software Product Line systems, aiming to facilitate research and development of more effective localization techniques for variability-related bugs.

Contribution

It provides the first large-scale benchmark with 1,570 buggy versions across six SPL systems to evaluate variability fault localization methods.

Findings

Baseline localization performance results included

Benchmark dataset available for community use

Encourages development of improved localization techniques

Abstract

Software fault localization is one of the most expensive, tedious, and time-consuming activities in program debugging. This activity becomes even much more challenging in Software Product Line (SPL) systems due to the variability of failures in SPL systems. These unexpected behaviors are caused by variability faults which can only be exposed under some combinations of system features. Although localizing bugs in non-configurable code has been investigated in-depth, variability fault localization in SPL systems still remains mostly unexplored. To approach this challenge, we propose a benchmark for variability fault localization with a large set of 1,570 buggy versions of six SPL systems and baseline variability fault localization performance results. Our hope is to engage the community to propose new and better approaches to the problem of variability fault localization in SPL systems.