The Interaction between Inputs and Configurations fed to Software Systems: an Empirical Study

TL;DR

This empirical study investigates how input data and configuration options interact in software systems, revealing significant impacts on performance and the challenges in tuning and modeling due to this variability.

Contribution

It provides a large-scale analysis of input-configuration interactions in eight configurable systems, highlighting their effects on performance and model relevance.

Findings

Inputs interact with configurations in non-monotonous ways

Tuning for input data can improve performance up to tenfold

Input variability can undermine performance prediction models

Abstract

Widely used software systems such as video encoders are by necessity highly configurable, with hundreds or even thousands of options to choose from. Their users often have a hard time finding suitable values for these options (i.e. finding a proper configuration of the software system) to meet their goals for the tasks at hand, e.g. compress a video down to a certain size. One dimension of the problem is of course that performance depends on the input data: a video as input to an encoder like x264 or a file system fed to a tool like xz. To achieve good performance, users should therefore take into account both dimensions of (1) software variability and (2) input data. In this problem-statement paper, we conduct a large study over 8 configurable systems that quantifies the existing interactions between input data and configurations of software systems. The results exhibit that (1) inputs fed to software systems interact with their configuration options in non monotonous ways, significantly impacting their performance properties (2) tuning a software system for its input data makes it possible to multiply its performance by up to ten (3) input variability can jeopardize the relevance of performance predictive models for a field deployment.