Unlocking the Secrets of Software Configuration Landscapes-Ruggedness, Accessibility, Escapability, and Transferability

TL;DR

This study conducts a comprehensive fitness landscape analysis of configurable software systems, revealing their ruggedness, accessibility of optimal configurations, and structural similarities across workloads, which aids in improving configuration tuning methods.

Contribution

First large-scale fitness landscape analysis of configurable software systems using 86 million configurations, uncovering key landscape properties and workload similarities to guide heuristic search.

Findings

Software landscapes are quite rugged with many local optima.

Top local optima are highly accessible with large basins of attraction.

Most poor local optima can be escaped with simple perturbations.

Abstract

Modern software systems are often highly configurable to tailor varied requirements from diverse stakeholders. Understanding the mapping between configurations and the desired performance attributes plays a fundamental role in advancing the controllability and tuning of the underlying system, yet has long been a dark hole of knowledge due to their black-box nature and the enormous combinatorial configuration space. In this paper, using $86$M evaluated configurations from three real-world systems on $32$ running workloads, we conducted one of its kind fitness landscape analysis (FLA) for configurable software systems. With comprehensive FLA methods, we for the first time show that: $i)$ the software configuration landscapes are fairly rugged, with numerous scattered local optima; $ii)$ nevertheless, the top local optima are highly accessible, featuring significantly larger basins of attraction; $iii)$ most inferior local optima are escapable with simple perturbations; $iv)$ landscapes of the same system with different workloads share structural similarities, which can be exploited to expedite heuristic search. Our results also provide valuable insights on the design of tailored meta-heuristics for configuration tuning; our FLA framework along with the collected data, build solid foundation for future research in this direction.