Rethinking Performance Analysis for Configurable Software Systems: A Case Study from a Fitness Landscape Perspective

TL;DR

This paper introduces a new landscape perspective and a graph-based framework for analyzing the performance of configurable software systems, revealing insights into their configuration space and aiding tuning and modeling.

Contribution

It proposes modeling configuration spaces as structured landscapes and develops an open-source framework for fitness landscape analysis of large-scale software configurations.

Findings

Holistic understanding of configuration space topography

Identification of local optima and landscape features

Implications for tuning and performance modeling

Abstract

Modern software systems are often highly configurable to tailor varied requirements from diverse stakeholders. Understanding the mapping between configurations and the desired performance attributes plays a fundamental role in advancing the controllability and tuning of the underlying system, yet has long been a dark hole of knowledge due to its black-box nature. While there have been previous efforts in performance analysis for these systems, they analyze the configurations as isolated data points without considering their inherent spatial relationships. This renders them incapable of interrogating many important aspects of the configuration space like local optima. In this work, we advocate a novel perspective to rethink performance analysis -- modeling the configuration space as a structured ``landscape''. To support this proposition, we designed \our, an open-source, graph data mining empowered fitness landscape analysis (FLA) framework. By applying this framework to $86$M benchmarked configurations from $32$ running workloads of $3$ real-world systems, we arrived at $6$ main findings, which together constitute a holistic picture of the landscape topography, with thorough discussions about their implications on both configuration tuning and performance modeling.