Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning

TL;DR

This extensive empirical study reveals that higher surrogate model accuracy does not always improve configuration tuning outcomes and can sometimes even worsen them, challenging the common belief that accuracy directly correlates with tuning quality.

Contribution

The paper provides the largest empirical analysis to date on the impact of surrogate model accuracy in configuration tuning, showing that accuracy is not always indicative of better tuning results and highlighting the need to reconsider current practices.

Findings

Higher accuracy often does not improve tuning outcomes (up to 58% cases)

In some cases, increased accuracy can degrade tuning quality (up to 24% cases)

Most proposed tuners use sub-optimal models and accuracy improvements vary with accuracy range

Abstract

To ease the expensive measurements during configuration tuning, it is natural to build a surrogate model as the replacement of the system, and thereby the configuration performance can be cheaply evaluated. Yet, a stereotype therein is that the higher the model accuracy, the better the tuning result would be. This "accuracy is all" belief drives our research community to build more and more accurate models and criticize a tuner for the inaccuracy of the model used. However, this practice raises some previously unaddressed questions, e.g., Do those somewhat small accuracy improvements reported in existing work really matter much to the tuners? What role does model accuracy play in the impact of tuning quality? To answer those related questions, we conduct one of the largest-scale empirical studies to date-running over the period of 13 months 24*7-that covers 10 models, 17 tuners, and 29 systems from the existing works while under four different commonly used metrics, leading to 13,612 cases of investigation. Surprisingly, our key findings reveal that the accuracy can lie: there are a considerable number of cases where higher accuracy actually leads to no improvement in the tuning outcomes (up to 58% cases under certain setting), or even worse, it can degrade the tuning quality (up to 24% cases under certain setting). We also discover that the chosen models in most proposed tuners are sub-optimal and that the required % of accuracy change to significantly improve tuning quality varies according to the range of model accuracy. Deriving from the fitness landscape analysis, we provide in-depth discussions of the rationale behind, offering several lessons learned as well as insights for future opportunities. Most importantly, this work poses a clear message to the community: we should take one step back from the natural "accuracy is all" belief for model-based configuration tuning.