HPO: We won't get fooled again

TL;DR

This paper investigates how the design of the evaluation pipeline in hyperparameter optimization can bias the landscape, affecting the performance and reliability of HPO methods, especially in the presence of majority class prediction models.

Contribution

It introduces a fitness landscape analysis approach to study the impact of HPO pipeline components on optimization performance, revealing potential biases and issues.

Findings

Large groups of hyperparameters often lead to poor performance, linked to majority class models.

Worsened correlation between observed and neighborhood fitness complicates local search strategies.

HPO pipeline design can negatively influence the landscape and optimization outcomes.

Abstract

Hyperparameter optimization (HPO) is a well-studied research field. However, the effects and interactions of the components in an HPO pipeline are not yet well investigated. Then, we ask ourselves: can the landscape of HPO be biased by the pipeline used to evaluate individual configurations? To address this question, we proposed to analyze the effect of the HPO pipeline on HPO problems using fitness landscape analysis. Particularly, we studied the DS-2019 HPO benchmark data set, looking for patterns that could indicate evaluation pipeline malfunction, and relate them to HPO performance. Our main findings are: (i) In most instances, large groups of diverse hyperparameters (i.e., multiple configurations) yield the same ill performance, most likely associated with majority class prediction models; (ii) in these cases, a worsened correlation between the observed fitness and average fitness in the neighborhood is observed, potentially making harder the deployment of local-search based HPO strategies. Finally, we concluded that the HPO pipeline definition might negatively affect the HPO landscape.