Iterative Deepening Hyperband

TL;DR

This paper introduces incremental variants of Hyperband for hyperparameter optimization, addressing the challenge of selecting the maximal budget and enabling more efficient, theoretically sound, and practical hyperparameter tuning.

Contribution

We propose incremental Hyperband variants that eliminate the need for predefining the maximal budget, maintaining theoretical guarantees and improving practical efficiency.

Findings

Incremental Hyperband matches original Hyperband's theoretical guarantees.

The new variants outperform standard Hyperband in practical experiments.

They adapt dynamically to different budgets, reducing re-computation.

Abstract

Hyperparameter optimization (HPO) is concerned with the automated search for the most appropriate hyperparameter configuration (HPC) of a parameterized machine learning algorithm. A state-of-the-art HPO method is Hyperband, which, however, has its own parameters that influence its performance. One of these parameters, the maximal budget, is especially problematic: If chosen too small, the budget needs to be increased in hindsight and, as Hyperband is not incremental by design, the entire algorithm must be re-run. This is not only costly but also comes with a loss of valuable knowledge already accumulated. In this paper, we propose incremental variants of Hyperband that eliminate these drawbacks, and show that these variants satisfy theoretical guarantees qualitatively similar to those for the original Hyperband with the "right" budget. Moreover, we demonstrate their practical utility in experiments with benchmark data sets.