Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

TL;DR

Hyperband is a new bandit-based algorithm that accelerates hyperparameter optimization by efficiently allocating resources and early-stopping, significantly speeding up the process compared to Bayesian methods.

Contribution

We introduce Hyperband, a novel resource allocation algorithm for hyperparameter tuning, with theoretical guarantees and superior empirical performance over Bayesian optimization.

Findings

Hyperband achieves over tenfold speedup in hyperparameter tuning.

It outperforms Bayesian optimization on deep learning and kernel methods.

Theoretical analysis confirms desirable properties of Hyperband.

Abstract

Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.