Bayes Optimal Early Stopping Policies for Black-Box Optimization

TL;DR

This paper develops an optimal Bayesian policy for early stopping in black-box optimization, significantly improving efficiency in hyperparameter tuning tasks like CIFAR-10 and ImageNet.

Contribution

It introduces a Bayesian optimal stopping policy that adaptively restarts algorithms, outperforming existing methods such as random search and Hyperband.

Findings

Up to 13x reduction in expected optimization time on CIFAR-10 and ImageNet.

Outperforms baseline adaptive policies by up to 3x.

Provides a unified framework for selecting optimal algorithms based on Bayesian priors.

Abstract

We derive an optimal policy for adaptively restarting a randomized algorithm, based on observed features of the run-so-far, so as to minimize the expected time required for the algorithm to successfully terminate. Given a suitable Bayesian prior, this result can be used to select the optimal black-box optimization algorithm from among a large family of algorithms that includes random search, Successive Halving, and Hyperband. On CIFAR-10 and ImageNet hyperparameter tuning problems, the proposed policies offer up to a factor of 13 improvement over random search in terms of expected time to reach a given target accuracy, and up to a factor of 3 improvement over a baseline adaptive policy that terminates a run whenever its accuracy is below-median.