FlexHB: a More Efficient and Flexible Framework for Hyperparameter Optimization

TL;DR

FlexHB introduces a flexible, efficient hyperparameter optimization framework that combines refined multi-fidelity Bayesian optimization with an adaptive early stopping strategy, significantly outperforming existing methods in speed and effectiveness.

Contribution

The paper proposes FlexHB, a novel framework that enhances multi-fidelity Bayesian optimization with a fine-grained fidelity method and a self-adaptive, flexible early stopping scheme.

Findings

FlexHB achieves up to 6.9X speedup over MFES-HB.

FlexHB achieves up to 11.1X speedup over BOHB.

FlexHB outperforms other methods on various HPO tasks.

Abstract

Given a Hyperparameter Optimization(HPO) problem, how to design an algorithm to find optimal configurations efficiently? Bayesian Optimization(BO) and the multi-fidelity BO methods employ surrogate models to sample configurations based on history evaluations. More recent studies obtain better performance by integrating BO with HyperBand(HB), which accelerates evaluation by early stopping mechanism. However, these methods ignore the advantage of a suitable evaluation scheme over the default HyperBand, and the capability of BO is still constrained by skewed evaluation results. In this paper, we propose FlexHB, a new method pushing multi-fidelity BO to the limit as well as re-designing a framework for early stopping with Successive Halving(SH). Comprehensive study on FlexHB shows that (1) our fine-grained fidelity method considerably enhances the efficiency of searching optimal configurations, (2) our FlexBand framework (self-adaptive allocation of SH brackets, and global ranking of configurations in both current and past SH procedures) grants the algorithm with more flexibility and improves the anytime performance. Our method achieves superior efficiency and outperforms other methods on various HPO tasks. Empirical results demonstrate that FlexHB can achieve up to 6.9X and 11.1X speedups over the state-of-the-art MFES-HB and BOHB respectively.