A Lipschitz Bandits Approach for Continuous Hyperparameter Optimization

TL;DR

This paper introduces BLiE, a Lipschitz-bandit-based algorithm for hyperparameter optimization that adaptively explores the hyperparameter space with theoretical guarantees and improved empirical performance.

Contribution

BLiE is the first HPO algorithm leveraging only Lipschitz continuity assumptions, providing theoretical bounds and demonstrating superior empirical results.

Findings

BLiE finds $\e$-optimal hyperparameters efficiently with theoretical guarantees.

BLiE outperforms existing HPO algorithms on benchmark tasks.

Applying BLiE to diffusion models' noise schedules improves sampling speed.

Abstract

One of the most critical problems in machine learning is HyperParameter Optimization (HPO), since choice of hyperparameters has a significant impact on final model performance. Although there are many HPO algorithms, they either have no theoretical guarantees or require strong assumptions. To this end, we introduce BLiE -- a Lipschitz-bandit-based algorithm for HPO that only assumes Lipschitz continuity of the objective function. BLiE exploits the landscape of the objective function to adaptively search over the hyperparameter space. Theoretically, we show that $(i)$ BLiE finds an $\epsilon$-optimal hyperparameter with $\mathcal{O} \left( \epsilon^{-(d_z + \beta)}\right)$ total budgets, where $d_z$ and $\beta$ are problem intrinsic; $(ii)$ BLiE is highly parallelizable. Empirically, we demonstrate that BLiE outperforms the state-of-the-art HPO algorithms on benchmark tasks. We also apply BLiE to search for noise schedule of diffusion models. Comparison with the default schedule shows that BLiE schedule greatly improves the sampling speed.