Dynamic Priors in Bayesian Optimization for Hyperparameter Optimization

TL;DR

DynaBO is a Bayesian optimization framework that allows continuous user control and incorporation of priors during hyperparameter tuning, improving efficiency and robustness in iterative machine learning workflows.

Contribution

We introduce DynaBO, a novel BO method that integrates user priors dynamically with theoretical guarantees and robustness features, advancing hyperparameter optimization techniques.

Findings

DynaBO outperforms state-of-the-art methods across various benchmarks.

It effectively incorporates and adapts to different user priors.

The framework maintains convergence guarantees even with misleading priors.

Abstract

Bayesian optimization (BO) is a widely used approach to hyperparameter optimization (HPO). However, most existing HPO methods only incorporate expert knowledge during initialization, limiting practitioners' ability to influence the optimization process as new insights emerge. This limits the applicability of BO in iterative machine learning development workflows. We propose DynaBO, a BO framework that enables continuous user control of the optimization process. Over time, DynaBO leverages provided user priors by augmenting the acquisition function with decaying, prior-weighted preferences while preserving asymptotic convergence guarantees. To reinforce robustness, we introduce a data-driven safeguard that detects and can be used to reject misleading priors. We prove theoretical results on near-certain convergence, robustness to adversarial priors, and accelerated convergence when informative priors are provided. Extensive experiments across various HPO benchmarks show that DynaBO consistently outperforms our state-of-the-art competitors across all benchmarks and for all prior kinds. Our results demonstrate that DynaBO enables reliable and efficient collaborative BO, bridging automated and manually controlled model development.