Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning

TL;DR

This paper introduces a cost-sensitive freeze-thaw Bayesian optimization method that incorporates user preferences to efficiently balance computational cost and hyperparameter performance, automatically stopping when further improvements are not cost-effective.

Contribution

It presents a novel utility-based framework with an acquisition function and stopping criterion, enhancing sample efficiency through transfer learning for cost-sensitive hyperparameter tuning.

Findings

Outperforms existing freeze-thaw and transfer-BO methods on benchmarks.

Achieves better cost-performance trade-offs.

Automatically stops HPO at optimal utility point.

Abstract

In this paper, we address the problem of \emph{cost-sensitive} hyperparameter optimization (HPO) built upon freeze-thaw Bayesian optimization (BO). Specifically, we assume a scenario where users want to early-stop the HPO process when the expected performance improvement is not satisfactory with respect to the additional computational cost. Motivated by this scenario, we introduce \emph{utility} in the freeze-thaw framework, a function describing the trade-off between the cost and performance that can be estimated from the user's preference data. This utility function, combined with our novel acquisition function and stopping criterion, allows us to dynamically continue training the configuration that we expect to maximally improve the utility in the future, and also automatically stop the HPO process around the maximum utility. Further, we improve the sample efficiency of existing freeze-thaw methods with transfer learning to develop a specialized surrogate model for the cost-sensitive HPO problem. We validate our algorithm on established multi-fidelity HPO benchmarks and show that it outperforms all the previous freeze-thaw BO and transfer-BO baselines we consider, while achieving a significantly better trade-off between the cost and performance. Our code is publicly available at https://github.com/db-Lee/CFBO.