Cost-aware Stopping for Bayesian Optimization
Qian Xie, Linda Cai, Alexander Terenin, Peter I. Frazier, Ziv Scully
TL;DR
This paper introduces a theoretically grounded, cost-aware stopping rule for Bayesian optimization that adapts to varying evaluation costs and guarantees performance bounds, improving efficiency in practical applications.
Contribution
It proposes a novel, principled stopping rule for Bayesian optimization that is theoretically justified and adapts to different costs without heuristic tuning.
Findings
The stopping rule provides theoretical guarantees on cost-adjusted simple regret.
Empirical results show the rule often outperforms or matches existing methods.
The approach is effective across synthetic and real-world tasks like hyperparameter tuning.
Abstract
In automated machine learning, scientific discovery, and other applications of Bayesian optimization, deciding when to stop evaluating expensive black-box functions in a cost-aware manner is an important but underexplored practical consideration. A natural performance metric for this purpose is the cost-adjusted simple regret, which captures the trade-off between solution quality and cumulative evaluation cost. While several heuristic or adaptive stopping rules have been proposed, they lack guarantees ensuring stopping before incurring excessive function evaluation costs. We propose a principled cost-aware stopping rule for Bayesian optimization that adapts to varying evaluation costs without heuristic tuning. Our rule is grounded in a theoretical connection to state-of-the-art cost-aware acquisition functions, namely the Pandora's Box Gittins Index (PBGI) and log expected improvement…
Peer Reviews
Decision·Submitted to ICLR 2026
- This study is significant in that it clearly formalizes the previously ambiguous stopping condition in cost-sensitive Bayesian optimization, where the evaluation cost varies across input points. The proposed stopping criterion determines when to terminate the optimization based on the theoretical properties of the acquisition function, thereby preventing excessive exploration and unnecessary computational costs while achieving efficient and stable optimization. - The proposed method is also n
- The meaning and derivation process of Equation (5) are not clearly explained in the text, making it extremely difficult to comprehend. Although the study relies heavily on the work of Xie et al., it lacks a self-contained explanation within the paper, which is problematic. Additional clarification should be provided so that readers can understand the content without referring to external sources. - The theoretical analysis is based on an overly simplified assumption of a function with a const
- The paper is well-written when explaining its methodology. - The work presents a stopping rule that is able to apply to a general BO algorithm. - There is a theoretical analysis to support the work.
1. My primary concern lies in the novelty of the work. The concept of the proposed PBGI/LogEIPC stopping rule appears to be essentially identical to the PBGI acquisition function introduced by Xie et al. (2024). In particular, the stopping rule directly uses the same decision criterion that motivated the PBGI acquisition function—namely, determining whether to evaluate the new potential candidate point or to stop and accept the current best observation (see Sec. 3.2, Xie et al., 2024). This work
* Adaptive stopping of Bayesian optimization is an important problem with practical relevance. This work is the first to study this problem in the cost-aware setting, which most closely resembles many practical settings. * The proposed stopping rule comes with theoretical guarantees * The work tackles a range of different cost-aware settings, such as budget-constrained and cost-per-sample, as well as settings where evaluation costs are or are not known in advance. * A comprehensive set of baseli
* The number and type of benchmarks considered are limited. Additional and more different benchmarks would be helpful. This is my main issue with this work. * A summary plot showing, e.g., average ranks or average normalized regret would be helpful for the overall evaluation. This would also allow you to display more different evaluation settings in the main paper. * Unclear what effect does the debounce strategy ("requiring the stopping rule to consistently indicate stopping over several consec
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsReservoir Engineering and Simulation Methods · Machine Learning and Algorithms · Simulation Techniques and Applications