Efficient Conditioning Why Pseudo Observation Batch Bayesian Optimization Works When It Does not

TL;DR

This paper provides a unified theoretical framework for batch Bayesian Optimization methods like Constant Liar and Kriging Believer, demonstrating their effectiveness through efficient conditioning with Gaussian Processes and validating predictions with extensive experiments.

Contribution

It introduces the concept of efficient conditioning as a key property, unifies existing batch methods under a single mechanism, and proposes the Structural Diversity Diagnostic for surrogate model testing.

Findings

Gaussian Processes produce distinct batch points with separation of order l.

Efficient conditioning extends to Multiquadric RBF networks.

Neural networks regain diversity at 15x the GP conditioning cost.

Abstract

Constant Liar (CL), Kriging Believer (KB), and fantasy models are widely used for batch selection in parallel Bayesian Optimization, yet a unified theory explaining their effectiveness and conditions under which they fail has been lacking. We identify efficient conditioning as the key surrogate property the ability to update predictions in closed form when data is augmented. We prove that Gaussian Processes satisfy this requirement, producing provably distinct batch points with separation of order l, and that this holds for any acquisition function monotonically non decreasing in posterior uncertainty (EI, UCB, PI), with qualitatively similar behavior for Thompson Sampling. We unify CL, KB, and fantasy models as instances of a single conditioning mechanism differing only in the lie value distribution, and draw quantitative connections to Local Penalization (LP) and qualitative connections to Determinantal Point Processes (DPPs). To disentangle model structure from optimizer randomness, we introduce the Structural Diversity Diagnostic (SDD), a reusable methodology for testing surrogate compatibility. Experiments on Hartmann6D, Ackley 8D, Levy10D, and SVM hyperparameter tuning validate all theoretical predictions: CL or KBs implicit penalty matches or outperforms explicit LP greedy conditioning achieves convergence on par with joint qEI efficient conditioning extends to Multiquadric RBF networks; and parametric surrogates produce degenerate batches even when fully retrained (random forests), while neural networks regain diversity only at 15x the wall clock cost of GP conditioning. Robustness is confirmed across multiple initial datasets and under observation noise.