Budgeted Batch Bayesian Optimization With Unknown Batch Sizes

TL;DR

This paper introduces B3O, a flexible batch Bayesian optimization method that automatically determines batch sizes using IGMM, improving efficiency and effectiveness in hyper-parameter tuning and experimental design tasks.

Contribution

The paper proposes B3O, a novel batch Bayesian optimization approach that adaptively identifies batch sizes using IGMM, addressing limitations of fixed batch size methods.

Findings

B3O outperforms fixed batch BO in finding optima.

B3O requires fewer evaluations, saving cost and time.

B3O is effective in real-world hyper-parameter tuning and experimental design.

Abstract

Parameter settings profoundly impact the performance of machine learning algorithms and laboratory experiments. The classical grid search or trial-error methods are exponentially expensive in large parameter spaces, and Bayesian optimization (BO) offers an elegant alternative for global optimization of black box functions. In situations where the black box function can be evaluated at multiple points simultaneously, batch Bayesian optimization is used. Current batch BO approaches are restrictive in that they fix the number of evaluations per batch, and this can be wasteful when the number of specified evaluations is larger than the number of real maxima in the underlying acquisition function. We present the Budgeted Batch Bayesian Optimization (B3O) for hyper-parameter tuning and experimental design - we identify the appropriate batch size for each iteration in an elegant way. To set the batch size flexible, we use the infinite Gaussian mixture model (IGMM) for automatically identifying the number of peaks in the underlying acquisition functions. We solve the intractability of estimating the IGMM directly from the acquisition function by formulating the batch generalized slice sampling to efficiently draw samples from the acquisition function. We perform extensive experiments for both synthetic functions and two real world applications - machine learning hyper-parameter tuning and experimental design for alloy hardening. We show empirically that the proposed B3O outperforms the existing fixed batch BO approaches in finding the optimum whilst requiring a fewer number of evaluations, thus saving cost and time.