Information Theoretic Bayesian Optimization over the Probability Simplex

TL;DR

This paper introduces $ extalpha$-GaBO, a Bayesian optimization method leveraging information geometry to efficiently optimize functions over the probability simplex, outperforming Euclidean-based approaches in various applications.

Contribution

It develops a novel Bayesian optimization framework using information geometry to better handle the probability simplex domain, including new kernels and optimization strategies.

Findings

Enhanced performance over Euclidean methods in benchmark tests.

Effective in real-world applications like mixture models and robotic control.

Validated through diverse experiments demonstrating improved efficiency.

Abstract

Bayesian optimization is a data-efficient technique that has been shown to be extremely powerful to optimize expensive, black-box, and possibly noisy objective functions. Many applications involve optimizing probabilities and mixtures which naturally belong to the probability simplex, a constrained non-Euclidean domain defined by non-negative entries summing to one. This paper introduces $\alpha$-GaBO, a novel family of Bayesian optimization algorithms over the probability simplex. Our approach is grounded in information geometry, a branch of Riemannian geometry which endows the simplex with a Riemannian metric and a class of connections. Based on information geometry theory, we construct Mat\'ern kernels that reflect the geometry of the probability simplex, as well as a one-parameter family of geometric optimizers for the acquisition function. We validate our method on benchmark functions and on a variety of real-world applications including mixtures of components, mixtures of classifiers, and a robotic control task, showing its increased performance compared to constrained Euclidean approaches.