Dealing with Categorical and Integer-valued Variables in Bayesian Optimization with Gaussian Processes

TL;DR

This paper proposes a more principled approach for Bayesian Optimization with Gaussian Processes that effectively handles categorical and integer-valued variables, improving optimization results over standard methods.

Contribution

It introduces a novel method to incorporate categorical and integer variables into Gaussian Process-based Bayesian Optimization, overcoming limitations of common approximation techniques.

Findings

Significant improvement over standard BO methods on synthetic problems.

Enhanced optimization performance on real-world tasks.

Addresses issues caused by naive encoding of variables.

Abstract

Bayesian Optimization (BO) methods are useful for optimizing functions that are expen- sive to evaluate, lack an analytical expression and whose evaluations can be contaminated by noise. These methods rely on a probabilistic model of the objective function, typically a Gaussian process (GP), upon which an acquisition function is built. The acquisition function guides the optimization process and measures the expected utility of performing an evaluation of the objective at a new point. GPs assume continous input variables. When this is not the case, for example when some of the input variables take categorical or integer values, one has to introduce extra approximations. Consider a suggested input location taking values in the real line. Before doing the evaluation of the objective, a common approach is to use a one hot encoding approximation for categorical variables, or to round to the closest integer, in the case of integer-valued variables. We show that this can lead to problems in the optimization process and describe a more principled approach to account for input variables that are categorical or integer-valued. We illustrate in both synthetic and a real experiments the utility of our approach, which significantly improves the results of standard BO methods using Gaussian processes on problems with categorical or integer-valued variables.