Sparse Bayesian Optimization

TL;DR

This paper introduces regularization-based methods, including a novel differentiable relaxation and a hyperparameter-free approach called SEBO, to enable sparse and interpretable configurations in Bayesian optimization, especially for recommendation systems.

Contribution

It presents new regularization techniques, notably a differentiable relaxation for $L_0$ regularization and SEBO, to promote sparsity and interpretability in Bayesian optimization.

Findings

SEBO effectively balances objective maximization and sparsity.

Regularization methods improve interpretability without sacrificing optimization performance.

The approaches are validated on synthetic and real-world problems.

Abstract

Bayesian optimization (BO) is a powerful approach to sample-efficient optimization of black-box objective functions. However, the application of BO to areas such as recommendation systems often requires taking the interpretability and simplicity of the configurations into consideration, a setting that has not been previously studied in the BO literature. To make BO useful for this setting, we present several regularization-based approaches that allow us to discover sparse and more interpretable configurations. We propose a novel differentiable relaxation based on homotopy continuation that makes it possible to target sparsity by working directly with $L_0$ regularization. We identify failure modes for regularized BO and develop a hyperparameter-free method, sparsity exploring Bayesian optimization (SEBO) that seeks to simultaneously maximize a target objective and sparsity. SEBO and methods based on fixed regularization are evaluated on synthetic and real-world problems, and we show that we are able to efficiently optimize for sparsity.