Predictive Modeling through Hyper-Bayesian Optimization

TL;DR

This paper introduces a novel hyper-Bayesian optimization method that simultaneously optimizes model selection and function optimization, leading to faster convergence and better sample efficiency in black-box optimization tasks.

Contribution

It proposes a new integrated approach combining model selection and Bayesian optimization, with a feedback mechanism that stabilizes and accelerates convergence.

Findings

Significant improvement in convergence speed over standard BO.

Enhanced sample efficiency in black-box function optimization.

Framework provides additional insights into the black-box function.

Abstract

Model selection is an integral problem of model based optimization techniques such as Bayesian optimization (BO). Current approaches often treat model selection as an estimation problem, to be periodically updated with observations coming from the optimization iterations. In this paper, we propose an alternative way to achieve both efficiently. Specifically, we propose a novel way of integrating model selection and BO for the single goal of reaching the function optima faster. The algorithm moves back and forth between BO in the model space and BO in the function space, where the goodness of the recommended model is captured by a score function and fed back, capturing how well the model helped convergence in the function space. The score function is derived in such a way that it neutralizes the effect of the moving nature of the BO in the function space, thus keeping the model selection problem stationary. This back and forth leads to quick convergence for both model selection and BO in the function space. In addition to improved sample efficiency, the framework outputs information about the black-box function. Convergence is proved, and experimental results show significant improvement compared to standard BO.