Derivation of Closed Form of Expected Improvement for Gaussian Process Trained on Log-Transformed Objective

TL;DR

This paper provides a clear derivation of the closed-form expected improvement for Gaussian processes trained on log-transformed objectives, addressing numerical challenges and improving Bayesian optimization performance.

Contribution

It offers the first detailed derivation of the closed-form EI for GPs on log-transformed objectives, enhancing understanding and implementation.

Findings

Derivation clarifies the mathematical form of EI for log-transformed GPs

Improves numerical stability and optimization performance

Facilitates more accurate Bayesian optimization strategies

Abstract

Expected Improvement (EI) is arguably the most widely used acquisition function in Bayesian optimization. However, it is often challenging to enhance the performance with EI due to its sensitivity to numerical precision. Previously, Hutter et al. (2009) tackled this problem by using Gaussian process trained on the log-transformed objective function and it was reported that this trick improves the predictive accuracy of GP, leading to substantially better performance. Although Hutter et al. (2009) offered the closed form of their EI, its intermediate derivation has not been provided so far. In this paper, we give a friendly derivation of their proposition.