Entropy Minimization for Optimization of Expensive, Unimodal Functions

TL;DR

This paper introduces an entropy-based optimization algorithm for expensive, noisy, unimodal functions, improving hyperparameter tuning efficiency by reducing uncertainty about the optimum with limited evaluations.

Contribution

It proposes a novel entropy minimization approach that enhances the efficiency of optimizing costly, noisy unimodal functions by leveraging a sampled belief model and information metrics.

Findings

Efficient estimation of the surrogate objective in entropy search.

Reduction in the number of function evaluations needed.

Improved optimization performance on noisy, expensive functions.

Abstract

Maximization of an expensive, unimodal function under random observations has been an important problem in hyperparameter tuning. It features expensive function evaluations (which means small budgets) and a high level of noise. We develop an algorithm based on entropy reduction of a probabilistic belief about the optimum. The algorithm provides an efficient way of estimating the computationally intractable surrogate objective in the general Entropy Search algorithm by leveraging a sampled belief model and designing a metric that measures the information value of any search point.