Learning How to Optimize Black-Box Functions With Extreme Limits on the Number of Function Evaluations

TL;DR

This paper introduces a novel hyperparameterized approach for black-box optimization with extremely limited function evaluations, significantly improving efficiency by learning to select promising candidate points in few-shot scenarios.

Contribution

It proposes a new method that combines diverse candidate points with an offline-optimized down-selection process for efficient black-box optimization under strict evaluation limits.

Findings

Achieves 50% reduction in normalized cost compared to state-of-the-art methods.

Effective in classical black box benchmarks with very few function evaluations.

Demonstrates the ability to adapt candidate selection based on optimization progress.

Abstract

We consider black-box optimization in which only an extremely limited number of function evaluations, on the order of around 100, are affordable and the function evaluations must be performed in even fewer batches of a limited number of parallel trials. This is a typical scenario when optimizing variable settings that are very costly to evaluate, for example in the context of simulation-based optimization or machine learning hyperparameterization. We propose an original method that uses established approaches to propose a set of points for each batch and then down-selects from these candidate points to the number of trials that can be run in parallel. The key novelty of our approach lies in the introduction of a hyperparameterized method for down-selecting the number of candidates to the allowed batch-size, which is optimized offline using automated algorithm configuration. We tune this method for black box optimization and then evaluate on classical black box optimization benchmarks. Our results show that it is possible to learn how to combine evaluation points suggested by highly diverse black box optimization methods conditioned on the progress of the optimization. Compared with the state of the art in black box minimization and various other methods specifically geared towards few-shot minimization, we achieve an average reduction of 50\% of normalized cost, which is a highly significant improvement in performance.