Continuous Optimization Benchmarks by Simulation

TL;DR

This paper introduces a simulation-based approach using Gaussian processes to create more accurate continuous optimization benchmarks that better reflect real-world problem landscapes, overcoming limitations of predictive models.

Contribution

It presents a novel spectral simulation method for Gaussian processes to generate realistic benchmark functions for continuous optimization, improving upon previous prediction-based approaches.

Findings

Spectral simulation yields more accurate benchmarks than Gaussian process predictions.

Simulation preserves covariance properties, better reflecting true landscape features.

Method is effective for continuous optimization problems, as demonstrated in experiments.

Abstract

Benchmark experiments are required to test, compare, tune, and understand optimization algorithms. Ideally, benchmark problems closely reflect real-world problem behavior. Yet, real-world problems are not always readily available for benchmarking. For example, evaluation costs may be too high, or resources are unavailable (e.g., software or equipment). As a solution, data from previous evaluations can be used to train surrogate models which are then used for benchmarking. The goal is to generate test functions on which the performance of an algorithm is similar to that on the real-world objective function. However, predictions from data-driven models tend to be smoother than the ground-truth from which the training data is derived. This is especially problematic when the training data becomes sparse. The resulting benchmarks may not reflect the landscape features of the ground-truth, are too easy, and may lead to biased conclusions. To resolve this, we use simulation of Gaussian processes instead of estimation (or prediction). This retains the covariance properties estimated during model training. While previous research suggested a decomposition-based approach for a small-scale, discrete problem, we show that the spectral simulation method enables simulation for continuous optimization problems. In a set of experiments with an artificial ground-truth, we demonstrate that this yields more accurate benchmarks than simply predicting with the Gaussian process model.