Adaptive Estimation of the Number of Algorithm Runs in Stochastic Optimization

TL;DR

This paper presents an online, adaptive method for estimating the number of runs needed in stochastic optimization experiments, significantly reducing computational effort while maintaining accuracy.

Contribution

It introduces a novel empirical, online approach that dynamically adjusts the number of algorithm runs based on data robustness, improving efficiency in benchmarking stochastic algorithms.

Findings

Achieved 82%-95% accuracy in run number estimation.

Reduced the number of runs by approximately 50%.

Enhanced benchmarking efficiency and energy sustainability.

Abstract

Determining the number of algorithm runs is a critical aspect of experimental design, as it directly influences the experiment's duration and the reliability of its outcomes. This paper introduces an empirical approach to estimating the required number of runs per problem instance for accurate estimation of the performance of the continuous single-objective stochastic optimization algorithm. The method leverages probability theory, incorporating a robustness check to identify significant imbalances in the data distribution relative to the mean, and dynamically adjusts the number of runs during execution as an online approach. The proposed methodology was extensively tested across two algorithm portfolios (104 Differential Evolution configurations and the Nevergrad portfolio) and the COCO benchmark suite, totaling 5748000 runs. The results demonstrate 82% - 95% accuracy in estimations across different algorithms, allowing a reduction of approximately 50% in the number of runs without compromising optimization outcomes. This online calculation of required runs not only improves benchmarking efficiency, but also contributes to energy reduction, fostering a more environmentally sustainable computing ecosystem.