Randomness as Reference: Benchmark Metric for Optimization in Engineering

TL;DR

This paper introduces a new benchmark suite of engineering optimization problems and a performance metric based on random sampling, enabling more realistic and unbiased evaluation of optimization algorithms.

Contribution

It presents a comprehensive engineering-focused benchmark suite and a novel random sampling-based metric for fair comparison of optimization methods.

Findings

Few algorithms perform consistently well across engineering problems.

Many metaheuristics show significant efficiency loss on engineering tasks.

The new benchmark and metric improve evaluation relevance for real-world applications.

Abstract

Benchmarking optimization algorithms is fundamental for the advancement of computational intelligence. However, widely adopted artificial test suites exhibit limited correspondence with the diversity and complexity of real-world engineering optimization tasks. This paper presents a new benchmark suite comprising 235 bounded, continuous, unconstrained optimization problems, the majority derived from engineering design and simulation scenarios, including computational fluid dynamics and finite element analysis models. In conjunction with this suite, a novel performance metric is introduced, which employs random sampling as a statistical reference, providing nonlinear normalization of objective values and enabling unbiased comparison of algorithmic efficiency across heterogeneous problems. Using this framework, 20 deterministic and stochastic optimization methods were systematically evaluated through hundreds of independent runs per problem, ensuring statistical robustness. The results indicate that only a few of the tested optimization methods consistently achieve excellent performance, while several commonly used metaheuristics exhibit severe efficiency loss on engineering-type problems, emphasizing the limitations of conventional benchmarks. Furthermore, the conducted tests are used for analyzing various features of the optimization methods, providing practical guidelines for their application. The proposed test suite and metric together offer a transparent, reproducible, and practically relevant platform for evaluating and comparing optimization methods, thereby narrowing the gap between the available benchmark tests and realistic engineering applications.