Statistical Models for the Analysis of Optimization Algorithms with Benchmark Functions

TL;DR

This paper advocates for Bayesian data analysis in the evaluation of optimization algorithms, highlighting its advantages over traditional methods and providing practical models and guidance for implementation.

Contribution

It introduces five Bayesian statistical models for analyzing optimization algorithms and offers a comprehensive guide to applying these models in empirical research.

Findings

Bayesian methods improve validity of statistical analysis

Provided step-by-step guide and code for models

Enhanced transparency and robustness in benchmarking

Abstract

Frequentist statistical methods, such as hypothesis testing, are standard practice in papers that provide benchmark comparisons. Unfortunately, these methods have often been misused, e.g., without testing for their statistical test assumptions or without controlling for family-wise errors in multiple group comparisons, among several other problems. Bayesian Data Analysis (BDA) addresses many of the previously mentioned shortcomings but its use is not widely spread in the analysis of empirical data in the evolutionary computing community. This paper provides three main contributions. First, we motivate the need for utilizing Bayesian data analysis and provide an overview of this topic. Second, we discuss the practical aspects of BDA to ensure that our models are valid and the results transparent. Finally, we provide five statistical models that can be used to answer multiple research questions. The online appendix provides a step-by-step guide on how to perform the analysis of the models discussed in this paper, including the code for the statistical models, the data transformations and the discussed tables and figures.