Evaluating Noisy Optimisation Algorithms: First Hitting Time is Problematic

TL;DR

This paper critiques the common use of First Hitting Time in noisy optimisation algorithm evaluation, arguing it overestimates performance and advocating for final solution quality as a more realistic measure.

Contribution

It highlights the flaws of First Hitting Time as a stopping criterion and demonstrates the importance of evaluating algorithms based on final solution quality in noisy settings.

Findings

First Hitting Time overestimates algorithm performance in noisy problems.

Final solution quality provides a more realistic assessment.

Different evaluation methods lead to contrasting conclusions.

Abstract

A key part of any evolutionary algorithm is fitness evaluation. When fitness evaluations are corrupted by noise, as happens in many real-world problems as a consequence of various types of uncertainty, a strategy is needed in order to cope with this. Resampling is one of the most common strategies, whereby each solution is evaluated many times in order to reduce the variance of the fitness estimates. When evaluating the performance of a noisy optimisation algorithm, a key consideration is the stopping condition for the algorithm. A frequently used stopping condition in runtime analysis, known as "First Hitting Time", is to stop the algorithm as soon as it encounters the optimal solution. However, this is unrealistic for real-world problems, as if the optimal solution were already known, there would be no need to search for it. This paper argues that the use of First Hitting Time, despite being a commonly used approach, is significantly flawed and overestimates the quality of many algorithms in real-world cases, where the optimum is not known in advance and has to be genuinely searched for. A better alternative is to measure the quality of the solution an algorithm returns after a fixed evaluation budget, i.e., to focus on final solution quality. This paper argues that focussing on final solution quality is more realistic and demonstrates cases where the results produced by each algorithm evaluation method lead to very different conclusions regarding the quality of each noisy optimisation algorithm.