Two Criteria for Performance Analysis of Optimization Algorithms

TL;DR

This paper proposes two criteria, isomorphism and IIA, to improve the robustness and fairness of performance analysis in optimization algorithms, addressing logical paradoxes caused by current statistical practices.

Contribution

It introduces two novel criteria to ensure performance evaluations are unaffected by irrelevant factors, enhancing rigor in optimization algorithm assessments.

Findings

The isomorphism criterion ensures model-independent evaluation.

The IIA criterion prevents third-party influence on algorithm comparisons.

Frameworks for assessing performance metrics are proposed.

Abstract

Performance analysis is crucial in optimization research, especially when addressing black-box problems through nature-inspired algorithms. Current practices often rely heavily on statistical methods, which can lead to various logical paradoxes. To address this challenge, this paper introduces two criteria to ensure that performance analysis is unaffected by irrelevant factors. The first is the isomorphism criterion, which asserts that performance evaluation should remain unaffected by the modeling approach. The second is the IIA criterion,stating that comparisons between two algorithms should not be influenced by irrelevant third-party algorithms. Additionally, we conduct a comprehensive examination of the underlying causes of these paradoxes, identify conditions for checking the criteria, and propose ideas to tackle these issues. The criteria presented offer a framework for researchers to critically assess the performance metrics or ranking methods, ultimately aiming to enhance the rigor of evaluation metrics and ranking methods.