Ranking of classification algorithms in terms of mean-standard deviation using A-TOPSIS

TL;DR

This paper introduces A-TOPSIS, a novel ranking method based on TOPSIS, to compare classification algorithms considering both mean and standard deviation of their performance, addressing limitations of traditional statistical tests.

Contribution

The paper proposes A-TOPSIS, a new multi-criteria decision-making approach for ranking algorithms using means and standard deviations, with case studies and a web framework for practical use.

Findings

A-TOPSIS effectively ranks classification algorithms considering mean and standard deviation.

The approach outperforms traditional statistical tests in algorithm comparison.

The web framework facilitates easy application of A-TOPSIS in research.

Abstract

In classification problems when multiples algorithms are applied to different benchmarks a difficult issue arises, i.e., how can we rank the algorithms? In machine learning it is common run the algorithms several times and then a statistic is calculated in terms of means and standard deviations. In order to compare the performance of the algorithms, it is very common to employ statistical tests. However, these tests may also present limitations, since they consider only the means and not the standard deviations of the obtained results. In this paper, we present the so called A-TOPSIS, based on TOPSIS (Technique for Order Preference by Similarity to Ideal Solution), to solve the problem of ranking and comparing classification algorithms in terms of means and standard deviations. We use two case studies to illustrate the A-TOPSIS for ranking classification algorithms and the results show the suitability of A-TOPSIS to rank the algorithms. The presented approach is general and can be applied to compare the performance of stochastic algorithms in machine learning. Finally, to encourage researchers to use the A-TOPSIS for ranking algorithms we also presented in this work an easy-to-use A-TOPSIS web framework.