Should we really use post-hoc tests based on mean-ranks?

TL;DR

This paper critiques the common use of mean-ranks post-hoc tests after Friedman tests in algorithm comparison, highlighting their inconsistencies and advocating for pairwise tests like Wilcoxon instead.

Contribution

It identifies fundamental issues with mean-ranks post-hoc tests, demonstrating their dependence on the entire algorithm pool and proposing more reliable pairwise alternatives.

Findings

Mean-ranks test outcomes depend on the pool of algorithms.

Using mean-ranks can lead to paradoxical significance results.

Pairwise tests like Wilcoxon are recommended for more consistent comparisons.

Abstract

The statistical comparison of multiple algorithms over multiple data sets is fundamental in machine learning. This is typically carried out by the Friedman test. When the Friedman test rejects the null hypothesis, multiple comparisons are carried out to establish which are the significant differences among algorithms. The multiple comparisons are usually performed using the mean-ranks test. The aim of this technical note is to discuss the inconsistencies of the mean-ranks post-hoc test with the goal of discouraging its use in machine learning as well as in medicine, psychology, etc.. We show that the outcome of the mean-ranks test depends on the pool of algorithms originally included in the experiment. In other words, the outcome of the comparison between algorithms A and B depends also on the performance of the other algorithms included in the original experiment. This can lead to paradoxical situations. For instance the difference between A and B could be declared significant if the pool comprises algorithms C, D, E and not significant if the pool comprises algorithms F, G, H. To overcome these issues, we suggest instead to perform the multiple comparison using a test whose outcome only depends on the two algorithms being compared, such as the sign-test or the Wilcoxon signed-rank test.