Multivariate Comparison of Classification Algorithms

TL;DR

This paper advocates for multivariate statistical tests, like Hotelling's T-squared and MANOVA, to compare classification algorithms across multiple performance measures simultaneously, revealing differences univariate tests might miss.

Contribution

It introduces the application of multivariate tests to algorithm comparison, demonstrating their higher power and ability to extract distinguishing performance measures.

Findings

Multivariate tests detect differences univariate tests may miss.

Multivariate analysis can identify key performance measures distinguishing algorithms.

Multivariate tests show higher statistical power in algorithm comparison.

Abstract

Statistical tests that compare classification algorithms are univariate and use a single performance measure, e.g., misclassification error, $F$ measure, AUC, and so on. In multivariate tests, comparison is done using multiple measures simultaneously. For example, error is the sum of false positives and false negatives and a univariate test on error cannot make a distinction between these two sources, but a 2-variate test can. Similarly, instead of combining precision and recall in $F$ measure, we can have a 2-variate test on (precision, recall). We use Hotelling's multivariate $T^2$ test for comparing two algorithms, and when we have three or more algorithms we use the multivariate analysis of variance (MANOVA) followed by pairwise post hoc tests. In our experiments, we see that multivariate tests have higher power than univariate tests, that is, they can detect differences that univariate tests cannot. We also discuss how multivariate analysis allows us to automatically extract performance measures that best distinguish the behavior of multiple algorithms.