Better-Than-Chance Classification for Signal Detection

TL;DR

This paper critically evaluates the effectiveness of using classifier accuracy as a statistical test for signal detection, showing it is often underpowered and proposing alternative methods for better detection.

Contribution

The paper demonstrates that accuracy-based tests are less powerful than multivariate tests for detecting differences, and suggests improvements like Leave-One-Out Bootstrap for classifier evaluation.

Findings

Accuracy-based tests have lower detection power than multivariate tests.

Discrete nature and regularization reduce accuracy test effectiveness.

Leave-One-Out Bootstrap improves classifier evaluation power.

Abstract

The estimated accuracy of a classifier is a random quantity with variability. A common practice in supervised machine learning, is thus to test if the estimated accuracy is significantly better than chance level. This method of signal detection is particularly popular in neuroimaging and genetics. We provide evidence that using a classifier's accuracy as a test statistic can be an underpowered strategy for finding differences between populations, compared to a bona-fide statistical test. It is also computationally more demanding than a statistical test. Via simulation, we compare test statistics that are based on classification accuracy, to others based on multivariate test statistics. We find that probability of detecting differences between two distributions is lower for accuracy based statistics. We examine several candidate causes for the low power of accuracy tests. These causes include: the discrete nature of the accuracy test statistic, the type of signal accuracy tests are designed to detect, their inefficient use of the data, and their regularization. When the purposes of the analysis is not signal detection, but rather, the evaluation of a particular classifier, we suggest several improvements to increase power. In particular, to replace V-fold cross validation with the Leave-One-Out Bootstrap.