More accurate tests for the statistical significance of result differences

TL;DR

This paper evaluates the accuracy of statistical significance tests in NLP, revealing that many underestimate significance due to independence assumptions, and highlights more reliable alternatives like randomization tests.

Contribution

It identifies limitations of common significance tests in NLP and advocates for more accurate methods that do not rely on independence assumptions.

Findings

Many standard tests underestimate significance

Independence assumption often violated in NLP metrics

Randomization tests provide more reliable significance assessment

Abstract

Statistical significance testing of differences in values of metrics like recall, precision and balanced F-score is a necessary part of empirical natural language processing. Unfortunately, we find in a set of experiments that many commonly used tests often underestimate the significance and so are less likely to detect differences that exist between different techniques. This underestimation comes from an independence assumption that is often violated. We point out some useful tests that do not make this assumption, including computationally-intensive randomization tests.