Using the Distribution of Performance for Studying Statistical NLP Systems and Corpora

TL;DR

This paper advocates for reporting performance distributions in statistical NLP evaluations, derived from data resampling, to better quantify differences and reduce reliance on single performance metrics.

Contribution

It introduces the use of performance distributions from resampling as a more informative evaluation method in statistical NLP.

Findings

Performance distributions provide more reliable comparisons.

Resampling reduces the impact of sampling noise.

Statistical statements about system differences are improved.

Abstract

Statistical NLP systems are frequently evaluated and compared on the basis of their performances on a single split of training and test data. Results obtained using a single split are, however, subject to sampling noise. In this paper we argue in favour of reporting a distribution of performance figures, obtained by resampling the training data, rather than a single number. The additional information from distributions can be used to make statistically quantified statements about differences across parameter settings, systems, and corpora.