Better than Average: Paired Evaluation of NLP Systems

TL;DR

This paper advocates for pairwise evaluation methods in NLP, like the Bradley-Terry model, over traditional averaging, demonstrating their impact on system ranking and providing tools for implementation.

Contribution

It introduces pairwise evaluation techniques for NLP system comparison, highlighting their advantages and offering practical tools for adoption.

Findings

Different aggregation methods lead to different system rankings in 30% of cases.

Pairwise methods better capture the relative performance of NLP systems.

The Bradley-Terry model improves evaluation accuracy over simple averages.

Abstract

Evaluation in NLP is usually done by comparing the scores of competing systems independently averaged over a common set of test instances. In this work, we question the use of averages for aggregating evaluation scores into a final number used to decide which system is best, since the average, as well as alternatives such as the median, ignores the pairing arising from the fact that systems are evaluated on the same test instances. We illustrate the importance of taking the instance-level pairing of evaluation scores into account and demonstrate, both theoretically and empirically, the advantages of aggregation methods based on pairwise comparisons, such as the Bradley-Terry (BT) model, a mechanism based on the estimated probability that a given system scores better than another on the test set. By re-evaluating 296 real NLP evaluation setups across four tasks and 18 evaluation metrics, we show that the choice of aggregation mechanism matters and yields different conclusions as to which systems are state of the art in about 30% of the setups. To facilitate the adoption of pairwise evaluation, we release a practical tool for performing the full analysis of evaluation scores with the mean, median, BT, and two variants of BT (Elo and TrueSkill), alongside functionality for appropriate statistical testing.