Is the Best Better? Bayesian Statistical Model Comparison for Natural Language Processing

TL;DR

This paper introduces a Bayesian model comparison method using k-fold cross-validation to reliably evaluate and rank NLP models across multiple datasets and metrics, addressing concerns about standard split-based evaluations.

Contribution

It presents a novel Bayesian statistical approach for model comparison in NLP, improving upon traditional evaluation methods by considering model uncertainty and multiple datasets.

Findings

The method effectively ranks six POS taggers across datasets.

It estimates the probability of one model outperforming another.

It assesses model equivalence with statistical confidence.

Abstract

Recent work raises concerns about the use of standard splits to compare natural language processing models. We propose a Bayesian statistical model comparison technique which uses k-fold cross-validation across multiple data sets to estimate the likelihood that one model will outperform the other, or that the two will produce practically equivalent results. We use this technique to rank six English part-of-speech taggers across two data sets and three evaluation metrics.