Posthoc Verification and the Fallibility of the Ground Truth

TL;DR

This paper critically examines the limitations of traditional evaluation metrics for classifiers that rely on human-annotated ground truth labels, proposing a posthoc verification approach that reveals models can outperform or match the quality of the original annotations.

Contribution

It introduces a posthoc verification methodology for entity linking that challenges the reliability of ground truth labels and suggests more realistic evaluation practices.

Findings

Posthoc verification shows models perform as well or better than ground truth.

Traditional metrics may underestimate model capabilities due to noisy labels.

Ground truth labels can be validated and questioned using posthoc methods.

Abstract

Classifiers commonly make use of pre-annotated datasets, wherein a model is evaluated by pre-defined metrics on a held-out test set typically made of human-annotated labels. Metrics used in these evaluations are tied to the availability of well-defined ground truth labels, and these metrics typically do not allow for inexact matches. These noisy ground truth labels and strict evaluation metrics may compromise the validity and realism of evaluation results. In the present work, we discuss these concerns and conduct a systematic posthoc verification experiment on the entity linking (EL) task. Unlike traditional methodologies, which asks annotators to provide free-form annotations, we ask annotators to verify the correctness of annotations after the fact (i.e., posthoc). Compared to pre-annotation evaluation, state-of-the-art EL models performed extremely well according to the posthoc evaluation methodology. Posthoc validation also permits the validation of the ground truth dataset. Surprisingly, we find predictions from EL models had a similar or higher verification rate than the ground truth. We conclude with a discussion on these findings and recommendations for future evaluations.