Approaching Peak Ground Truth

TL;DR

This paper introduces the concept of Peak Ground Truth (PGT) to address the limitations of similarity-based evaluation in subjective annotation tasks, proposing a method to approximate PGT and strategies to enhance model performance.

Contribution

It presents the theoretical framework of PGT, a technique to estimate it using reliability measures, and reviews strategies for PGT-aware evaluation and improvement.

Findings

PGT defines the point where increased similarity no longer improves real-world performance.

A quantitative method to approximate PGT using inter- and intra-rater reliability is proposed.

Four categories of PGT-aware strategies for evaluation and improvement are reviewed.

Abstract

Machine learning models are typically evaluated by computing similarity with reference annotations and trained by maximizing similarity with such. Especially in the biomedical domain, annotations are subjective and suffer from low inter- and intra-rater reliability. Since annotations only reflect one interpretation of the real world, this can lead to sub-optimal predictions even though the model achieves high similarity scores. Here, the theoretical concept of PGT is introduced. PGT marks the point beyond which an increase in similarity with the \emph{reference annotation} stops translating to better RWMP. Additionally, a quantitative technique to approximate PGT by computing inter- and intra-rater reliability is proposed. Finally, four categories of PGT-aware strategies to evaluate and improve model performance are reviewed.