"All that Glitters": Approaches to Evaluations with Unreliable Model and Human Annotations

TL;DR

This paper explores methods to evaluate and improve the reliability of model and human annotations in tasks with inherently noisy labels, using large language models and novel assessment metrics across multiple quality dimensions.

Contribution

It introduces new evaluation approaches for label quality in noisy settings and demonstrates how LLMs can both outperform humans and reveal biases in classroom annotation tasks.

Findings

Encoder models achieve state-of-the-art results, even surpassing human performance.

Standard metrics can mask issues like biases and spurious correlations.

Using rigorous evaluation reveals racial biases and impacts human-model collaboration.

Abstract

"Gold" and "ground truth" human-mediated labels have error. The effects of this error can escape commonly reported metrics of label quality or obscure questions of accuracy, bias, fairness, and usefulness during model evaluation. This study demonstrates methods for answering such questions even in the context of very low reliabilities from expert humans. We analyze human labels, GPT model ratings, and transformer encoder model annotations describing the quality of classroom teaching, an important, expensive, and currently only human task. We answer the question of whether such a task can be automated using two Large Language Model (LLM) architecture families--encoders and GPT decoders, using novel approaches to evaluating label quality across six dimensions: Concordance, Confidence, Validity, Bias, Fairness, and Helpfulness. First, we demonstrate that using standard metrics in the presence of poor labels can mask both label and model quality: the encoder family of models achieve state-of-the-art, even "super-human", results across all classroom annotation tasks. But not all these positive results remain after using more rigorous evaluation measures which reveal spurious correlations and nonrandom racial biases across models and humans. This study then expands these methods to estimate how model use would change to human label quality if models were used in a human-in-the-loop context, finding that the variance captured in GPT model labels would worsen reliabilities for humans influenced by these models. We identify areas where some LLMs, within the generalizability of the current data, could improve the quality of expensive human ratings of classroom instruction.