REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations

TL;DR

REALM is a novel fine-tuning method for large language models that unsupervisedly learns annotator expertise, improving robustness and accuracy in noisy annotation scenarios across multiple benchmarks.

Contribution

It introduces a joint learning approach for model parameters and annotator expertise without supervision, extending to multi-task settings with an expertise matrix.

Findings

Outperforms naive noisy supervised fine-tuning in most settings.

Achieves up to 50% accuracy improvement in adversarial noise regimes.

Gains increase with larger model capacities.

Abstract

Supervised fine-tuning of large language models relies on human-annotated data, yet annotation pipelines routinely involve multiple crowdworkers of heterogeneous expertise. Standard practice aggregates labels via majority vote or simple averaging, discarding annotator identity and causing the model to absorb the errors of unreliable annotators directly into its parameters. We propose REALM, a method that jointly learns the model parameters and a scalar expertise value for each annotator entirely unsupervised, requiring no supervision beyond annotator identity. The key idea is to model each observed label as a mixture between the model's prediction and a uniform random guess, weighted by the annotator's learned expertise. We extend REALM to a multi-task setting via a learned expertise matrix that captures per-annotator reliability across tasks. We evaluate on five question answering benchmarks, fine-tuning three sizes of Flan-T5 under simulated noisy annotations. The proposed algorithm consistently outperforms the naive noisy SFT in the large majority of single- and multi-task settings, across datasets, model sizes, and noise types, with accuracy improvements of up to $50\%$ in the most adversarial regime and gains that grow with model capacity.