Improving Reproducibility in Evaluation through Multi-Level Annotator Modeling

TL;DR

This paper proposes a multi-level bootstrapping method to model annotator behavior, analyzing how increasing annotations and raters improves reproducibility in AI evaluation.

Contribution

It introduces a novel approach to account for annotator variance using datasets with persistent rater identifiers, addressing reproducibility issues.

Findings

Analyzes tradeoffs between number of items and responses per item for significance.

Models individual annotator variance to improve evaluation reliability.

Provides insights into data collection strategies for reproducible AI assessments.

Abstract

As generative AI models such as large language models (LLMs) become more pervasive, ensuring the safety, robustness, and overall trustworthiness of these systems is paramount. However, AI is currently facing a reproducibility crisis driven by unreliable evaluations and unrepeatable experimental results. While human raters are often used to assess models for utility and safety, they introduce divergent biases and subjective opinions into their annotations. Overcoming this variance is exceptionally challenging because very little data exists to study how experimental repeatability actually improves as the annotator pool grows. Standard evaluation practices typically rely on a small number of annotations per item (often 3 to 5) and lack the persistent rater identifiers necessary to model individual variance across items. In this work, we introduce a multi-level bootstrapping approach to realistically model annotator behavior. Leveraging datasets with a large number of ratings and persistent rater identifiers, we analyze the tradeoffs between the number of items ($N$) and the number of responses per item ($K$) required to achieve statistical significance.