Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations

TL;DR

This paper introduces a statistical framework for language model evaluations, emphasizing error bars and proper analysis to improve the reliability and informativeness of experimental results.

Contribution

It applies statistical methods from other sciences to LLM evaluations, providing formulas and recommendations for better experiment analysis and reporting.

Findings

Provides formulas for analyzing evaluation data.

Recommends best practices for experiment design.

Enhances reliability of language model evaluation results.

Abstract

Evaluations are critical for understanding the capabilities of large language models (LLMs). Fundamentally, evaluations are experiments; but the literature on evaluations has largely ignored the literature from other sciences on experiment analysis and planning. This article shows researchers with some training in statistics how to think about and analyze data from language model evaluations. Conceptualizing evaluation questions as having been drawn from an unseen super-population, we present formulas for analyzing evaluation data, measuring differences between two models, and planning an evaluation experiment. We make a number of specific recommendations for running language model evaluations and reporting experiment results in a way that minimizes statistical noise and maximizes informativeness.