Benchmarks Are Not That Out of Distribution: Word Overlap Predicts Performance

TL;DR

This paper shows that the performance of language models on benchmarks is largely predicted by the overlap in word-level statistics between training and evaluation data, challenging the idea that benchmarks are strongly out-of-distribution.

Contribution

It introduces a method to measure word overlap using unigram cross-entropy and demonstrates its strong correlation with benchmark performance across multiple datasets and models.

Findings

Word overlap predicts benchmark performance.

Larger datasets with similar word distributions improve results.

Benchmarks are only weakly out-of-distribution relative to training data.

Abstract

Understanding what constitutes high-quality pre-training data remains a central question in language model training. In this work, we investigate whether benchmark performance is primarily driven by the degree of statistical pattern overlap between pre-training corpora and evaluation datasets. We measure this overlap using word-level unigram cross-entropy and word frequency statistics, and perform controlled experiments across $10$ zero-shot benchmarks, $4$ pre-training datasets spanning $8.5\mathrm{B}$ to $60\mathrm{B}$ tokens, and model sizes ranging from $400\mathrm{M}$ to $3\mathrm{B}$ parameters. Our results demonstrate a robust inverse relationship between word-level unigram cross-entropy and benchmark performance, suggesting that widely used benchmarks are strongly influenced by word overlap between training and evaluation data. Thus, larger pre-training subsets with similar word-level unigram cross-entropy yield improved downstream results, indicating that word frequency statistics play an additional role in shaping benchmark scores. Taken together, these results suggest that many standard benchmarks are only weakly out-of-distribution relative to pre-training corpora, so that simple word-overlap statistics predict benchmark performance.