Stronger Random Baselines for In-Context Learning

TL;DR

This paper introduces a stronger random baseline for evaluating in-context learning in language models, accounting for validation reuse and small datasets, which better predicts performance and avoids false positives.

Contribution

It proposes a maximum random baseline that improves upon the standard baseline by considering the best among multiple random classifiers, especially useful for small datasets and validation reuse.

Findings

Over 20% of few-shot results surpass the standard baseline but not the new stronger baseline.

The stronger baseline better predicts held-out test performance.

It serves as an easy-to-calculate replacement for the standard baseline.

Abstract

Evaluating the in-context learning classification performance of language models poses challenges due to small dataset sizes, extensive prompt-selection using the validation set, and intentionally difficult tasks that lead to near-random performance. The standard random baseline--the expected accuracy of guessing labels uniformly at random--is stable when the evaluation set is used only once or when the dataset is large. We account for the common practice of validation set reuse and existing small datasets with a stronger random baseline: the expected maximum accuracy across multiple random classifiers. When choosing the best prompt demonstrations across six quantized language models applied to 16 BIG-bench Lite tasks, more than 20% of the few-shot results that exceed the standard baseline do not exceed this stronger random baseline. When held-out test sets are available, this stronger baseline is also a better predictor of held-out performance than the standard baseline, avoiding unnecessary test set evaluations. This maximum random baseline provides an easily calculated drop-in replacement for the standard baseline.