Reusing Pre-Training Data at Test Time is a Compute Multiplier

TL;DR

This paper shows that test-time retrieval significantly enhances language model accuracy, acting as a compute multiplier, and reveals that current pre-training methods underutilize available dataset information.

Contribution

The study quantifies the dataset value left unused by pre-training and demonstrates how retrieval at test time can substantially improve model performance across multiple benchmarks.

Findings

Retrieval at test time acts as a ~5x compute multiplier for accuracy.

Additional compute for parsing retrieved context yields a 10% improvement on MMLU.

Pre-training methods leave significant dataset information untapped, indicating room for progress.

Abstract

Large language models learn from their vast pre-training corpora, gaining the ability to solve an ever increasing variety of tasks; yet although researchers work to improve these datasets, there is little effort to understand how efficient the pre-training apparatus is at extracting ideas and knowledge from the data. In this work, we use retrieval augmented generation along with test-time compute as a way to quantify how much dataset value was left behind by the process of pre-training, and how this changes across scale. We demonstrate that pre-training then retrieving from standard and largely open-sourced datasets results in significant accuracy gains in MMLU, Math-500, and SimpleQA, which persist through decontamination. For MMLU we observe that retrieval acts as a ~5x compute multiplier versus pre-training alone. We show that these results can be further improved by leveraging additional compute at test time to parse the retrieved context, demonstrating a 10 percentage point improvement on MMLU for the public LLaMA 3.1 8B model. Overall, our results suggest that today's pre-training methods do not make full use of the information in existing pre-training datasets, leaving significant room for progress.