Scaling Test-Time Compute Without Verification or RL is Suboptimal

TL;DR

This paper demonstrates that verifier-based finetuning of large language models significantly outperforms verifier-free methods as test-time compute and data scale, especially for heterogeneous solution distributions.

Contribution

The paper proves the superiority of verifier-based finetuning over verifier-free approaches in scaling test-time compute for large language models.

Findings

Verifier-based methods outperform verifier-free methods as compute scales.

Performance gap widens with larger test-time budgets and data.

Empirical validation on multiple reasoning tasks with various model sizes.

Abstract

Despite substantial advances in scaling test-time compute, an ongoing debate in the community is how it should be scaled up to enable continued and efficient improvements with scaling. There are largely two approaches: first, distilling successful search or thinking traces; and second, using verification (e.g., 0/1 outcome rewards, reward models, or verifiers) to guide reinforcement learning (RL) and search algorithms. In this paper, we prove that finetuning LLMs with verifier-based (VB) methods based on RL or search is far superior to verifier-free (VF) approaches based on distilling or cloning search traces, given a fixed amount of compute/data budget. Further, we show that as we scale test-time compute (measured as the output token length) and training data, suboptimality of VF methods scales poorly compared to VB when the base pre-trained LLM presents a heterogeneous distribution over correct solution traces (e.g., different lengths, styles, etc.) and admits a non-sharp distribution over rewards on traces sampled from it. We formalize this condition using anti-concentration [Erd\H{o}s, 1945]. This implies a stronger result that VB methods scale better asymptotically, with the performance gap between VB and VF methods widening as test-time budget grows. We corroborate our theory empirically on both didactic and math reasoning problems with 3/8/32B-sized pre-trained LLMs, where we find verification is crucial for scaling test-time compute.