ROC-n-reroll: How verifier imperfection affects test-time scaling

TL;DR

This paper provides a theoretical analysis of how verifier imperfection impacts test-time scaling methods like Best-of-N and Rejection Sampling, revealing the role of the verifier's ROC curve in determining accuracy and performance limits.

Contribution

It introduces a formal characterization of test-time scaling performance based on the verifier's ROC curve, bridging the gap between empirical results and theoretical understanding.

Findings

Rejection Sampling outperforms Best-of-N at fixed compute.

Both methods converge to similar accuracy with infinite compute.

High-compute performance cannot be reliably predicted from low-compute observations.

Abstract

Test-time scaling aims to improve language model performance by leveraging additional compute during inference. Many works have empirically studied techniques such as Best-of-N (BoN) and Rejection Sampling (RS) that make use of a verifier to enable test-time scaling. However, to date there is little theoretical understanding of how verifier imperfection affects performance -- a gap we address in this work. Specifically, we prove that the instance-level accuracy of these methods is precisely characterized by the geometry of the verifier's ROC curve. Our theory has two important takeaways, confirmed by experiments with Qwen and LLama models on GSM8K and MATH500. First, RS outperforms BoN for fixed compute, while both methods converge to the same accuracy in the infinite-compute limit. Second, it is generally impossible to predict the high-compute performance of either method based on observations in the low-compute regime.