On the Power of (Approximate) Reward Models for Inference-Time Scaling

TL;DR

This paper analyzes how approximate reward models can effectively guide inference-time scaling in large language models, showing that bounded Bellman error enables exponential efficiency gains with polynomial complexity.

Contribution

It provides a theoretical framework linking Bellman error bounds of approximate reward models to the efficiency of SMC-based inference in large language models.

Findings

Bounded Bellman error by O(1/T) ensures polynomial complexity

Approximate reward models can achieve exponential efficiency gains

Theoretical justification for using approximate rewards in inference scaling

Abstract

Inference-time scaling has recently emerged as a powerful paradigm for improving the reasoning capability of large language models. Among various approaches, Sequential Monte Carlo (SMC) has become a particularly important framework, enabling iterative generation, evaluation, rejection, and resampling of intermediate reasoning trajectories. A central component in this process is the reward model, which evaluates partial solutions and guides the allocation of computation during inference. However, in practice, true reward models are never available. All deployed systems rely on approximate reward models, raising a fundamental question: Why and when do approximate reward models suffice for effective inference-time scaling? In this work, we provide a theoretical answer. We identify the Bellman error of the approximate reward model as the key quantity governing the effectiveness of SMC-based inference-time scaling. For a reasoning process of length $T$, we show that if the Bellman error of the approximate reward model is bounded by $O(1/T)$, then combining this reward model with SMC reduces the computational complexity of reasoning from exponential in $T$ to polynomial in $T$. This yields an exponential improvement in inference efficiency despite using only approximate rewards.