Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo

TL;DR

This paper presents a training-free, reward-guided decoding framework for large language models using Sequential Monte Carlo methods, improving sequence-level quality without retraining models.

Contribution

It introduces a novel inference-time sampling approach that optimizes sequence-level rewards, surpassing existing decoding strategies and reinforcement learning methods.

Findings

Significant performance improvements on code generation and mathematical reasoning tasks.

Up to 54.9% performance gain on HumanEval with 7B models.

Outperforms reinforcement learning method GRPO across benchmarks.

Abstract

We introduce a principled probabilistic framework for reward-guided decoding in large language models, addressing the limitations of standard decoding methods that optimize token-level likelihood rather than sequence-level quality. Our method defines a reward-augmented target distribution over complete sequences by combining model transition probabilities with prefix-dependent reward potentials. Importantly, the approach is training-free: it leaves model weights unchanged and instead modifies the inference distribution via reward potentials, with all gains arising purely from inference-time sampling. To sample from this distribution, we develop Sequential Monte Carlo algorithms, including a computationally efficient prefix-only variant and a lookahead variant whose intermediate targets match the exact marginals of the full sequence distribution. The framework also integrates resample-move updates with Metropolis-Hastings rejuvenation and supports block-wise generation, subsuming common decoding strategies such as temperature sampling and power-tempered objectives. Empirical results across three 7B models show significant gains. On code generation (HumanEval), our method improves base performance by up to 54.9% and surpasses the strongest sampling baselines by 9.1%-15.3%. On mathematical reasoning (MATH500), it achieves gains of up to 8.8%. Notably, it reaches 87.8% on HumanEval and 78.4% on MATH500 with Qwen2.5-7B, consistently outperforming the reinforcement learning method GRPO.