Learning Generative Selection for Best-of-N

TL;DR

This paper demonstrates that small reasoning models can learn effective generative selection strategies through reinforcement learning, significantly improving their ability to select correct solutions in reasoning tasks, thus enabling scalable test-time inference.

Contribution

The authors introduce a reinforcement learning approach to train small models for generative selection, achieving performance comparable to larger models on reasoning benchmarks.

Findings

Small models outperform baselines in math and code reasoning tasks.

Reinforcement learning enables small models to generalize to stronger model outputs.

Models often match or surpass larger models in selection quality.

Abstract

Scaling test-time compute via parallel sampling can substantially improve LLM reasoning, but is often limited by Best-of-N selection quality. Generative selection methods, such as GenSelect, address this bottleneck, yet strong selection performance remains largely limited to large models. We show that small reasoning models can acquire strong GenSelect capabilities through targeted reinforcement learning. To this end, we synthesize selection tasks from large-scale math and code instruction datasets by filtering to instances with both correct and incorrect candidate solutions, and train 1.7B-parameter models with DAPO to reward correct selections. Across math (AIME24, AIME25, HMMT25) and code (LiveCodeBench) reasoning benchmarks, our models consistently outperform prompting and majority-voting baselines, often approaching or exceeding much larger models. Moreover, these gains generalize to selecting outputs from stronger models despite training only on outputs from weaker models. Overall, our results establish reinforcement learning as a scalable way to unlock strong generative selection in small models, enabling efficient test-time scaling.