Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement

TL;DR

This paper introduces DoPR, a resource-efficient reinforcement learning method for large language models that reduces training costs by dynamically selecting informative samples, maintaining reasoning performance.

Contribution

It establishes a theoretical lower bound on sample complexity for reasoning in LLMs and proposes DoPR, a novel uncertainty-aware strategy that significantly reduces rollout overhead.

Findings

DoPR reduces rollout overhead by nearly tenfold.

Strong reasoning performance achieved with fewer training instances.

Theoretical lower bounds inform efficient training strategies.

Abstract

Large language models (LLMs) have exhibited remarkable performance on complex reasoning tasks, with reinforcement learning under verifiable rewards (RLVR) emerging as a principled framework for aligning model behavior with reasoning chains. Despite its promise, RLVR remains prohibitively resource-intensive, requiring extensive reward signals and incurring substantial rollout costs during training. In this work, we revisit the fundamental question of data and compute efficiency in RLVR. We first establish a theoretical lower bound on the sample complexity required to unlock reasoning capabilities, and empirically validate that strong performance can be achieved with a surprisingly small number of training instances. To tackle the computational burden, we propose Dynamic One-Shot Policy Refinement (DoPR), an uncertainty-aware RL strategy that dynamically selects a single informative training sample per batch for policy updates, guided by reward volatility and exploration-driven acquisition. DoPR reduces rollout overhead by nearly an order of magnitude while preserving competitive reasoning accuracy, offering a scalable and resource-efficient solution for LLM post-training. This approach offers a practical path toward more efficient and accessible RL-based training for reasoning-intensive LLM applications.