Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models

TL;DR

This paper introduces Dynamics-Predictive Sampling (DPS), a novel method that models prompt solving progress as a dynamical system to efficiently select training data, reducing computational costs and improving reasoning abilities of large language models during RL finetuning.

Contribution

DPS models prompt solving progress as a dynamical system using Bayesian inference, enabling efficient prompt selection without extensive rollouts in RL finetuning of large language models.

Findings

DPS significantly reduces the number of rollouts needed for training.

DPS accelerates RL finetuning compared to existing prompt selection methods.

DPS achieves superior reasoning performance across diverse tasks.

Abstract

Reinforcement learning (RL) finetuning has become a key technique for enhancing the reasoning abilities of large language models (LLMs). However, its effectiveness critically depends on the selection of training data. Recent advances underscore the importance of online prompt selection methods, which typically concentrate training on partially solved or moderately challenging examples under the current policy, thereby yielding more effective model updates. While significantly accelerating RL finetuning in terms of training steps, they also incur substantial computational overhead by requiring extensive LLM rollouts over large candidate batches to identify informative samples, an expense that can outweigh the finetuning process itself. To address this challenge, this work proposes Dynamics-Predictive Sampling (DPS), which online predicts and selects informative prompts by inferring their learning dynamics prior to costly rollouts. Specifically, we introduce a new perspective by modeling each prompt's solving progress during RL finetuning as a dynamical system, where the extent of solving is represented as the state and the transition is characterized by a hidden Markov model. Using historical rollout reward signals, we perform online Bayesian inference to estimate evolving state distributions, and the inference outcome provides a predictive prior for efficient prompt selection without rollout-intensive filtering. Empirical results across diverse reasoning tasks, including mathematics, planning, and visual geometry, demonstrate that DPS substantially reduces redundant rollouts, accelerates the training process, and achieves superior reasoning performance.