Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?

TL;DR

This paper introduces MoPPS, a Bayesian framework for online prompt difficulty prediction that accelerates reinforcement learning finetuning of reasoning models by reducing costly LLM interactions.

Contribution

It presents MoPPS, a novel Bayesian risk-predictive method for online prompt difficulty estimation that improves training efficiency without extensive LLM evaluations.

Findings

MoPPS accurately predicts prompt difficulty across tasks.

It significantly reduces the number of LLM rollouts needed.

Training acceleration is achieved without sacrificing performance.

Abstract

Recent advances have witnessed the effectiveness of reinforcement learning (RL) finetuning in enhancing the reasoning capabilities of large language models (LLMs). The optimization process often requires numerous iterations to achieve satisfactory performance, resulting in high computational costs due to the need for frequent prompt evaluations under intensive LLM interactions and repeated policy updates. Appropriate online prompt selection methods reduce iteration steps by prioritizing informative prompts during training, while the pipeline's reliance on exhaustive prompt evaluation and subset selection for optimization still incurs substantial computational overhead due to frequent LLM inference calls. Distinguished from these direct evaluate-then-select schemes, this work investigates iterative approximate evaluation for arbitrary prompts and introduces Model Predictive Prompt Selection (MoPPS), a Bayesian risk-predictive framework that online estimates prompt difficulty without requiring costly LLM interactions. Technically, MoPPS models each prompt's success rate as a latent variable, performs streaming Bayesian inference, and employs posterior sampling in a constructed multi-armed bandit machine, enabling sample efficient and adaptive prompt selection. Extensive experiments across mathematics, planning, and vision-based geometry tasks show that MoPPS reliably predicts prompt difficulty and accelerates training with significantly reduced LLM rollouts. Our code is available at https://github.com/thu-rllab/MoPPS.