Selector-Guided Autonomous Curriculum for One-Shot Reinforcement Learning from Verifiable Rewards

TL;DR

This paper introduces a learnable selector model for autonomous curriculum learning in one-shot reinforcement learning, improving reasoning accuracy by focusing on output disagreement over reward variance.

Contribution

It proposes a novel Selector-Guided Autonomous Curriculum (SGAC) that uses multiple features, especially output disagreement, to select training instances, outperforming heuristic-based methods.

Findings

SGAC achieves 68.0% accuracy on Hendrycks MATH benchmark.

Output disagreement is a stronger predictor of reasoning gains than reward variance.

SGAC outperforms state-of-the-art models and previous RLVR checkpoints.

Abstract

Recently, Reinforcement Learning from Verifiable Rewards (RLVR) has been established as a highly effective technique for augmenting the math reasoning skills of Large Language Models (LLMs) based on a single instance. Current state-of-the-art 1-shot RLVR models adopt heuristics for selecting instances, mostly based on historical variance in rewards, which we find to be inherently misleading as a measure of transferability value. In this paper, we propose a Selector-Guided Autonomous Curriculum (SGAC) approach, which employs a learnable selector model on a multi-dimensional feature space consisting of success probability, reward variance, output disagreement (entropy), and semantic difficulty level, instead of the static reward variance heuristic. In our empirical evaluation on pools of candidate problems, we observed that output disagreement, rather than reward variance, is the strongest predictor of reasoning gains in subsequent iterations. Leveraging this finding, we develop an autonomous curriculum algorithm for dynamically siphoning candidate problems from a large pool, ranking them by the learned selector, and running micro-bursts of 1-shot GRPO. Our framework is evaluated using the Hendrycks MATH benchmark, with the Qwen2.5-Math-1.5B model serving as the baseline. Our framework obtains an accuracy of 68.0\% on the hold-out dataset, which is better than the accuracy obtained from the state-of-the-art model, 64.0\%, as well as the 1-shot RLVR checkpoint proposed by Wang et al., which achieved an accuracy of 66.0\%. The results confirm that entropy-based intelligent data curation leads to strict reasoning improvement over static training methods, particularly in severely limited data conditions.