Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes

TL;DR

This paper empirically studies how small language models perform with RLVR in low data settings, revealing that mixed complexity training enhances sample efficiency and model generalization.

Contribution

It provides the first comprehensive analysis of open-source SLMs with RLVR in low data regimes across diverse procedural datasets, highlighting benefits of mixed complexity training.

Findings

Procedural datasets enable controllable evaluation and dataset development.

Models trained on lower complexity tasks can generalize to higher complexity tasks.

Training on mixed complexity datasets yields up to 5x sample efficiency in low data regimes.

Abstract

Fine-tuning Large Language Models (LLMs) typically relies on large quantities of high-quality annotated data, or questions with well-defined ground truth answers in the case of Reinforcement Learning with Verifiable Rewards (RLVR). While previous work has explored the benefits to model reasoning capabilities by scaling both data and compute used for RLVR, these results lack applicability in many real-world settings where annotated data and accessible compute may be scarce. In this work, we present a comprehensive empirical study of open-source Small Language Model (SLM) performance after RLVR in low data regimes. Across three novel datasets covering number counting problems, graph reasoning, and spatial reasoning, we characterize how model performance scales with dataset size, diversity, and complexity. We demonstrate that (1) procedural datasets allow for fine-grained evaluation and training dataset development with controllable properties (size, diversity, and complexity), (2) under RLVR, models trained on lower complexity tasks can generalize to higher complexity tasks, and (3) training on mixed complexity datasets is associated with the greatest benefits in low data regimes, providing up to 5x sample efficiency versus training on easy tasks. These findings inspire future work on the development of data scaling laws for RLVR and the use of procedural data generators to further understand effective data development for efficient LLM fine-tuning.