Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Models

TL;DR

This paper introduces Infinite Sampling, a framework that enables large group reinforcement learning training for large language models by reducing memory overhead and increasing efficiency through micro sampling, continuous sampling, and a length-aware scheduler.

Contribution

It proposes a novel Infinite Sampling framework that decouples group size from memory constraints, improving scalability and stability in group-based RL training for LLMs.

Findings

Reduces peak memory usage by over 50%.

Improves throughput by over 25%.

Maintains full-length completions with larger groups.

Abstract

Group-based reinforcement learning algorithms such as Group Reward Policy Optimization (GRPO) have proven effective for fine-tuning large language models (LLMs) with human feedback. However, generating and storing multiple responses per prompt incurs substantial memory overhead, especially as the sample group size increases, limiting scalability under constrained hardware. We propose Infinite Sampling, a framework that enables efficient and stable GRPO training by decoupling group size from GPU memory usage. It consists of: (1) micro sampling groups that decompose large groups into memory-feasible rounds; (2) continuous sampling that interleaves generation across groups to improve utilization; and (3) a length-aware scheduler combining token-conditioned sequence length prediction with a two-stage plan: global grouping via FPTAS and runtime refill via SJF. Experiments show that our Micro Sampling Groups reduce peak memory usage by over 50% compared to full-group decoding (e.g., from 21.55 GB to 10.64 GB on Qwen3-1.7B). Building on this, Infinite Sampling improves throughput by over 25% compared to the naive micro sampling group method, reducing decoding steps while maintaining full-length completions and memory usage. Our hybrid scheduling ensures efficient and stable GRPO training with larger groups under realistic GPU memory constraints.