ROSE: Rollout On Serving GPUs via Cooperative Elasticity for Agentic RL

TL;DR

ROSE introduces a cooperative elasticity system that dynamically shares GPU resources between serving and rollout workloads in agentic RL, significantly improving throughput and reducing rollout time while maintaining service quality.

Contribution

It presents a novel system architecture for elastic GPU sharing in agentic RL, combining co-serving execution, fast weight transfer, and dynamic scheduling.

Findings

End-to-end throughput improved by up to 3.3x.

Rollout time reduced by up to 1.5x.

No serving SLO violations observed.

Abstract

Agentic reinforcement learning (RL) is reshaping LLM post-training, but end-to-end training time is dominated by compute-intensive, multi-turn rollouts whose resource demand varies significantly across training steps. Resource-fixed systems cannot adapt to this variation, while resource-elastic approaches that provision external GPUs on demand suffer from high allocation overhead and limited availability. We observe that serving clusters leave substantial GPU compute and memory idle, and propose cooperative elasticity: sharing already-deployed serving GPUs with rollout workloads to provide on-demand elastic capacity. Realizing this is non-trivial, as it must preserve serving SLOs under bursty traffic while minimizing cross-cluster communication overhead. We present ROSE, a system that realizes cooperative elasticity for agentic RL post-training, comprising three components: (1) an SLO-safe co-serving executor that co-locates heterogeneous serving and rollout models on the same GPUs, dynamically sharing memory and compute while preserving serving SLOs; (2) a cross-cluster weight transfer engine that leverages shard-aware routing and weight sparsity for fast synchronization; and (3) an elastic rollout scheduler that dynamically routes rollouts across dedicated and opportunistic serving GPUs. Experiments across multiple model sizes and cluster scales show that ROSE improves end-to-end throughput by 1.3 - 3.3 x over resource-fixed baselines and reduces rollout time by 1.2 - 1.5 x over resource-elastic baselines, with no serving SLO violations.