DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training

TL;DR

DORA is a scalable asynchronous RL system for language model training that significantly improves throughput and training speed while maintaining convergence, by introducing multi-version streaming rollout to address long-tailed trajectories.

Contribution

The paper presents DORA, a novel asynchronous RL training system with multi-version streaming rollout, addressing long-tailed trajectories and achieving higher efficiency without losing convergence.

Findings

DORA achieves 2-3x higher throughput than state-of-the-art systems.

In industrial settings, DORA accelerates RL training by 2-4x over synchronous methods.

Open-source models trained with DORA match advanced LLMs on reasoning benchmarks.

Abstract

Reinforcement learning (RL) has become a critical paradigm for LLM post-training, yet the rollout phase -- accounting for 50--80% of total step time -- is bottlenecked by skewed generation: long-tailed trajectories indispensable for model performance block the entire training pipeline. Asynchronous training offers a natural remedy by overlapping generation with training, but introduces a fundamental tension between efficiency and algorithmic correctness. We identify three constraints in asynchronous training to preserve convergence: intra-trajectory policy consistency, data integrity, and bounded staleness. Existing approaches fail to intrinsically address the long-tailed trajectory problem, which is further exacerbated by the imbalance characteristic of Mix-of-Experts models, or deviate from the standard RL training formulation, thereby hindering model convergence. Therefore, we propose DORA (Dynamic ORchestration for Asynchronous Rollout), which addresses this challenge through algorithm-system co-design. DORA introduces multi-version streaming rollout, a novel asynchronous paradigm that maintains multiple policy versions concurrently -- simultaneously achieving full bubble elimination without compromising algorithmic constraints. Experimental results demonstrate that our DORA system achieves substantial improvements in throughput -- up to 2--3 times higher than state-of-the-art systems on open-source benchmarks -- without compromising convergence. Furthermore, in large-scale industrial applications with tens of thousands of accelerators, DORA accelerates RL training by 2--4 times compared to synchronous training across various scenarios. The resultant open-source models, LongCat-Flash-Thinking, exhibit competitive performance on complex reasoning benchmarks, matching the capability of most advanced LLMs.