DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training

TL;DR

DistFlow is a fully distributed reinforcement learning framework for large language models that achieves near-linear scalability and significant efficiency improvements by eliminating centralized control and enabling independent worker operation.

Contribution

It introduces a novel multi-controller, fully distributed RL framework that enhances scalability and flexibility for LLM post-training.

Findings

Achieves near-linear scalability up to 1024 GPUs.

Up to 7x throughput improvement over SOTA frameworks.

Decouples resource configuration from execution logic.

Abstract

Reinforcement learning (RL) has become the pivotal post-training technique for large language model (LLM). Effectively scaling reinforcement learning is now the key to unlocking advanced reasoning capabilities and ensuring safe, goal-aligned behavior in the most powerful LLMs. Mainstream frameworks usually employ a hybrid-controller architecture where a single-controller dispatches the overall execution logic and manages overall data transfer and the multi-controller executes distributed computation. For large-scale reinforcement learning, minor load imbalances can introduce significant bottlenecks, ultimately constraining the scalability of the system. To address this limitation, we introduce DistFlow, a novel, fully distributed RL framework designed to break scaling barrier. We adopt a multi-controller paradigm that dispatches data transfer and execution tasks to all workers, which eliminates the centralized node. This allows each worker to operate independently, leading to near-linear scalability up to 1024 GPUs and dramatic efficiency gains. Furthermore, our architecture decouples resource configuration from execution logic, allowing each worker to have a unique execution flow, offering significant flexibility for rapid and cost-effective algorithmic experimentation. Extensive experiments show that DistFlow achieves excellent linear scalability and up to a 7x end-to-end throughput improvement in specific scenarios over state-of-the-art (SOTA) frameworks.