Highly Parallelized Reinforcement Learning Training with Relaxed Assignment Dependencies

TL;DR

This paper introduces TianJi, a high-throughput distributed reinforcement learning system that relaxes assignment dependencies, enabling asynchronous communication and achieving significant speedups in training convergence and data transmission efficiency.

Contribution

TianJi is the first system to relax assignment dependencies in DRL training, improving parallelization, scalability, and convergence speed while maintaining convergence guarantees.

Findings

Achieves up to 4.37x faster convergence time.

Scales to 8 nodes with 1.6x convergence and 7.13x throughput speedup.

Outperforms existing systems in data transmission efficiency.

Abstract

As the demands for superior agents grow, the training complexity of Deep Reinforcement Learning (DRL) becomes higher. Thus, accelerating training of DRL has become a major research focus. Dividing the DRL training process into subtasks and using parallel computation can effectively reduce training costs. However, current DRL training systems lack sufficient parallelization due to data assignment between subtask components. This assignment issue has been ignored, but addressing it can further boost training efficiency. Therefore, we propose a high-throughput distributed RL training system called TianJi. It relaxes assignment dependencies between subtask components and enables event-driven asynchronous communication. Meanwhile, TianJi maintains clear boundaries between subtask components. To address convergence uncertainty from relaxed assignment dependencies, TianJi proposes a distributed strategy based on the balance of sample production and consumption. The strategy controls the staleness of samples to correct their quality, ensuring convergence. We conducted extensive experiments. TianJi achieves a convergence time acceleration ratio of up to 4.37 compared to related comparison systems. When scaled to eight computational nodes, TianJi shows a convergence time speedup of 1.6 and a throughput speedup of 7.13 relative to XingTian, demonstrating its capability to accelerate training and scalability. In data transmission efficiency experiments, TianJi significantly outperforms other systems, approaching hardware limits. TianJi also shows effectiveness in on-policy algorithms, achieving convergence time acceleration ratios of 4.36 and 2.95 compared to RLlib and XingTian. TianJi is accessible at https://github.com/HiPRL/TianJi.git.