Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL

TL;DR

This paper introduces PULSE, a novel approach leveraging weight update sparsity in distributed RL to drastically reduce communication costs while maintaining model fidelity.

Contribution

It proposes compute-visible sparsification and two algorithms, PULSESync and PULSELoCo, to significantly improve communication efficiency in distributed RL training.

Findings

PULSESync reduces weight-synchronization communication by over 100x.

PULSELoCo decreases trainer-to-trainer communication by over 17x.

Approximately 99% of weight updates are invisible after BF16 casting, enabling sparsification.

Abstract

Bandwidth-constrained distributed reinforcement learning (RL) post-training of large language models is bottlenecked by two channels: weight synchronization from trainers to inference workers, and gradient or pseudo-gradient synchronization across trainers. We find that approximately 99% of per-step weight updates are invisible after the BF16 cast used by standard training and inference forward passes. We explain this sparsity by showing that, at typical RL post-training learning rates, Adam updates often fall below the local BF16 rounding threshold. We turn this observation into an algorithmic principle called compute-visible sparsification: transmit only updates that would change the next forward pass. PULSE (Precision-gated Updates for Low-precision Sparse Exchange) turns this principle into two communication algorithms: PULSESync sends lossless sparse BF16 weight patches from trainers to inference workers, and PULSELoCo sparsifies DiLoCo-style FP32 pseudo-gradient synchronization with error feedback. Over bandwidth-constrained commodity networks, PULSESync cuts weight-synchronization communication by over 100x while reconstructing trainer weights bit-identically. PULSELoCo matches DiLoCo across four models while reducing trainer-to-trainer communication by over 17x versus DiLoCo and over 100x versus DDP in the largest evaluated setting.