LoRDO: Distributed Low-Rank Optimization with Infrequent Communication

TL;DR

LoRDO is a novel framework that combines low-rank optimization with infrequent communication in distributed training, significantly reducing communication costs while maintaining high performance in language models.

Contribution

LoRDO introduces a unified approach that addresses the limitations of low-rank optimizers in local-update regimes by incorporating a full-rank quasi-hyperbolic update, enabling efficient distributed training.

Findings

Achieves near-parity with low-rank DDP in language modeling.

Reduces communication by approximately 10 times.

Improves performance in low-memory settings with small rank/batch size.

Abstract

Distributed training of foundation models via $\texttt{DDP}$ is limited by interconnect bandwidth. While infrequent communication strategies reduce synchronization frequency, they remain bottlenecked by the memory and communication requirements of optimizer states. Low-rank optimizers can alleviate these constraints; however, in the local-update regime, workers lack access to the full-batch gradients required to compute low-rank projections, which degrades performance. We propose $\texttt{LoRDO}$, a principled framework unifying low-rank optimization with infrequent synchronization. We first demonstrate that, while global projections based on pseudo-gradients are theoretically superior, they permanently restrict the optimization trajectory to a low-rank subspace. To restore subspace exploration, we introduce a full-rank quasi-hyperbolic update. $\texttt{LoRDO}$ achieves near-parity with low-rank $\texttt{DDP}$ in language modeling and downstream tasks at model scales of $125$M--$720$M, while reducing communication by $\approx 10 \times$. Finally, we show that $\texttt{LoRDO}$ improves performance even more in very low-memory settings with small rank/batch size.