ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training

TL;DR

ACCO is a novel distributed training algorithm for large language models that reduces communication overhead and GPU idle time by synchronizing delayed gradients, enabling faster and more scalable training across heterogeneous hardware.

Contribution

We introduce ACCO, a memory-efficient optimization algorithm that synchronizes delayed gradients during training, improving scalability and efficiency over existing methods like ZeRO-1.

Findings

ACCO reduces training time compared to ZeRO-1.

Supports heterogeneous hardware environments effectively.

Maintains convergence properties similar to standard distributed optimization.

Abstract

Training LLMs relies on distributed implementations using multiple GPUs to compute gradients in parallel with sharded optimizers. However, synchronizing gradients in data parallel setups introduces communication overhead that grows with the number of workers, limiting parallelization efficiency. Local optimization algorithms reduce communications but incur high memory costs as they prevent optimizer state sharding, hindering scalability. To address this, we propose \textbf{AC}cumulate while \textbf{CO}mmunicate (ACCO), a memory-efficient optimization algorithm for distributed LLM training. By synchronizing delayed gradients while computing new ones, ACCO reduces GPU idle time and supports heterogeneous hardware. To mitigate the convergence issues caused by delayed updates, we introduce a novel technique ensuring training dynamics align with standard distributed optimization. Compared to ZeRO-1, our approach is significantly faster and scales effectively across heterogeneous hardware.