FCDP: Fully Cached Data Parallel for Communication-Avoiding Large-Scale Training

TL;DR

FCDP introduces a caching strategy that leverages host memory as a fast cache to significantly reduce inter-node communication in large-scale GPU training, enabling higher throughput on commodity hardware.

Contribution

FCDP presents a novel communication-avoiding method that caches parameters in host memory, reducing inter-node communication by up to 99% while maintaining minimal GPU memory usage.

Findings

Achieves up to 100x higher throughput than ZeRO-3.

Reduces inter-node all-gather by 50%.

Maintains maximum batch size of ZeRO-3 on commodity clusters.

Abstract

Training billion-parameter models requires distributing model states across GPUs using fully sharded data parallel (i.e., ZeRO-3). While ZeRO-3 succeeds on clusters with high-bandwidth NVLink and InfiniBand interconnects, researchers with commodity hardware face severe inter-node all-gather bottlenecks. Existing optimizations take two approaches: GPU memory caching (MiCS, ZeRO++) trades memory capacity for reduced communication, triggering out-of-memory failures on large models; host memory offloading (ZeRO-Offload, ZeRO-Infinity) extends capacity but degrades throughput due to PCIe overhead. We observe that on bandwidth-limited clusters, host memory can serve not as an overflow tier but as a fast caching layer that outperforms inter-node communication. Based on this insight, we propose FCDP, which eliminates redundant inter-node communication while preserving ZeRO-3's minimal GPU memory footprint. FCDP caches forward-pass parameters in host memory and reuses them during the backward pass via fast intra-node all-gather, reducing inter-node all-gather by 50%. For parameter-efficient fine-tuning (PEFT), FCDP selectively communicates only trainable parameters to maximize caching, reducing inter-node traffic by over 99%. In our commodity cluster setup, FCDP achieves up to 100x higher throughput than ZeRO-3 and 51x higher than ZeRO++, while maintaining ZeRO-3's maximum batch size.