Collective Communication for 100k+ GPUs

TL;DR

This paper introduces NCCLX, a novel collective communication framework optimized for clusters exceeding 100,000 GPUs, significantly improving throughput and latency for large-scale LLM training and inference.

Contribution

The paper presents NCCLX, a new communication framework designed specifically for ultra-large GPU clusters, addressing scalability and performance challenges in LLM development.

Findings

Substantial improvements in communication efficiency on Llama4

Supports reliable, high-throughput, low-latency data exchange at 100k+ GPU scale

Enables next-generation large language models to operate efficiently at unprecedented scales

Abstract

The increasing scale of large language models (LLMs) necessitates highly efficient collective communication frameworks, particularly as training workloads extend to hundreds of thousands of GPUs. Traditional communication methods face significant throughput and latency limitations at this scale, hindering both the development and deployment of state-of-the-art models. This paper presents the NCCLX collective communication framework, developed at Meta, engineered to optimize performance across the full LLM lifecycle, from the synchronous demands of large-scale training to the low-latency requirements of inference. The framework is designed to support complex workloads on clusters exceeding 100,000 GPUs, ensuring reliable, high-throughput, and low-latency data exchange. Empirical evaluation on the Llama4 model demonstrates substantial improvements in communication efficiency. This research contributes a robust solution for enabling the next generation of LLMs to operate at unprecedented scales.