Lagom: Unleashing the Power of Communication and Computation Overlapping for Distributed LLM Training

TL;DR

Lagom is a system that optimizes communication and computation overlap in distributed large-model training, significantly improving training speed by intelligently tuning communication parameters using a cost model and search algorithm.

Contribution

Lagom introduces a unified cost model and a priority-based search algorithm to efficiently optimize communication parameters for overlapping communication and computation in distributed training.

Findings

Achieves 1.07-1.33x speedup over NCCL on high-bandwidth clusters.

Achieves 1.03-1.27x speedup over AutoCCL on diverse models.

Reduces optimization complexity from exponential to linear.

Abstract

Overlapping communication with computation is crucial for distributed large-model training, yet optimizing it - especially when computation becomes the bottleneck-remains challenging. We present Lagom, a system that co-tunes communication parameters to balance resource usage between computation and communication. By introducing a unified cost model and a priority-based search algorithm, Lagom reduces optimization complexity from exponential to linear. Evaluations on high- and low-bandwidth GPU clusters show that Lagom achieves 1.07-1.33x and 1.03-1.27x speedup over NCCL and AutoCCL across diverse models and parallelizations.