Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch

TL;DR

This paper enhances distributed training of large language models by reducing communication bandwidth through parameter subset synchronization, concurrent training during synchronization, and data quantization, enabling efficient billion-scale model training.

Contribution

It introduces a novel combination of techniques to significantly lower bandwidth requirements in distributed LLM training without sacrificing model quality.

Findings

Bandwidth reduced by two orders of magnitude.

Able to train billion-scale models efficiently.

Maintains training quality comparable to traditional methods.

Abstract

Training of large language models (LLMs) is typically distributed across a large number of accelerators to reduce training time. Since internal states and parameter gradients need to be exchanged at each and every single gradient step, all devices need to be co-located using low-latency high-bandwidth communication links to support the required high volume of exchanged bits. Recently, distributed algorithms like DiLoCo have relaxed such co-location constraint: accelerators can be grouped into ``workers'', where synchronizations between workers only occur infrequently. This in turn means that workers can afford being connected by lower bandwidth communication links without affecting learning quality. However, in these methods, communication across workers still requires the same peak bandwidth as before, as the synchronizations require all parameters to be exchanged across all workers. In this paper, we improve DiLoCo in three ways. First, we synchronize only subsets of parameters in sequence, rather than all at once, which greatly reduces peak bandwidth. Second, we allow workers to continue training while synchronizing, which decreases wall clock time. Third, we quantize the data exchanged by workers, which further reduces bandwidth across workers. By properly combining these modifications, we show experimentally that we can distribute training of billion-scale parameters and reach similar quality as before, but reducing required bandwidth by two orders of magnitude.