HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training

TL;DR

HALoS introduces a hierarchical asynchronous training framework for geo-distributed large language models, significantly reducing communication costs and training time while maintaining model accuracy.

Contribution

The paper presents HALoS, a novel hierarchical asynchronous optimization method with local and global parameter servers, improving efficiency and convergence in geo-distributed LLM training.

Findings

HALoS achieves up to 7.5x faster convergence than synchronous baselines.

HALoS outperforms existing asynchronous methods by up to 2.1x.

HALoS maintains or exceeds model accuracy of synchronous training.

Abstract

Training large language models (LLMs) increasingly relies on geographically distributed accelerators, causing prohibitive communication costs across regions and uneven utilization of heterogeneous hardware. We propose HALoS, a hierarchical asynchronous optimization framework that tackles these issues by introducing local parameter servers (LPSs) within each region and a global parameter server (GPS) that merges updates across regions. This hierarchical design minimizes expensive inter-region communication, reduces straggler effects, and leverages fast intra-region links. We provide a rigorous convergence analysis for HALoS under non-convex objectives, including theoretical guarantees on the role of hierarchical momentum in asynchronous training. Empirically, HALoS attains up to 7.5x faster convergence than synchronous baselines in geo-distributed LLM training and improves upon existing asynchronous methods by up to 2.1x. Crucially, HALoS preserves the model quality of fully synchronous SGD-matching or exceeding accuracy on standard language modeling and downstream benchmarks-while substantially lowering total training time. These results demonstrate that hierarchical, server-side update accumulation and global model merging are powerful tools for scalable, efficient training of new-era LLMs in heterogeneous, geo-distributed environments.