Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm

TL;DR

This paper introduces SPES, a memory-efficient decentralized framework for pretraining large mixture-of-experts language models across distributed GPUs, reducing memory use and communication costs while maintaining competitive performance.

Contribution

The authors propose a novel decentralized training method for MoE LLMs that trains only subsets of experts per node and introduces expert-merging warm-up, enabling training of large models with less memory and communication.

Findings

Trained a 2B-parameter MoE LLM on 16 GPUs with competitive results.

Successfully scaled to 7B and 9B models matching centralized baselines.

Achieved training over internet connections with reduced memory footprint.

Abstract

Pretraining large language models (LLMs) typically requires centralized clusters with thousands of high-memory GPUs (e.g., H100/A100). Recent decentralized training methods reduce communication overhead by employing federated optimization; however, they still need to train the entire model on each node, remaining constrained by GPU memory limitations. In this work, we propose SParse Expert Synchronization (SPES), a memory-efficient decentralized framework for pretraining mixture-of-experts (MoE) LLMs. SPES trains only a subset of experts per node, substantially lowering the memory footprint. Each node updates its local experts and periodically synchronizes with other nodes, eliminating full-parameter transmission while ensuring efficient knowledge sharing. To accelerate convergence, we introduce an expert-merging warm-up strategy, where experts exchange knowledge early in training, to rapidly establish foundational capabilities. With SPES, we train a 2B-parameter MoE LLM using 16 standalone 48GB GPUs over internet connections, which achieves competitive performance with centrally trained LLMs under similar computational budgets. We further demonstrate scalability by training a 7B model from scratch and a 9B model upcycled from a dense checkpoint, both of which match prior centralized baselines. Our code is available at https://github.com/zjr2000/SPES.