PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs

TL;DR

PC-MoE introduces a memory-efficient, privacy-preserving collaborative training method for large language models that maintains high accuracy and robustness while significantly reducing GPU memory usage.

Contribution

It proposes a novel decentralized MoE training framework that preserves data privacy and reduces memory requirements without sacrificing model performance.

Findings

Achieves near-centralized model performance on seven benchmarks.

Reduces GPU memory usage by approximately 70%.

Maintains robustness against reconstruction attacks.

Abstract

Mixture-of-Experts (MoE) has been gaining popularity due to its successful adaptation to large language models (LLMs). In this work, we introduce Privacy-preserving Collaborative Mixture-of-Experts (PC-MoE), which leverages the sparsity of the MoE architecture for memory-efficient decentralized collaborative LLM training, enabling multiple parties with limited GPU-memory and data resources to collectively train more capable LLMs than they could achieve individually. At the same time, this approach protects training data privacy of each participant by keeping training data, as well as parts of the forward pass signal and gradients locally within each party. By design, PC-MoE synergistically combines the strengths of distributed computation with strong confidentiality assurances. Unlike most privacy-preserving schemes, which pay for confidentiality with lower task accuracy, our framework breaks that trade-off: across seven popular LLM benchmarks, it almost matches (and sometimes exceeds) the performance and convergence rate of a fully centralized model, enjoys near 70% peak GPU RAM reduction, while being fully robust against reconstruction attacks.