Communication-Efficient Gluon in Federated Learning

TL;DR

This paper introduces Gluon, a communication-efficient federated learning algorithm utilizing variance reduction and compression techniques, demonstrating superior performance in reducing communication costs for large models.

Contribution

Gluon extends Muon optimizers with layer-wise smoothness and compression, achieving faster convergence and lower communication costs in federated learning.

Findings

Gluon achieves faster convergence than previous methods.

The proposed algorithms significantly reduce communication costs.

Numerical experiments confirm the effectiveness of the compression techniques.

Abstract

Recent developments have shown that Muon-type optimizers based on linear minimization oracles (LMOs) over non-Euclidean norm balls have the potential to get superior practical performance than Adam-type methods in the training of large language models. Since large-scale neural networks are trained across massive machines, communication cost becomes the bottleneck. To address this bottleneck, we investigate Gluon, which is an extension of Muon under the more general layer-wise $(L^0, L^1)$-smooth setting, with both unbiased and contraction compressors. In order to reduce the compression error, we employ the variance reduced technique in SARAH in our compressed methods. The convergence rates and improved communication cost are achieved under certain conditions. As a byproduct, a new variance reduced algorithm with faster convergence rate than Gluon is obtained. We also incorporate momentum variance reduction (MVR) to these compressed algorithms and comparable communication cost is derived under weaker conditions when $L_i^1 \neq 0$. Finally, several numerical experiments are conducted to verify the superior performance of our compressed algorithms in terms of communication cost.