SUDA-Muon: Structural Design Principles and Boundaries for Fully Decentralized Muon

TL;DR

SUDA-Muon introduces a unified framework for fully decentralized Muon algorithms, enabling modular design and providing convergence guarantees, with empirical validation on CIFAR-100 and GPT-2 tasks.

Contribution

It proposes SUDA, a primal-dual communication template that modularizes decentralized Muon algorithms and establishes convergence and boundary conditions.

Findings

SUDA-Muon achieves non-asymptotic convergence with topology-independent bounds.

Tracking-before-polarization is necessary to avoid non-stationary fixed points.

In non-IID settings, SUDA-Muon outperforms DeMuon in accuracy and loss.

Abstract

Fully decentralized Muon is difficult because its nonlinear matrix-sign operator does not commute with linear gossip averaging. This makes decentralized Muon a structural design problem: in designing the algorithm, one must distinguish modular components from non-modular ones. We propose \sudamuon{}, which realizes this separation through a unified primal--dual communication template called SUDA; within this template, ED/D$^2$, EXTRA, and gradient tracking become modular backbone choices. We prove a topology-separated non-asymptotic convergence guarantee in the nuclear-norm geometry: the dominant term scales as $\mathcal{O}((1+\sigma/\sqrt{N})K^{-1/4})$ and does not explicitly involve graph quantities, identifying the communication backbone as the modular axis in the structure design. We then establish two complementary non-modular boundaries. Internally, tracking-before-polarization is necessary for this natural no-tracking variant to avoid non-stationary fixed points under heterogeneous objectives. Externally, in the absence of a central server, a fully decentralized method cannot perform the federated average-then-polarize update; we show that this non-modular local-polarize-then-average design is the essential reason why can fail to exhibit linear speedup. Experiments on CIFAR-100 and GPT-2 fine-tuning support the same picture: the unified template makes different communication algorithms directly comparable. In mild near-IID regimes, the resulting variants perform similarly, while in the more difficult long-horizon non-IID CIFAR-100 setting, \sudamuon{} achieves higher accuracy and lower loss than \textsc{DeMuon}.