Phases of Muon: When Muon Eclipses SignSGD

TL;DR

This paper analyzes the behavior of spectral optimizers Muon and SignSGD, revealing their different convergence properties and phase-dependent performance on high-dimensional least squares problems.

Contribution

It provides explicit deterministic dynamics for Muon and SignSGD, clarifying their preconditioning effects and phase transitions based on data covariance structure.

Findings

SignSVD performs square-root preconditioning at large batch sizes.

Small batch sizes cause smaller eigenmodes to behave like SGD, slowing convergence.

Three phases exist in data covariance space where either SignSGD or SignSVD is favored.

Abstract

Recently, Muon and related spectral optimizers have demonstrated strong empirical performance as scalable stochastic methods, often outperforming Adam. Yet their behaviour remains poorly understood. We analyze stochastic spectral optimizers, including Muon, on a high-dimensional matrix-valued least squares problem. We derive explicit deterministic dynamics that provide a tractable framework for studying learning behaviour with a focus on (stochastic) SignSVD, which Muon approximates, and (stochastic) SignSGD, the latter serving as a proxy for Adam. Our analysis shows that for large batch size, SignSVD performs a square-root preconditioning with respect to the data covariance spectrum, while for small batch size smaller eigenmodes behave like SGD, slowing down convergence. We contrast with SignSGD which for generic covariance performs no preconditioning and has no transition, leading to different optimal learning rates and convergence characteristics. The two methods match up to a constant factor with isotropic data, but behave differently with anisotropic data. An analysis of a power law covariance model with data exponent $\alpha$ and target exponent $\beta$ shows there are three phases in the $(\alpha,\beta)$ plane: one where SignSGD is uniformly favored, one where SignSVD is uniformly favored, and a third where the two methods exhibit a trade-off in performance.