Insights on Muon from Simple Quadratics

TL;DR

This paper investigates the empirical success of Muon optimization by analyzing its behavior on simple quadratic functions, revealing effects beyond traditional local and worst-case analyses.

Contribution

It uncovers how polar approximation errors and structural properties influence Muon's performance, challenging existing theoretical explanations.

Findings

Polar step approximation errors can improve finite-time performance.

Structural properties of objectives affect optimization constants.

Existing theories overlook these effects, requiring new explanations.

Abstract

Muon updates weight matrices along (approximate) polar factors of the gradients and has shown strong empirical performance in large-scale training. Existing attempts at explaining its performance largely focus on single-step comparisons (on quadratic proxies) and worst-case guarantees that treat the inexactness of the polar-factor as a nuisance ``to be argued away''. We show that already on simple strongly convex functions such as $L(W)=\frac12\|W\|_{\text{F}}^2$, these perspectives are insufficient, suggesting that understanding Muon requires going beyond local proxies and pessimistic worst-case bounds. Instead, our analysis exposes two observations that already affect behavior on simple quadratics and are not well captured by prevailing abstractions: (i) approximation error in the polar step can qualitatively alter discrete-time dynamics and improve reachability and finite-time performance -- an effect practitioners exploit to tune Muon, but that existing theory largely treats as a pure accuracy compromise; and (ii) structural properties of the objective affect finite-budget constants beyond the prevailing conditioning-based explanations. Thus, any general theory covering these cases must either incorporate these ingredients explicitly or explain why they are irrelevant in the regimes of interest.