Muon is Not That Special: Random or Inverted Spectra Work Just as Well

TL;DR

This paper challenges the importance of geometric structure in optimization algorithms, showing that performance can be achieved through randomness and local quantities like alignment and descent potential.

Contribution

It introduces Freon and Kaon optimizers, demonstrating that geometric structure is not essential for optimization success, and highlights the role of local quantities in step-size tuning.

Findings

Freon interpolates between SGD and Muon, working well in quasi-norm regimes.

Kaon, with random noise, matches Muon's performance despite lacking geometric structure.

Performance is governed by alignment and descent potential, not strict geometric properties.

Abstract

The recent empirical success of the Muon optimizer has renewed interest in non-Euclidean optimization, typically justified by similarities with second-order methods, and linear minimization oracle (LMO) theory. In this paper, we challenge this geometric narrative through three contributions, demonstrating that precise geometric structure is not the key factor affecting optimization performance. First, we introduce Freon, a family of optimizers based on Schatten (quasi-)norms, powered by a novel, provably optimal QDWH-based iterative approximation. Freon naturally interpolates between SGD and Muon, while smoothly extrapolating into the quasi-norm regime. Empirically, the best-performing Schatten parameters for GPT-2 lie strictly within the quasi-norm regime, and thus cannot be represented by any unitarily invariant LMO. Second, noting that Freon performs well across a wide range of exponents, we introduce Kaon, an absurd optimizer that replaces singular values with random noise. Despite lacking any coherent geometric structure, Kaon matches Muon's performance and retains classical convergence guarantees, proving that strict adherence to a precise geometry is practically irrelevant. Third, having shown that geometry is not the primary driver of performance, we demonstrate it is instead controlled by two local quantities: alignment and descent potential. Ultimately, each optimizer must tune its step size around these two quantities. While their dynamics are difficult to predict a-priori, evaluating them within a stochastic random feature model yields a precise insight: Muon succeeds not by tracking an ideal global geometry, but by guaranteeing step-size optimality.