Towards a Principled Muon under $\mu\mathsf{P}$: Ensuring Spectral Conditions throughout Training

TL;DR

This paper introduces Muon++, a variant of the Muon optimizer that guarantees spectral conditions throughout training for large language models, enabling predictable scaling and reducing computational overhead.

Contribution

We propose a practical method to ensure spectral conditions for Muon during entire training, eliminating the need for repeated spectral normalization and enabling scalable, predictable LLM training.

Findings

Muon++ maintains spectral conditions throughout training.

Eliminates the need for explicit spectral normalization of weights.

Improves practical deployment of matrix-based optimizers in long-horizon training.

Abstract

The $\mu$-parameterization ($\mu$P) provides a principled foundation for large language model (LLM) training by prescribing width-independent learning dynamics, which in turn enables predictable scaling behavior and robust hyperparameter transfer across model sizes. A central requirement of $\mu$P is the satisfaction of certain spectral conditions on weight matrices, which ensure consistent feature learning and optimization behavior as model width grows. While these conditions are well understood in theory, guaranteeing their validity in practical training for matrix-based optimizers such as Muon is still under studied. Existing works that study Muon under $\mu$P exhibit important limitations: they either do not ensure that the spectral conditions hold throughout the entire training horizon, or require repeated spectral normalization (or Newton-Schulz iterations) applied to both weights and updates, leading to significant computational overhead and reduced practicality. In this work, we show how to reliably guarantee the spectral conditions required by $\mu$P for Muon during the entire training process. Our key insight is that for moderately large models, maintaining spectral control at the level of optimizer updates alone is sufficient to preserve $\mu$P-compatible scaling, eliminating the need for explicit spectral normalization of the weights. Based on this principle, we develop a variant of Muon, namely Muon++, that satisfies spectral condition throughout the training process. Our results bridge the gap between the theoretical promises of $\mu$P and the practical deployment of matrix-based optimizers in long-horizon training. We also take the first step towards an adaptive spectral condition by incorporating data-dependent effects, making it better suited for long-horizon LLM training.