Uniform Spectral Growth and Convergence of Muon in LoRA-Style Matrix Factorization

TL;DR

This paper investigates the spectral dynamics of Muon in LoRA-style matrix factorization, revealing uniform singular value growth and proving convergence properties of the spectral gradient flow in simplified models.

Contribution

It uncovers the uniform spectral growth phenomenon in Muon for LoRA fine-tuning and provides theoretical analysis of spectral gradient flow convergence and dynamics.

Findings

Singular values grow uniformly across the spectrum under Muon in LoRA.

Spectral gradient flow converges to global minima from almost all initializations.

Smaller singular values reach their targets earlier than larger ones, contrasting standard gradient flow.

Abstract

Spectral gradient descent (SpecGD) orthogonalizes the matrix parameter updates and has inspired practical optimizers such as Muon. They often perform well in large language model (LLM) training, but their dynamics remain poorly understood. In the low-rank adaptation (LoRA) setting, where weight updates are parameterized as a product of two low-rank factors, we find a distinctive spectral phenomenon under Muon in LoRA fine-tuning of LLMs: singular values of the LoRA product show near-uniform growth across the spectrum, despite orthogonalization being performed on the two factors separately. Motivated by this observation, we analyze spectral gradient flow (SpecGF)-a continuous-time analogue of SpecGD-in a simplified LoRA-style matrix factorization setting and prove "equal-rate" dynamics: all singular values grow at equal rates up to small deviations. Consequently, smaller singular values attain their target values earlier than larger ones, sharply contrasting with the largest-first stepwise learning observed in standard gradient flow. Moreover, we prove that SpecGF in our setting converges to global minima from almost all initializations, provided the factor norms remain bounded; with $\ell_2$ regularization, we obtain global convergence. Lastly, we corroborate our theory with experiments in the same setting.