AMUSE: Anytime Muon with Stable Gradient Evaluation

TL;DR

AMUSE introduces a novel optimization method that combines Muon's rapid progress with stable gradient evaluation, eliminating the need for learning rate schedules and enabling effective anytime training across vision and language tasks.

Contribution

The paper proposes AMUSE, a new optimizer that integrates Muon's fast bulk progress with stable averaging, improving performance without schedules.

Findings

AMUSE outperforms AdamW and Muon on vision and language tasks.

It achieves better performance-iteration trade-offs.

Supports anytime training without learning rate schedules.

Abstract

Modern deep learning commonly relies on AdamW with prescribed learning rate schedules, but recent works challenge both components: Schedule-Free optimization removes explicit schedules via iterate averaging, and Muon improves the update geometry by orthogonalizing momentum for matrix parameters. Despite Muon's strong empirical performance, its underlying mechanism remains partially understood. We study Muon through the river-valley loss landscape, where useful training progress occurs along a flat, low-curvature bulk subspace (the river), while high-curvature dominant directions form steep valley walls that induce oscillations. We empirically show that while Muon's orthogonalization accelerates river progress by increasing the bulk component, it also amplifies dominant-direction noise, causing oscillatory trajectories. Building on this, we propose Anytime MUon with Stable gradient Evaluation (AMUSE), which integrates Muon's rapid bulk progress with the stabilizing effect of Schedule-Free averaging. AMUSE uses a time-varying interpolation coefficient that initially evaluates gradients near the fast Muon sequence for rapid adaptation, then gradually shifts toward the stable averaged sequence to suppress valley-wall oscillations. As a result, AMUSE requires no learning rate schedules and supports anytime training. Across vision tasks and large language model pretraining, AMUSE consistently improves the performance-iteration Pareto frontier over (Schedule-Free) AdamW and Muon.