Controlled LLM Training on Spectral Sphere

TL;DR

This paper introduces the Spectral Sphere Optimizer (SSO), a new optimization method for large language models that enforces spectral constraints to improve stability, convergence, and performance during training.

Contribution

The paper presents SSO, a spectral sphere-based optimizer fully aligned with Maximal Update Parametrization, enabling stable large-scale training of diverse models.

Findings

SSO outperforms AdamW and Muon in large-scale pretraining.

SSO improves model stability and activation bounds.

Enhanced load balancing and reduced outliers in MoE models.

Abstract

Scaling large models requires optimization strategies that ensure rapid convergence grounded in stability. Maximal Update Parametrization ($\boldsymbol{\mu}$P) provides a theoretical safeguard for width-invariant $\Theta(1)$ activation control, whereas emerging optimizers like Muon are only ``half-aligned'' with these constraints: they control updates but allow weights to drift. To address this limitation, we introduce the \textbf{Spectral Sphere Optimizer (SSO)}, which enforces strict module-wise spectral constraints on both weights and their updates. By deriving the steepest descent direction on the spectral sphere, SSO realizes a fully $\boldsymbol{\mu}$P-aligned optimization process. To enable large-scale training, we implement SSO as an efficient parallel algorithm within Megatron. Through extensive pretraining on diverse architectures, including Dense 1.7B, MoE 8B-A1B, and 200-layer DeepNet models, SSO consistently outperforms AdamW and Muon. Furthermore, we observe significant practical stability benefits, including improved MoE router load balancing, suppressed outliers, and strictly bounded activations.