Manifold constrained steepest descent

TL;DR

This paper introduces MCSD, a single-loop optimization framework for manifold-constrained problems that improves stability and performance over traditional methods by efficiently selecting descent directions and ensuring convergence.

Contribution

The paper proposes MCSD, a novel single-loop manifold optimization method that simplifies tangent-space subproblem solutions and provides convergence guarantees, with a scalable spectral-norm variant SPEL.

Findings

MCSD achieves stable convergence under standard assumptions.

SPEL enables scalable implementations on the Stiefel manifold.

Experiments show competitive performance on PCA, CNNs, and LLM tuning.

Abstract

Norm-constrained linear minimization oracle (LMO)-based optimizers such as spectral gradient descent and Muon are attractive in large-scale learning, but extending them to manifold-constrained problems is nontrivial and often leads to nested-loop schemes that solve tangent-space subproblems iteratively. We propose \emph{Manifold Constrained Steepest Descent} (MCSD), a single-loop framework for optimization over manifolds that selects a norm-induced steepest-descent direction via an LMO applied to the Riemannian gradient, and then returns to the manifold via projection. Under standard smoothness assumptions, we establish convergence guarantees for MCSD and a stochastic momentum variant. We further introduce \emph{SPEL}, the spectral-norm specialization of MCSD on the Stiefel manifold, which admits scalable implementations via fast matrix sign computations. Experiments on PCA, orthogonality-constrained CNNs, and manifold-constrained LLM adapter tuning demonstrate improved stability and competitive performance relative to standard Riemannian baselines and existing manifold-aware LMO methods.