Orthogonal Directions Constrained Gradient Method: from non-linear equality constraints to Stiefel manifold

TL;DR

This paper introduces ODCGM, a simple and efficient algorithm for non-convex optimization on manifolds that avoids complex retractions and achieves near-optimal convergence rates, extending previous methods for Stiefel manifold optimization.

Contribution

The paper proposes ODCGM, a novel projection-based gradient method that simplifies manifold optimization and extends the analysis of existing algorithms with near-optimal complexity guarantees.

Findings

ODCGM converges towards the manifold with simple projections.

The method achieves near-optimal oracle complexities in deterministic and stochastic settings.

Numerical experiments demonstrate high-dimensional efficiency of ODCGM.

Abstract

We consider the problem of minimizing a non-convex function over a smooth manifold $\mathcal{M}$. We propose a novel algorithm, the Orthogonal Directions Constrained Gradient Method (ODCGM) which only requires computing a projection onto a vector space. ODCGM is infeasible but the iterates are constantly pulled towards the manifold, ensuring the convergence of ODCGM towards $\mathcal{M}$. ODCGM is much simpler to implement than the classical methods which require the computation of a retraction. Moreover, we show that ODCGM exhibits the near-optimal oracle complexities $\mathcal{O}(1/\varepsilon^2)$ and $\mathcal{O}(1/\varepsilon^4)$ in the deterministic and stochastic cases, respectively. Furthermore, we establish that, under an appropriate choice of the projection metric, our method recovers the landing algorithm of Ablin and Peyr\'e (2022), a recently introduced algorithm for optimization over the Stiefel manifold. As a result, we significantly extend the analysis of Ablin and Peyr\'e (2022), establishing near-optimal rates both in deterministic and stochastic frameworks. Finally, we perform numerical experiments which shows the efficiency of ODCGM in a high-dimensional setting.