Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraints

TL;DR

This paper introduces new practical and theoretical algorithms for optimization under orthogonality constraints that are faster and easier to implement than traditional Riemannian methods, with proven convergence and promising experimental results.

Contribution

It extends the landing algorithm to the Stiefel manifold and develops stochastic and variance-reduction variants with comparable convergence rates to Riemannian methods.

Findings

All proposed methods converge to the manifold at the same rate as Riemannian algorithms.

The algorithms are computationally cheaper and easier to implement.

Experimental results show effectiveness on machine learning problems with orthogonality constraints.

Abstract

Orthogonality constraints naturally appear in many machine learning problems, from principal component analysis to robust neural network training. They are usually solved using Riemannian optimization algorithms, which minimize the objective function while enforcing the constraint. However, enforcing the orthogonality constraint can be the most time-consuming operation in such algorithms. Recently, Ablin & Peyr\'e (2022) proposed the landing algorithm, a method with cheap iterations that does not enforce the orthogonality constraints but is attracted towards the manifold in a smooth manner. This article provides new practical and theoretical developments for the landing algorithm. First, the method is extended to the Stiefel manifold, the set of rectangular orthogonal matrices. We also consider stochastic and variance reduction algorithms when the cost function is an average of many functions. We demonstrate that all these methods have the same rate of convergence as their Riemannian counterparts that exactly enforce the constraint, and converge to the manifold. Finally, our experiments demonstrate the promise of our approach to an array of machine-learning problems that involve orthogonality constraints.