Local Linear Convergence of Infeasible Optimization with Orthogonal Constraints

TL;DR

This paper proves local linear convergence of the landing algorithm, an infeasible optimization method for orthogonality-constrained problems, showing it is as effective as retraction-based methods but more computationally efficient.

Contribution

It establishes a novel local linear convergence rate for the landing algorithm under a Riemannian P{\

Findings

Landing algorithm achieves convergence similar to state-of-the-art methods.

The method significantly reduces computational overhead.

Numerical experiments confirm theoretical results.

Abstract

Many classical and modern machine learning algorithms require solving optimization tasks under orthogonality constraints. Solving these tasks with feasible methods requires a gradient descent update followed by a retraction operation on the Stiefel manifold, which can be computationally expensive. Recently, an infeasible retraction-free approach, termed the landing algorithm, was proposed as an efficient alternative. Motivated by the common occurrence of orthogonality constraints in tasks such as principle component analysis and training of deep neural networks, this paper studies the landing algorithm and establishes a novel linear convergence rate for smooth non-convex functions using only a local Riemannian P{\L} condition. Numerical experiments demonstrate that the landing algorithm performs on par with the state-of-the-art retraction-based methods with substantially reduced computational overhead.