Optimization without Retraction on the Random Generalized Stiefel Manifold

TL;DR

This paper introduces a stochastic iterative optimization method for the generalized Stiefel manifold that converges to critical points without retraction, reducing computational costs in applications like CCA, ICA, and GEVP.

Contribution

It presents a novel stochastic algorithm that avoids retraction, converging in expectation with lower per-iteration costs for problems involving generalized orthogonality constraints.

Findings

Method achieves convergence rates comparable to full-matrix approaches.

Experiments validate effectiveness in machine learning applications.

Requires only matrix multiplications, reducing computational complexity.

Abstract

Optimization over the set of matrices $X$ that satisfy $X^\top B X = I_p$, referred to as the generalized Stiefel manifold, appears in many applications involving sampled covariance matrices such as the canonical correlation analysis (CCA), independent component analysis (ICA), and the generalized eigenvalue problem (GEVP). Solving these problems is typically done by iterative methods that require a fully formed $B$. We propose a cheap stochastic iterative method that solves the optimization problem while having access only to random estimates of $B$. Our method does not enforce the constraint in every iteration; instead, it produces iterations that converge to critical points on the generalized Stiefel manifold defined in expectation. The method has lower per-iteration cost, requires only matrix multiplications, and has the same convergence rates as its Riemannian optimization counterparts that require the full matrix $B$. Experiments demonstrate its effectiveness in various machine learning applications involving generalized orthogonality constraints, including CCA, ICA, and the GEVP.