Sequential subspace methods on Stiefel manifold optimization

TL;DR

This paper introduces sequential subspace methods for optimizing quadratic functions over Stiefel manifolds, reducing computational complexity and ensuring convergence to high-quality solutions with potential acceleration via Newton directions.

Contribution

It proposes a novel SSM framework that guarantees convergence to qualified critical points in Stiefel manifold optimization, incorporating eigenvector and Newton directions for improved efficiency.

Findings

Guaranteed convergence to qualified critical points

Incorporation of Newton directions accelerates convergence

Effective reduction of high-dimensional problems to low-dimensional subproblems

Abstract

We investigate the minimization of a quadratic function over Stiefel manifolds (the set of all orthogonal $r$- frames in $\mathbf{R}^n$), which has applications in high-dimensional semi-supervised classification tasks. To reduce the computational complexity, we employ sequential subspace methods(SSM) to transform the high-dimensional problem to a series of low-dimensional ones. In this paper, our goal is to achieve an optimal solution of high quality, referred to as a ''qualified critical point". Qualified critical points are defined as those where the associated multiplier matrix meets specific upper-bound conditions. These points exhibit near-global optimality in quadratic optimization problems. In the context of a general quadratic, SSM generates a sequence of qualified critical points through low-dimensional surrogate regularized models. The convergence to a qualified critical point is guaranteed, when each SSM subspace is constructed from the following vectors: (i) a set of orthogonal unit vectors associated with the current iterate, (ii) a set of vectors representing the gradient of the objective, and (iii) a set of eigenvectors links to the smallest $r$ eigenvalues of the system matrix. Furthermore, incorporating Newton direction vectors into the subspaces can significantly accelerate the convergence of SSM.