A Smooth Locally Exact Penalty Method for Optimization Problems over Generalized Stiefel Manifolds

TL;DR

This paper introduces a Smooth Locally Exact Penalty method for optimization on generalized Stiefel manifolds, effectively handling singular matrices and reducing computational costs compared to traditional Riemannian approaches.

Contribution

The paper proposes a novel penalty model that simplifies optimization over generalized Stiefel manifolds, especially for singular matrices, with proven equivalence and convergence guarantees.

Findings

The penalty model is equivalent to the original problem at stationary points.

The approach reduces per-iteration computational costs compared to Riemannian methods.

Numerical experiments confirm the effectiveness and practical potential of the proposed method.

Abstract

In this paper, we consider a class of optimization problems constrained to the generalized Stiefel manifold. Such problems are fundamental to a wide range of real-world applications, including generalized canonical correlation analysis, linear discriminant analysis, and electronic structure calculations. Existing works mainly focuses on cases where the generalized orthogonality constraint is induced by a symmetric positive definite matrix M, a setting where the geometry essentially reduces to that of the standard Stiefel manifold. However, many practical scenarios involve a singular M, which introduces significant analytical and computational challenges. Therefore, we propose a Smooth Locally Exact Penalty model (SLEP) and establish its equivalence to the original problem in the aspect of stationary points under a finitly large penalty parameter. This penalty model admits the direct application of various unconstrained optimization techniques, with convergence guarantees inherited from established results. Compared to Riemannian optimization approaches, our proposed penalty mode eliminates the need for retractions and vector transports, hence significantly reducing per-iteration computational costs. Extensive numerical experiments validate our theoretical results and demonstrate the effectiveness and practical potential of the proposed penalty model SLEP.