Nonlinear Eigen-approach ADMM for Sparse Optimization on Stiefel Manifold

TL;DR

This paper introduces a novel nonlinear eigen-approach ADMM method for sparse optimization on the Stiefel manifold, combining eigenvalue problems and proximal gradient techniques for efficient convergence.

Contribution

It proposes a new ADMM-based framework that transforms the problem into a nonlinear eigenvalue form and solves it efficiently, improving over existing methods.

Findings

Efficient convergence under mild assumptions.

Closed-form solution for the convex non-smooth part.

Effective handling of orthogonal constraints with sparsity.

Abstract

With the growing interest and applications in machine learning and data science, finding an efficient method to sparse analysis the high-dimensional data and optimizing a dimension reduction model to extract lower dimensional features has becoming more and more important. Orthogonal constraints (Stiefel manifold) is a commonly met constraint in these applications, and the sparsity is usually enforced through the element-wise L1 norm. Many applications can be found on optimization over Stiefel manifold within the area of physics and machine learning. In this paper, we propose a novel idea by tackling the Stiefel manifold through an nonlinear eigen-approach by first using ADMM to split the problem into smooth optimization over manifold and convex non-smooth optimization, and then transforming the former into the form of nonlinear eigenvalue problem with eigenvector dependency (NEPv) which is solved by self-consistent field (SCF) iteration, and the latter can be found to have an closed-form solution through proximal gradient. Compared with existing methods, our proposed algorithm takes the advantage of specific structure of the objective function, and has efficient convergence results under mild assumptions.