A Support-Set Algorithm for Optimization Problems with Nonnegative and Orthogonal Constraints

TL;DR

This paper introduces a support-set algorithm for optimization problems with nonnegative and orthogonal constraints, leveraging structural properties for computational efficiency and demonstrating strong empirical performance.

Contribution

The paper presents a novel support-set algorithm that preserves feasibility and converges globally for a class of constrained optimization problems, with proven iteration complexity.

Findings

Algorithm converges to a first-order stationary point.

Supports real-world applications like PCA, clustering, and community detection.

Achieves efficient computation via closed-form solutions for subproblems.

Abstract

In this paper, we investigate optimization problems with nonnegative and orthogonal constraints, where any feasible matrix of size $n \times p$ exhibits a sparsity pattern such that each row accommodates at most one nonzero entry. Our analysis demonstrates that, by fixing the support set, the global solution of the minimization subproblem for the proximal linearization of the objective function can be computed in closed form with at most $n$ nonzero entries. Exploiting this structural property offers a powerful avenue for dramatically enhancing computational efficiency. Guided by this insight, we propose a support-set algorithm preserving strictly the feasibility of iterates. A central ingredient is a strategically devised update scheme for support sets that adjusts the placement of nonzero entries. We establish the global convergence of the support-set algorithm to a first-order stationary point, and show that its iteration complexity required to reach an $\epsilon$-approximate first-order stationary point is $O (\epsilon^{-2})$. Numerical results are strongly in favor of our algorithm in real-world applications, including nonnegative PCA, clustering, and community detection.