Primal-Dual Methods for Nonsmooth Nonconvex Optimization with Orthogonality Constraints

TL;DR

This paper introduces a primal-dual method for nonsmooth, nonconvex optimization with orthogonality constraints, achieving optimal iteration complexity and demonstrating superior scalability and efficiency.

Contribution

It proposes a retraction-free, single-loop primal-dual algorithm with linearized smoothing augmented Lagrangian for orthogonally constrained problems, matching best-known complexity results.

Findings

Achieves $O( ext{epsilon}^{-3})$ iteration complexity for $ ext{epsilon}$-KKT points.

Demonstrates asymptotic convergence under Kurdyka-Lojasiewicz property.

Shows superior computational efficiency and scalability in numerical experiments.

Abstract

Recent advancements in data science have significantly elevated the importance of orthogonally constrained optimization problems. The Riemannian approach has become a popular technique for addressing these problems due to the advantageous computational and analytical properties of the Stiefel manifold. Nonetheless, the interplay of nonsmoothness alongside orthogonality constraints introduces substantial challenges to current Riemannian methods, including scalability, parallelizability, complicated subproblems, and cumulative numerical errors that threaten feasibility. In this paper, we take a retraction-free primal-dual approach and propose a linearized smoothing augmented Lagrangian method specifically designed for nonsmooth and nonconvex optimization with orthogonality constraints. Our proposed method is single-loop and free of subproblem solving. We establish its iteration complexity of $O(\epsilon^{-3})$ for finding $\epsilon$-KKT points, matching the best-known results in the Riemannian optimization literature. Additionally, by invoking the standard Kurdyka-Lojasiewicz (KL) property, we demonstrate asymptotic sequential convergence of the proposed algorithm. Numerical experiments on both smooth and nonsmooth orthogonal constrained problems demonstrate the superior computational efficiency and scalability of the proposed method compared with state-of-the-art algorithms.