Smoothing algorithms for nonsmooth and nonconvex minimization over the stiefel manifold

TL;DR

This paper introduces three algorithms combining smoothing techniques and existing methods to efficiently solve nonsmooth, nonconvex optimization problems on the Stiefel manifold, with proven convergence to stationary points.

Contribution

The paper develops novel smoothing algorithms for nonsmooth, nonconvex problems on the Stiefel manifold, utilizing the Moreau envelope and demonstrating convergence properties.

Findings

Algorithms effectively solve nonsmooth nonconvex problems.

Numerical experiments show efficiency on graph Fourier basis tasks.

Convergence to stationary points is theoretically guaranteed.

Abstract

We consider a class of nonsmooth and nonconvex optimization problems over the Stiefel manifold where the objective function is the summation of a nonconvex smooth function and a nonsmooth Lipschitz continuous convex function composed with an linear mapping. We propose three numerical algorithms for solving this problem, by combining smoothing methods and some existing algorithms for smooth optimization over the Stiefel manifold. In particular, we approximate the aforementioned nonsmooth convex function by its Moreau envelope in our smoothing methods, and prove that the Moreau envelope has many favorable properties. Thanks to this and the scheme for updating the smoothing parameter, we show that any accumulation point of the solution sequence generated by the proposed algorithms is a stationary point of the original optimization problem. Numerical experiments on building graph Fourier basis are conducted to demonstrate the efficiency of the proposed algorithms.