ADMM for Nonconvex Optimization under Minimal Continuity Assumption

TL;DR

This paper proposes a new ADMM-based method for nonconvex optimization that requires minimal continuity assumptions, achieves convergence with proven complexity, and demonstrates effectiveness on sparse PCA problems.

Contribution

Introduces IPDS-ADMM, a novel ADMM variant that relaxes continuity requirements and provides the first complexity analysis for nonconvex nonsmooth problems.

Findings

Achieves an $ ext{O}(rac{1}{ ext{epsilon}^3})$ convergence complexity.

Requires continuity in only one block of the objective.

Demonstrates effectiveness on sparse PCA experiments.

Abstract

This paper introduces a novel approach to solving multi-block nonconvex composite optimization problems through a proximal linearized Alternating Direction Method of Multipliers (ADMM). This method incorporates an Increasing Penalization and Decreasing Smoothing (IPDS) strategy. Distinguishing itself from existing ADMM-style algorithms, our approach (denoted IPDS-ADMM) imposes a less stringent condition, specifically requiring continuity in just one block of the objective function. IPDS-ADMM requires that the penalty increases and the smoothing parameter decreases, both at a controlled pace. When the associated linear operator is bijective, IPDS-ADMM uses an over-relaxation stepsize for faster convergence; however, when the linear operator is surjective, IPDS-ADMM uses an under-relaxation stepsize for global convergence. We devise a novel potential function to facilitate our convergence analysis and prove an oracle complexity $\mathcal{O}(\epsilon^{-3})$ to achieve an $\epsilon$-approximate critical point. To the best of our knowledge, this is the first complexity result for using ADMM to solve this class of nonsmooth nonconvex problems. Finally, some experiments on the sparse PCA problem are conducted to demonstrate the effectiveness of our approach.