A Stochastic Alternating Direction Method of Multipliers for Non-smooth and Non-convex Optimization

TL;DR

This paper introduces a stochastic ADMM method enhanced with variance reduction for large-scale non-convex, non-smooth optimization, proving convergence under KL property and demonstrating effectiveness on practical problems.

Contribution

It combines stochastic gradient variance reduction with ADMM for large-scale non-convex problems, establishing convergence under KL property, and provides empirical validation.

Findings

Converges globally under KL property.

Effective on graph-guided fused Lasso and CT reconstruction.

Outperforms traditional ADMM in large-scale settings.

Abstract

Alternating direction method of multipliers (ADMM) is a popular first-order method owing to its simplicity and efficiency. However, similar to other proximal splitting methods, the performance of ADMM degrades significantly when the scale of the optimization problems to solve becomes large. In this paper, we consider combining ADMM with a class of stochastic gradient with variance reduction for solving large-scale non-convex and non-smooth optimization problems. Global convergence of the generated sequence is established under the extra additional assumption that the object function satisfies Kurdyka-Lojasiewicz (KL) property. Numerical experiments on graph-guided fused Lasso and computed tomography are presented to demonstrate the performance of the proposed methods.