Convergence on a symmetric accelerated stochastic ADMM with larger stepsizes

TL;DR

This paper introduces a symmetric accelerated stochastic ADMM that allows larger stepsizes and achieves an O(1/T) convergence rate, improving efficiency for large-scale convex optimization problems with linear constraints.

Contribution

The paper proposes a novel symmetric accelerated stochastic ADMM with larger stepsizes and a flexible dual update, enhancing convergence properties for big-data convex optimization.

Findings

Achieves ergodic convergence rate of O(1/T) in expectation.

Effective for large-scale separable convex optimization problems.

Extensible to 3-block and accelerated stochastic augmented Lagrangian methods.

Abstract

In this paper, we develop a symmetric accelerated stochastic Alternating Direction Method of Multipliers (SAS-ADMM) for solving separable convex optimization problems with linear constraints. The objective function is the sum of a possibly nonsmooth convex function and an average function of many smooth convex functions. Our proposed algorithm combines both ideas of ADMM and the techniques of accelerated stochastic gradient methods possibly with variance reduction to solve the smooth subproblem. One main feature of SAS-ADMM is that its dual variable is symmetrically updated after each update of the separated primal variable, which would allow a more flexible and larger convergence region of the dual variable compared with that of standard deter-ministic or stochastic ADMM. This new stochastic optimization algorithm is shown to have ergodic converge in expectation with O(1/T) convergence rate, where T is the number of outer iterations. Our preliminary experiments indicate the proposed algorithm is very effective for solving separable optimization problems from big-data applications. Finally, 3-block extensions of the algorithm and its variant of an accelerated stochastic augmented Lagrangian method are discussed in the appendix.