Fast Stochastic Alternating Direction Method of Multipliers

TL;DR

This paper introduces a stochastic ADMM algorithm that achieves a faster convergence rate of O(1/T), matching batch ADMM performance without processing all samples each iteration, and demonstrates significant speed improvements in experiments.

Contribution

The paper presents a novel stochastic ADMM method with improved convergence rate of O(1/T), reducing computational complexity while maintaining accuracy.

Findings

Convergence rate improved from O(1/√T) to O(1/T) for convex problems.

Algorithm significantly faster than existing stochastic and batch ADMM methods.

Effective in graph-guided fused lasso applications.

Abstract

In this paper, we propose a new stochastic alternating direction method of multipliers (ADMM) algorithm, which incrementally approximates the full gradient in the linearized ADMM formulation. Besides having a low per-iteration complexity as existing stochastic ADMM algorithms, the proposed algorithm improves the convergence rate on convex problems from $O(\frac 1 {\sqrt{T}})$ to $O(\frac 1 T)$, where $T$ is the number of iterations. This matches the convergence rate of the batch ADMM algorithm, but without the need to visit all the samples in each iteration. Experiments on the graph-guided fused lasso demonstrate that the new algorithm is significantly faster than state-of-the-art stochastic and batch ADMM algorithms.