An Inexact Accelerated Stochastic ADMM for Separable Convex Optimization

TL;DR

This paper introduces an inexact accelerated stochastic ADMM algorithm that efficiently solves structured convex optimization problems common in machine learning, achieving favorable convergence rates and demonstrating effectiveness in large-scale data applications.

Contribution

It develops a novel inexact stochastic ADMM method combining variance reduction and linearization, with proven convergence properties and practical effectiveness in big data scenarios.

Findings

Objective error and constraint violation are $ ext{O}(1/k)$.

Expected iterate error converges linearly under strong convexity.

Numerical experiments show superior performance over traditional ADMM.

Abstract

An inexact accelerated stochastic Alternating Direction Method of Multipliers (AS-ADMM) scheme is developed for solving structured separable convex optimization problems with linear constraints. The objective function is the sum of a possibly nonsmooth convex function and a smooth function which is an average of many component convex functions. Problems having this structure often arise in machine learning and data mining applications. AS-ADMM combines the ideas of both ADMM and the stochastic gradient methods using variance reduction techniques. One of the ADMM subproblems employs a linearization technique while a similar linearization could be introduced for the other subproblem. For a specified choice of the algorithm parameters, it is shown that the objective error and the constraint violation are $\mathcal{O}(1/k)$ relative to the number of outer iterations $k$. Under a strong convexity assumption, the expected iterate error converges to zero linearly. A linearized variant of AS-ADMM and incremental sampling strategies are also discussed. Numerical experiments with both stochastic and deterministic ADMM algorithms show that AS-ADMM can be particularly effective for structured optimization arising in big data applications.