Stochastic ADMM with batch size adaptation for nonconvex nonsmooth optimization

TL;DR

This paper introduces an adaptive batch size scheme for stochastic ADMM that dynamically balances variance reduction and computational cost, with proven convergence and extensions to variance-reduced methods, validated by numerical experiments.

Contribution

It proposes a practical adaptive batch size method for SADMM with convergence guarantees and extends it to improve existing variance-reduced algorithms.

Findings

Adaptive batch size improves efficiency in nonconvex nonsmooth optimization.

The method matches the best-known complexity bounds.

Numerical results confirm the effectiveness of the proposed approach.

Abstract

Stochastic alternating direction method of multipliers (SADMM) is a popular method for solving nonconvex nonsmooth optimization in various applications. However, it typically requires an empirical selection of the static batch size for gradient estimation, resulting in a challenging trade-off between variance reduction and computational cost. This paper proposes adaptive batch size SADMM, a practical method that dynamically adjusts the batch size based on accumulated differences along the optimization path. We develop a simple convergence analysis to handle the dependence of batch size adaptation that matches the best-known complexity with flexible parameter choices. We further extend this adaptive scheme to reduce the overall complexity of the popular variance-reduced methods, SVRG-ADMM and SPIDER-ADMM. Numerical results validate the effectiveness of our proposed methods.