ALiA: Adaptive Linearized ADMM

TL;DR

ALiA is an adaptive variant of ADMM that automatically adjusts stepsizes, improving convergence speed and performance without backtracking, applicable to convex differentiable problems.

Contribution

We introduce ALiA, an adaptive, function-linearized ADMM variant with a novel stepsize scheme that enhances convergence speed and practical performance.

Findings

ALiA converges for convex differentiable objectives.

ALiA outperforms standard FLiP ADMM in experiments.

ALiA matches or exceeds existing adaptive methods' performance.

Abstract

We propose ALiA, a novel adaptive variant of the alternating direction method of multipliers (ADMM). Specifically, ALiA is a variant of function-linearized proximal ADMM (FLiP ADMM), which generalizes the classical ADMM by leveraging the differentiable structure of the objective function, making it highly versatile. Notably, ALiA features an adaptive stepsize selection scheme that eliminates the need for backtracking linesearch. Motivated by recent advances in adaptive gradient and proximal methods, we establish point convergence of ALiA for convex and differentiable objectives. Furthermore, by introducing negligible computational overhead, we develop an alternative stepsize selection scheme for ALiA that improves the convergence speed both theoretically and empirically. Extensive numerical experiments on practical datasets confirm the accelerated performance of ALiA compared to standard FLiP ADMM. Additionally, we demonstrate that ALiA either outperforms or matches the practical performance of existing adaptive methods across problem classes where it is applicable.