On the Asymptotic Linear Convergence Speed of Anderson Acceleration Applied to ADMM

TL;DR

This paper analyzes and quantifies how Anderson acceleration can theoretically improve the asymptotic linear convergence speed of ADMM, providing spectral insights and optimal convergence factors for stationary AA applied to ADMM.

Contribution

It offers a theoretical framework for understanding the asymptotic convergence speedup of AA-accelerated ADMM through spectral analysis and optimal convergence factor computation.

Findings

Spectral properties of Jacobians explain convergence improvements.

Optimal stationary AA coefficients can be analytically or numerically computed.

Numerical results validate the spectral analysis and convergence estimates.

Abstract

Empirical results show that Anderson acceleration (AA) can be a powerful mechanism to improve the asymptotic linear convergence speed of the Alternating Direction Method of Multipliers (ADMM) when ADMM by itself converges linearly. However, theoretical results to quantify this improvement do not exist yet. In this paper we explain and quantify this improvement in linear asymptotic convergence speed for the special case of a stationary version of AA applied to ADMM. We do so by considering the spectral properties of the Jacobians of ADMM and the stationary version of AA evaluated at the fixed point, where the coefficients of the stationary AA method are computed such that its asymptotic linear convergence factor is optimal. The optimal linear convergence factors of this stationary AA-ADMM method are computed analytically or by optimization, based on previous work on optimal stationary AA acceleration. Using this spectral picture and those analytical results, our approach provides new insight into how and by how much the stationary AA method can improve the asymptotic linear convergence factor of ADMM. Numerical results also indicate that the optimal linear convergence factor of the stationary AA methods gives a useful estimate for the asymptotic linear convergence speed of the non-stationary AA method that is used in practice.