General Optimal Step-size for ADMM-type Algorithms: Domain Parametrization and Optimal Rates

TL;DR

This paper solves a 49-year open problem by deriving a closed-form, optimal step-size for ADMM algorithms that minimizes worst-case fixed-point convergence rates, applicable to convex programs with arbitrary initialization.

Contribution

The paper provides the first explicit polynomial-based formula for the optimal step-size in ADMM, improving convergence analysis and practical implementation.

Findings

Closed-form solution for optimal ADMM step-size derived

Adaptive step-size selection performs nearly as well as theoretical optimum

Significant advancement in understanding ADMM convergence rates

Abstract

In this work, we solve a 49-year open problem, the general optimal step-size for ADMM-type algorithms. For a convex program: $\text{min.} \,\, f({x}) + g({z})$, $\text{s.t.}\, {A}{x} - {B}{z} = {c} $, given an arbitrary fixed-point initialization $ {\zeta}^0 $, an optimal step-size choice is given by a root of the following polynomial: \begin{equation*} \rho^4\Vert {A}{x}^\star\Vert^2 - \rho^3\langle {A}{x}^\star, {\zeta}^0\rangle + \rho\langle {\lambda}^\star,{\zeta}^0\rangle - \Vert{\lambda}^\star\Vert^2 = 0, \end{equation*} with $ \rho \neq 0 $ a domain step-size, which relates to the classical positive one via $ \gamma = \rho^2$. We denote by $ \cdot^\star $ the optimal solution, by $ {\lambda} $ the Lagrange multiplier associated with the equality constraint (dual variable). The above polynomial always admits a closed-form solution. The optimality is in the sense that a worst-case fixed-point convergence rate is minimized, which is a balance of the normalized primal and dual iterates convergence speed (reciprocally related). In cases where either the primal or dual solution is trivial (a zero vector), improvement can be made by accelerating the non-trivial sequence only. For practical use, adaptively replace the above optimal solutions with the current iterates, which are known at every iteration. Numerically, it exhibits almost identical performance as the theoretical one (after a few iterations), similar to the underlying best fixed step-size (found by exhaustive grid search).