Cyclic Coordinate Dual Averaging with Extrapolation

TL;DR

This paper introduces a novel cyclic coordinate dual averaging method with extrapolation for variational inequality problems, achieving optimal convergence bounds and improved efficiency through variance reduction.

Contribution

It presents a new block coordinate method applicable to variational inequalities, with convergence guarantees matching full gradient methods and a variance-reduced variant for finite-sum problems.

Findings

Convergence bounds match those of full gradient methods.

Gradient Lipschitz constant is at most √m times the Euclidean constant.

Variance reduction improves per-iteration cost and convergence in certain regimes.

Abstract

Cyclic block coordinate methods are a fundamental class of optimization methods widely used in practice and implemented as part of standard software packages for statistical learning. Nevertheless, their convergence is generally not well understood and so far their good practical performance has not been explained by existing convergence analyses. In this work, we introduce a new block coordinate method that applies to the general class of variational inequality (VI) problems with monotone operators. This class includes composite convex optimization problems and convex-concave min-max optimization problems as special cases and has not been addressed by the existing work. The resulting convergence bounds match the optimal convergence bounds of full gradient methods, but are provided in terms of a novel gradient Lipschitz condition w.r.t.~a Mahalanobis norm. For $m$ coordinate blocks, the resulting gradient Lipschitz constant in our bounds is never larger than a factor $\sqrt{m}$ compared to the traditional Euclidean Lipschitz constant, while it is possible for it to be much smaller. Further, for the case when the operator in the VI has finite-sum structure, we propose a variance reduced variant of our method which further decreases the per-iteration cost and has better convergence rates in certain regimes. To obtain these results, we use a gradient extrapolation strategy that allows us to view a cyclic collection of block coordinate-wise gradients as one implicit gradient.