Decentralized Stochastic Variance Reduced Extragradient Method

TL;DR

This paper introduces a decentralized stochastic variance reduced extragradient method for convex-concave minimax problems, achieving optimal complexity bounds and demonstrating superior empirical performance over existing methods.

Contribution

It proposes a novel decentralized algorithm with the best known SFO complexity for convex-concave minimax problems, improving efficiency and convergence guarantees.

Findings

Achieves optimal SFO complexity for strongly-convex-strongly-concave problems.

Achieves near-optimal SFO complexity for general convex-concave problems.

Numerical experiments show superior performance compared to baseline methods.

Abstract

This paper studies decentralized convex-concave minimax optimization problems of the form $\min_x\max_y f(x,y) \triangleq\frac{1}{m}\sum_{i=1}^m f_i(x,y)$, where $m$ is the number of agents and each local function can be written as $f_i(x,y)=\frac{1}{n}\sum_{j=1}^n f_{i,j}(x,y)$. We propose a novel decentralized optimization algorithm, called multi-consensus stochastic variance reduced extragradient, which achieves the best known stochastic first-order oracle (SFO) complexity for this problem. Specifically, each agent requires $\mathcal O((n+\kappa\sqrt{n})\log(1/\varepsilon))$ SFO calls for strongly-convex-strongly-concave problem and $\mathcal O((n+\sqrt{n}L/\varepsilon)\log(1/\varepsilon))$ SFO call for general convex-concave problem to achieve $\varepsilon$-accurate solution in expectation, where $\kappa$ is the condition number and $L$ is the smoothness parameter. The numerical experiments show the proposed method performs better than baselines.