Near-Optimal Algorithms for Making the Gradient Small in Stochastic Minimax Optimization

TL;DR

This paper introduces RAIN, a new stochastic algorithm that achieves near-optimal convergence rates for stochastic minimax problems, including convex-concave and nonconvex-nonconcave cases.

Contribution

The paper proposes RAIN, a novel stochastic algorithm that extends EAG methods to stochastic settings and achieves near-optimal complexity for various minimax problems.

Findings

RAIN achieves near-optimal SFO complexity in convex-concave cases.

RAIN extends to nonconvex-nonconcave minimax problems with similar efficiency.

The method outperforms existing stochastic algorithms in convergence speed.

Abstract

We study the problem of finding a near-stationary point for smooth minimax optimization. The recently proposed extra anchored gradient (EAG) methods achieve the optimal convergence rate for the convex-concave minimax problem in the deterministic setting. However, the direct extension of EAG to stochastic optimization is not efficient. In this paper, we design a novel stochastic algorithm called Recursive Anchored IteratioN (RAIN). We show that the RAIN achieves near-optimal stochastic first-order oracle (SFO) complexity for stochastic minimax optimization in both convex-concave and strongly-convex-strongly-concave cases. In addition, we extend the idea of RAIN to solve structured nonconvex-nonconcave minimax problem and it also achieves near-optimal SFO complexity.