Decentralized Stochastic Gradient Descent Ascent for Finite-Sum Minimax Problems

TL;DR

This paper introduces a novel decentralized stochastic gradient descent ascent method for finite-sum minimax problems, achieving new theoretical complexities and demonstrating effectiveness in AUC maximization.

Contribution

It develops the first decentralized stochastic method with variance reduction for finite-sum minimax problems, providing optimal theoretical complexity bounds.

Findings

Achieves $O(rac{ ext{sqrt}(n) ext{κ}^3}{(1- ext{λ})^2 ext{ε}^2})$ sample complexity.

Achieves $O(rac{ ext{κ}^3}{(1- ext{λ})^2 ext{ε}^2})$ communication complexity.

Experimental results confirm the method's effectiveness in AUC maximization.

Abstract

Minimax optimization problems have attracted significant attention in recent years due to their widespread application in numerous machine learning models. To solve the minimax problem, a wide variety of stochastic optimization methods have been proposed. However, most of them ignore the distributed setting where the training data is distributed on multiple workers. In this paper, we developed a novel decentralized stochastic gradient descent ascent method for the finite-sum minimax problem. In particular, by employing the variance-reduced gradient, our method can achieve $O(\frac{\sqrt{n}\kappa^3}{(1-\lambda)^2\epsilon^2})$ sample complexity and $O(\frac{\kappa^3}{(1-\lambda)^2\epsilon^2})$ communication complexity for the nonconvex-strongly-concave minimax problem. As far as we know, our work is the first one to achieve such theoretical complexities for this kind of minimax problem. At last, we apply our method to AUC maximization, and the experimental results confirm the effectiveness of our method.