Stability and Generalization of the Decentralized Stochastic Gradient Descent Ascent Algorithm

TL;DR

This paper studies the stability and generalization properties of the decentralized stochastic gradient descent ascent algorithm, showing it maintains good generalization similar to centralized methods under various network topologies.

Contribution

It provides the first theoretical analysis of the primal-dual generalization bounds for decentralized minimax algorithms, extending stability theory to decentralized settings.

Findings

Decentralized structure does not impair stability or generalization of D-SGDA.

Generalization bounds depend on network topology beyond sample size and learning rate.

Numerical experiments confirm theoretical predictions.

Abstract

The growing size of available data has attracted increasing interest in solving minimax problems in a decentralized manner for various machine learning tasks. Previous theoretical research has primarily focused on the convergence rate and communication complexity of decentralized minimax algorithms, with little attention given to their generalization. In this paper, we investigate the primal-dual generalization bound of the decentralized stochastic gradient descent ascent (D-SGDA) algorithm using the approach of algorithmic stability under both convex-concave and nonconvex-nonconcave settings. Our theory refines the algorithmic stability in a decentralized manner and demonstrates that the decentralized structure does not destroy the stability and generalization of D-SGDA, implying that it can generalize as well as the vanilla SGDA in certain situations. Our results analyze the impact of different topologies on the generalization bound of the D-SGDA algorithm beyond trivial factors such as sample sizes, learning rates, and iterations. We also evaluate the optimization error and balance it with the generalization gap to obtain the optimal population risk of D-SGDA in the convex-concave setting. Additionally, we perform several numerical experiments which validate our theoretical findings.