Decentralized SGD and Average-direction SAM are Asymptotically Equivalent

TL;DR

This paper demonstrates that decentralized stochastic gradient descent (D-SGD) asymptotically behaves like an average-direction Sharpness-aware Minimization (SAM) algorithm, revealing benefits for generalization and posterior estimation in decentralized learning.

Contribution

It proves the asymptotic equivalence between D-SGD and average-direction SAM, providing new insights into decentralization benefits and regularization effects.

Findings

D-SGD implicitly minimizes an average-direction SAM loss.

Decentralization offers a free uncertainty evaluation mechanism.

Sharpness regularization in D-SGD does not diminish with larger batch sizes.

Abstract

Decentralized stochastic gradient descent (D-SGD) allows collaborative learning on massive devices simultaneously without the control of a central server. However, existing theories claim that decentralization invariably undermines generalization. In this paper, we challenge the conventional belief and present a completely new perspective for understanding decentralized learning. We prove that D-SGD implicitly minimizes the loss function of an average-direction Sharpness-aware minimization (SAM) algorithm under general non-convex non-$\beta$-smooth settings. This surprising asymptotic equivalence reveals an intrinsic regularization-optimization trade-off and three advantages of decentralization: (1) there exists a free uncertainty evaluation mechanism in D-SGD to improve posterior estimation; (2) D-SGD exhibits a gradient smoothing effect; and (3) the sharpness regularization effect of D-SGD does not decrease as total batch size increases, which justifies the potential generalization benefit of D-SGD over centralized SGD (C-SGD) in large-batch scenarios. The code is available at https://github.com/Raiden-Zhu/ICML-2023-DSGD-and-SAM.