Unveiling High-Probability Generalization in Decentralized SGD

TL;DR

This paper develops a high-probability generalization theory for decentralized SGD, achieving optimal bounds that bridge the gap with traditional SGD and cover convex, non-convex, and distributed settings.

Contribution

It introduces a novel high-probability analysis for D-SGD using pointwise uniform stability, improving existing bounds across various learning scenarios.

Findings

Achieves optimal high-probability generalization bounds of O(1/√(mn)) log(1/δ).

Provides bounds for convex, strongly convex, and non-convex cases.

Analyzes communication overhead effects on generalization in distributed models.

Abstract

Decentralized stochastic gradient descent (D-SGD) is an efficient method for large-scale distributed learning. Existing generalization studies mainly address expected results, achieving rates limited to $\mathcal{O}\left(\frac{1}{\delta \sqrt{mn}}\right)$, where $\delta$ is the confidence parameter, $m$ the number of workers, and $n$ the sample size. When $m=1$, D-SGD reduces to traditional SGD, whose optimal high-probability generalization bound is $\mathcal{O}\left(\frac{1}{\sqrt{n}}\log (1/\delta)\right)$. This discrepancy reveals a gap between high-probability guarantees for SGD and those for D-SGD. To close this, we develop a high-probability learning theory for D-SGD, aiming for the optimal $\mathcal{O}\left(\frac{1}{\sqrt{mn}}\log (1/\delta)\right)$ rate. We refine bounds for D-SGD using pointwise uniform stability in distributed learning-a weaker notion than uniform stability-and analyze them across convex, strongly convex, and non-convex settings. We also provide high-probability results for gradient-based measures in non-convex cases where only local minima exist, and derive optimization error and excess risk bounds. Finally, accounting for communication overhead, we analyze generalization bounds for local models within time-varying frameworks.