Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learning

TL;DR

This paper compares centralized and decentralized federated learning, revealing theoretical and empirical differences in their generalization and stability, and providing guidance on their efficiency and performance factors.

Contribution

It offers a comprehensive theoretical analysis of stability and generalization gaps between CFL and DFL, including conditions for optimal performance and topology requirements.

Findings

CFL generally generalizes better than DFL on smooth non-convex objectives.

Partial participation in CFL yields the best performance.

Topology in DFL is crucial to prevent performance collapse with scale.

Abstract

As two mainstream frameworks in federated learning (FL), both centralized and decentralized approaches have shown great application value in practical scenarios. However, existing studies do not provide sufficient evidence and clear guidance for analysis of which performs better in the FL community. Although decentralized methods have been proven to approach the comparable convergence of centralized with less communication, their test performance always falls short of expectations in empirical studies. To comprehensively and fairly compare their efficiency gaps in FL, in this paper, we explore their stability and generalization efficiency. Specifically, we prove that on the general smooth non-convex objectives, 1) centralized FL (CFL) always generalizes better than decentralized FL (DFL); 2) CFL achieves the best performance via adopting partial participation instead of full participation; and, 3) there is a necessary requirement for the topology in DFL to avoid performance collapse as the training scale increases. We also conduct extensive experiments on several common setups in FL to validate that our theoretical analysis is consistent with experimental phenomena and contextually valid in several general and practical scenarios.