Generalization Error Analysis for Attack-Free and Byzantine-Resilient Decentralized Learning with Data Heterogeneity

TL;DR

This paper provides a detailed analysis of the generalization errors in decentralized learning systems, considering attack-free and Byzantine-resilient scenarios with data heterogeneity, highlighting factors influencing model performance in real-world applications.

Contribution

It offers the first fine-grained generalization error analysis for heterogeneous data in decentralized learning, including Byzantine attack impacts, under mild assumptions.

Findings

Data heterogeneity significantly affects generalization error.

Byzantine attacks increase generalization error independently of sample size.

Model initialization and stochastic gradient noise influence generalization performance.

Abstract

Decentralized learning, which facilitates joint model training across geographically scattered agents, has gained significant attention in the field of signal and information processing in recent years. While the optimization errors of decentralized learning algorithms have been extensively studied, their generalization errors remain relatively under-explored. As the generalization errors reflect the scalability of trained models on unseen data and are crucial in determining the performance of trained models in real-world applications, understanding the generalization errors of decentralized learning is of paramount importance. In this paper, we present fine-grained generalization error analysis for both attack-free and Byzantine-resilient decentralized learning with heterogeneous data as well as under mild assumptions, in contrast to prior studies that consider homogeneous data and/or rely on a stringent bounded stochastic gradient assumption. Our results shed light on the impact of data heterogeneity, model initialization and stochastic gradient noise -- factors that have not been closely investigated before -- on the generalization error of decentralized learning. We also reveal that Byzantine attacks performed by malicious agents largely affect the generalization error, and their negative impact is inherently linked to the data heterogeneity while remaining independent on the sample size. Numerical experiments on both convex and non-convex tasks are conducted to validate our theoretical findings.