SADDLe: Sharpness-Aware Decentralized Deep Learning with Heterogeneous Data

TL;DR

SADDLe introduces sharpness-aware decentralized deep learning algorithms that improve model generalization and robustness in heterogeneous data environments while reducing communication costs.

Contribution

It proposes novel sharpness-aware algorithms for decentralized learning that handle data heterogeneity and communication compression effectively.

Findings

Achieves 1-20% higher test accuracy than existing methods.

Maintains only 1% accuracy drop under 4x communication compression.

Enhances robustness and generalization in decentralized training with heterogeneous data.

Abstract

Decentralized training enables learning with distributed datasets generated at different locations without relying on a central server. In realistic scenarios, the data distribution across these sparsely connected learning agents can be significantly heterogeneous, leading to local model over-fitting and poor global model generalization. Another challenge is the high communication cost of training models in such a peer-to-peer fashion without any central coordination. In this paper, we jointly tackle these two-fold practical challenges by proposing SADDLe, a set of sharpness-aware decentralized deep learning algorithms. SADDLe leverages Sharpness-Aware Minimization (SAM) to seek a flatter loss landscape during training, resulting in better model generalization as well as enhanced robustness to communication compression. We present two versions of our approach and conduct extensive experiments to show that SADDLe leads to 1-20% improvement in test accuracy compared to other existing techniques. Additionally, our proposed approach is robust to communication compression, with an average drop of only 1% in the presence of up to 4x compression.