Addressing Data Heterogeneity in Decentralized Learning via Topological Pre-processing

TL;DR

This paper introduces a topological pre-processing method for decentralized learning that constructs proxy-based heterogeneous graphs to improve convergence, scalability, and robustness while preserving data privacy.

Contribution

It proposes a novel peer clumping strategy for efficient graph construction, demonstrating improved convergence and scalability in decentralized learning with data heterogeneity.

Findings

Locally heterogeneous graphs outperform homogeneous ones in convergence.

The proposed pre-processing overhead remains small in large graphs.

The approach is robust against network partitions.

Abstract

Recently, local peer topology has been shown to influence the overall convergence of decentralized learning (DL) graphs in the presence of data heterogeneity. In this paper, we demonstrate the advantages of constructing a proxy-based locally heterogeneous DL topology to enhance convergence and maintain data privacy. In particular, we propose a novel peer clumping strategy to efficiently cluster peers before arranging them in a final training graph. By showing how locally heterogeneous graphs outperform locally homogeneous graphs of similar size and from the same global data distribution, we present a strong case for topological pre-processing. Moreover, we demonstrate the scalability of our approach by showing how the proposed topological pre-processing overhead remains small in large graphs while the performance gains get even more pronounced. Furthermore, we show the robustness of our approach in the presence of network partitions.