Sparse-Push: Communication- & Energy-Efficient Decentralized Distributed Learning over Directed & Time-Varying Graphs with non-IID Datasets

TL;DR

Sparse-Push introduces a communication-efficient decentralized training algorithm for directed, time-varying graphs, significantly reducing communication costs with minimal performance loss, and addresses non-IID data challenges with an improved variant.

Contribution

The paper presents Sparse-Push, a novel decentralized training algorithm supporting directed, time-varying graphs with high communication efficiency and robustness to non-IID data distributions.

Findings

Achieves 466x reduction in communication with 1% performance degradation.

Demonstrates effectiveness on ResNet-20 and VGG11 models on CIFAR-10.

Proposes Skew-Compensated Sparse Push to recover performance in non-IID scenarios.

Abstract

Current deep learning (DL) systems rely on a centralized computing paradigm which limits the amount of available training data, increases system latency, and adds privacy and security constraints. On-device learning, enabled by decentralized and distributed training of DL models over peer-to-peer wirelessly connected edge devices, not only alleviate the above limitations but also enable next-gen applications that need DL models to continuously interact and learn from their environment. However, this necessitates the development of novel training algorithms that train DL models over time-varying and directed peer-to-peer graph structures while minimizing the amount of communication between the devices and also being resilient to non-IID data distributions. In this work we propose, Sparse-Push, a communication efficient decentralized distributed training algorithm that supports training over peer-to-peer, directed, and time-varying graph topologies. The proposed algorithm enables 466x reduction in communication with only 1% degradation in performance when training various DL models such as ResNet-20 and VGG11 over the CIFAR-10 dataset. Further, we demonstrate how communication compression can lead to significant performance degradation in-case of non-IID datasets, and propose Skew-Compensated Sparse Push algorithm that recovers this performance drop while maintaining similar levels of communication compression.