Peer-to-Peer Learning + Consensus with Non-IID Data

TL;DR

This paper introduces P2PL with Affinity, a novel method that reduces test performance oscillations in peer-to-peer deep learning on non-IID data, enhancing stability without extra communication overhead.

Contribution

The paper proposes P2PL with Affinity, a new approach that mitigates model drift and performance oscillations in decentralized learning with non-IID data, without increasing communication costs.

Findings

P2PL with Affinity dampens test performance oscillations.

Model drift causes significant test performance fluctuations.

The method improves stability in non-IID distributed learning.

Abstract

Peer-to-peer deep learning algorithms are enabling distributed edge devices to collaboratively train deep neural networks without exchanging raw training data or relying on a central server. Peer-to-Peer Learning (P2PL) and other algorithms based on Distributed Local-Update Stochastic/mini-batch Gradient Descent (local DSGD) rely on interleaving epochs of training with distributed consensus steps. This process leads to model parameter drift/divergence amongst participating devices in both IID and non-IID settings. We observe that model drift results in significant oscillations in test performance evaluated after local training and consensus phases. We then identify factors that amplify performance oscillations and demonstrate that our novel approach, P2PL with Affinity, dampens test performance oscillations in non-IID settings without incurring any additional communication cost.