Decentralized Deep Learning using Momentum-Accelerated Consensus

TL;DR

This paper introduces a decentralized deep learning algorithm that employs momentum-accelerated consensus over fixed communication networks, enabling efficient collaborative training without a central server, suitable for networked systems.

Contribution

The paper proposes a novel decentralized deep learning method using momentum-based consensus, applicable to non-centralized topologies, with theoretical analysis and empirical validation.

Findings

Effective in both convex and non-convex settings

Outperforms existing decentralized methods in experiments

Works well across various communication topologies

Abstract

We consider the problem of decentralized deep learning where multiple agents collaborate to learn from a distributed dataset. While there exist several decentralized deep learning approaches, the majority consider a central parameter-server topology for aggregating the model parameters from the agents. However, such a topology may be inapplicable in networked systems such as ad-hoc mobile networks, field robotics, and power network systems where direct communication with the central parameter server may be inefficient. In this context, we propose and analyze a novel decentralized deep learning algorithm where the agents interact over a fixed communication topology (without a central server). Our algorithm is based on the heavy-ball acceleration method used in gradient-based optimization. We propose a novel consensus protocol where each agent shares with its neighbors its model parameters as well as gradient-momentum values during the optimization process. We consider both strongly convex and non-convex objective functions and theoretically analyze our algorithm's performance. We present several empirical comparisons with competing decentralized learning methods to demonstrate the efficacy of our approach under different communication topologies.