Finite-Time Consensus Learning for Decentralized Optimization with Nonlinear Gossiping

TL;DR

This paper introduces a nonlinear gossiping framework for decentralized learning that achieves finite-time consensus, improving synchronization and convergence rates in distributed optimization, especially for deep neural networks.

Contribution

The paper proposes a novel decentralized learning method based on nonlinear gossiping with finite-time consensus, enhancing synchronization and convergence in distributed settings.

Findings

NGO achieves faster convergence than traditional methods.

Communication delays and randomness are effectively handled.

Empirical results show improved performance on neural network training.

Abstract

Distributed learning has become an integral tool for scaling up machine learning and addressing the growing need for data privacy. Although more robust to the network topology, decentralized learning schemes have not gained the same level of popularity as their centralized counterparts for being less competitive performance-wise. In this work, we attribute this issue to the lack of synchronization among decentralized learning workers, showing both empirically and theoretically that the convergence rate is tied to the synchronization level among the workers. Such motivated, we present a novel decentralized learning framework based on nonlinear gossiping (NGO), that enjoys an appealing finite-time consensus property to achieve better synchronization. We provide a careful analysis of its convergence and discuss its merits for modern distributed optimization applications, such as deep neural networks. Our analysis on how communication delay and randomized chats affect learning further enables the derivation of practical variants that accommodate asynchronous and randomized communications. To validate the effectiveness of our proposal, we benchmark NGO against competing solutions through an extensive set of tests, with encouraging results reported.