Peer-to-Peer Learning Dynamics of Wide Neural Networks

TL;DR

This paper characterizes the training dynamics of wide neural networks in peer-to-peer distributed learning environments, providing insights into how these networks evolve during training without a central server.

Contribution

It offers an explicit analysis of the learning dynamics of wide neural networks trained with distributed gradient descent, combining NTK theory and consensus methods.

Findings

Accurately predicts parameter and error dynamics of wide neural networks

Provides theoretical insights into training behavior in decentralized settings

Validates predictions with empirical classification results

Abstract

Peer-to-peer learning is an increasingly popular framework that enables beyond-5G distributed edge devices to collaboratively train deep neural networks in a privacy-preserving manner without the aid of a central server. Neural network training algorithms for emerging environments, e.g., smart cities, have many design considerations that are difficult to tune in deployment settings -- such as neural network architectures and hyperparameters. This presents a critical need for characterizing the training dynamics of distributed optimization algorithms used to train highly nonconvex neural networks in peer-to-peer learning environments. In this work, we provide an explicit characterization of the learning dynamics of wide neural networks trained using popular distributed gradient descent (DGD) algorithms. Our results leverage both recent advancements in neural tangent kernel (NTK) theory and extensive previous work on distributed learning and consensus. We validate our analytical results by accurately predicting the parameter and error dynamics of wide neural networks trained for classification tasks.