Consensus Driven Learning

TL;DR

This paper introduces a decentralized, asynchronous learning method inspired by consensus algorithms, enabling distributed neural network training with limited communication and data privacy, effective on biased datasets and unreliable networks.

Contribution

It proposes a novel distributed learning approach that coordinates nodes asynchronously without central control, maintaining data privacy and robustness to communication failures.

Findings

Effective on MNIST, Fashion MNIST, CIFAR10 datasets.

Handles biased datasets and communication failures.

Reduces communication overhead in distributed training.

Abstract

As the complexity of our neural network models grow, so too do the data and computation requirements for successful training. One proposed solution to this problem is training on a distributed network of computational devices, thus distributing the computational and data storage loads. This strategy has already seen some adoption by the likes of Google and other companies. In this paper we propose a new method of distributed, decentralized learning that allows a network of computation nodes to coordinate their training using asynchronous updates over an unreliable network while only having access to a local dataset. This is achieved by taking inspiration from Distributed Averaging Consensus algorithms to coordinate the various nodes. Sharing the internal model instead of the training data allows the original raw data to remain with the computation node. The asynchronous nature and lack of centralized coordination allows this paradigm to function with limited communication requirements. We demonstrate our method on the MNIST, Fashion MNIST, and CIFAR10 datasets. We show that our coordination method allows models to be learned on highly biased datasets, and in the presence of intermittent communication failure.