Decentralized federated learning of deep neural networks on non-iid data

TL;DR

This paper introduces PENS, a decentralized federated learning approach where clients with similar data distributions collaborate to improve personalized deep learning models in non-iid settings, outperforming existing methods.

Contribution

The paper proposes a novel neighbor selection method for decentralized federated learning that enhances model accuracy on heterogeneous data distributions.

Findings

PENS achieves higher accuracy than baseline methods.

Clients with similar data distributions effectively collaborate.

Decentralized learning improves personalization in non-iid scenarios.

Abstract

We tackle the non-convex problem of learning a personalized deep learning model in a decentralized setting. More specifically, we study decentralized federated learning, a peer-to-peer setting where data is distributed among many clients and where there is no central server to orchestrate the training. In real world scenarios, the data distributions are often heterogeneous between clients. Therefore, in this work we study the problem of how to efficiently learn a model in a peer-to-peer system with non-iid client data. We propose a method named Performance-Based Neighbor Selection (PENS) where clients with similar data distributions detect each other and cooperate by evaluating their training losses on each other's data to learn a model suitable for the local data distribution. Our experiments on benchmark datasets show that our proposed method is able to achieve higher accuracies as compared to strong baselines.