Decentralized Federated Learning by Partial Message Exchange

TL;DR

This paper introduces PaME, a novel decentralized federated learning algorithm that reduces communication costs and preserves privacy by exchanging only sparse message coordinates, while ensuring convergence and handling data heterogeneity.

Contribution

The paper proposes PaME, a new DFL algorithm that exchanges partial messages, achieves linear convergence under mild assumptions, and effectively addresses data heterogeneity.

Findings

PaME significantly reduces communication costs.

It maintains high privacy levels without sacrificing accuracy.

Numerical experiments show superior performance over existing algorithms.

Abstract

Decentralized federated learning (DFL) has emerged as a transformative server-free paradigm that enables collaborative learning over large-scale heterogeneous networks. However, it continues to face fundamental challenges, including data heterogeneity, restrictive assumptions for theoretical analysis, and degraded convergence when standard communication- or privacyenhancing techniques are applied. To overcome these drawbacks, this paper develops a novel algorithm, PaME (DFL by Partial Message Exchange). The central principle is to allow only randomly selected sparse coordinates to be exchanged between two neighbor nodes. Consequently, PaME achieves substantial reductions in communication costs while still preserving a high level of privacy, without sacrificing accuracy. Moreover, grounded in rigorous analysis, the algorithm is shown to converge at a linear rate under the gradient to be locally Lipschitz continuous and the communication matrix to be doubly stochastic. These two mild assumptions not only dispense with many restrictive conditions commonly imposed by existing DFL methods but also enables PaME to effectively address data heterogeneity. Furthermore, comprehensive numerical experiments demonstrate its superior performance compared with several representative decentralized learning algorithms.