Decentralized Personalized Federated Learning based on a Conditional Sparse-to-Sparser Scheme

TL;DR

This paper introduces DA-DPFL, a decentralized federated learning scheme that reduces training costs through a dynamic sparse-to-sparser model update process, improving accuracy and energy efficiency.

Contribution

It proposes a novel sparse-to-sparser training scheme with dynamic aggregation, enhancing efficiency and personalization in decentralized federated learning.

Findings

Outperforms baseline DFL in test accuracy

Achieves up to 5 times reduction in energy costs

Provides theoretical convergence analysis

Abstract

Decentralized Federated Learning (DFL) has become popular due to its robustness and avoidance of centralized coordination. In this paradigm, clients actively engage in training by exchanging models with their networked neighbors. However, DFL introduces increased costs in terms of training and communication. Existing methods focus on minimizing communication often overlooking training efficiency and data heterogeneity. To address this gap, we propose a novel \textit{sparse-to-sparser} training scheme: DA-DPFL. DA-DPFL initializes with a subset of model parameters, which progressively reduces during training via \textit{dynamic aggregation} and leads to substantial energy savings while retaining adequate information during critical learning periods. Our experiments showcase that DA-DPFL substantially outperforms DFL baselines in test accuracy, while achieving up to $5$ times reduction in energy costs. We provide a theoretical analysis of DA-DPFL's convergence by solidifying its applicability in decentralized and personalized learning. The code is available at:https://github.com/EricLoong/da-dpfl