Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchors

TL;DR

This paper introduces DeSA, a novel decentralized federated learning method using synthetic anchors to improve model generalization across heterogeneous clients by facilitating mutual knowledge transfer and domain adaptation.

Contribution

DeSA is the first approach to synthesize global anchors in decentralized FL, enhancing knowledge sharing and model generalization without a central server.

Findings

DeSA improves inter- and intra-domain accuracy across diverse data distributions.

Synthetic anchors facilitate effective mutual knowledge transfer among clients.

Theoretical and empirical results validate DeSA's effectiveness in heterogeneous settings.

Abstract

Conventional Federated Learning (FL) involves collaborative training of a global model while maintaining user data privacy. One of its branches, decentralized FL, is a serverless network that allows clients to own and optimize different local models separately, which results in saving management and communication resources. Despite the promising advancements in decentralized FL, it may reduce model generalizability due to lacking a global model. In this scenario, managing data and model heterogeneity among clients becomes a crucial problem, which poses a unique challenge that must be overcome: How can every client's local model learn generalizable representation in a decentralized manner? To address this challenge, we propose a novel Decentralized FL technique by introducing Synthetic Anchors, dubbed as DeSA. Based on the theory of domain adaptation and Knowledge Distillation (KD), we theoretically and empirically show that synthesizing global anchors based on raw data distribution facilitates mutual knowledge transfer. We further design two effective regularization terms for local training: 1) REG loss that regularizes the distribution of the client's latent embedding with the anchors and 2) KD loss that enables clients to learn from others. Through extensive experiments on diverse client data distributions, we showcase the effectiveness of DeSA in enhancing both inter- and intra-domain accuracy of each client.