Adaptive Latent-Space Constraints in Personalized Federated Learning

TL;DR

This paper explores adaptive latent-space constraints using MMD measures in personalized federated learning, significantly enhancing model performance across heterogeneous datasets and demonstrating broad applicability to various pFL techniques.

Contribution

It introduces theoretically supported adaptive MMD constraints into pFL, improving personalization and performance across diverse tasks and datasets.

Findings

Adaptive MMD measures improve pFL performance

Constraints tailored to heterogeneity enhance model accuracy

Method applicable to multiple pFL techniques

Abstract

Federated learning (FL) is an effective and widely used approach to training deep learning models on decentralized datasets held by distinct clients. FL also strengthens both security and privacy protections for training data. Common challenges associated with statistical heterogeneity between distributed datasets have spurred significant interest in personalized FL (pFL) methods, where models combine aspects of global learning with local modeling specific to each client's unique characteristics. This work investigates the efficacy of theoretically supported, adaptive MMD measures in pFL, primarily focusing on the Ditto framework, a state-of-the-art technique for distributed data heterogeneity. The use of such measures significantly improves model performance across a variety of tasks, especially those with pronounced feature heterogeneity. Additional experiments demonstrate that such measures are directly applicable to other pFL techniques and yield similar improvements across a number of datasets. Finally, the results motivate the use of constraints tailored to the various kinds of heterogeneity expected in FL systems.