Personalized Federated Learning under Model Dissimilarity Constraints

TL;DR

This paper introduces KARULA, a personalized federated learning method that constrains model dissimilarities based on data distribution differences, effectively handling complex client heterogeneity.

Contribution

It proposes a novel regularized strategy with a specialized algorithm for personalized federated learning under model dissimilarity constraints.

Findings

KARULA effectively captures complex client relationships.

The algorithm converges with rate O(1/K) for non-convex losses.

Experimental results show improved personalization on real datasets.

Abstract

One of the defining challenges in federated learning is that of statistical heterogeneity among clients. We address this problem with KARULA, a regularized strategy for personalized federated learning, which constrains the pairwise model dissimilarities between clients based on the difference in their distributions, as measured by a surrogate for the 1-Wasserstein distance adapted for the federated setting. This allows the strategy to adapt to highly complex interrelations between clients, that e.g., clustered approaches fail to capture. We propose an inexact projected stochastic gradient algorithm to solve the constrained problem that the strategy defines, and show theoretically that it converges with smooth, possibly non-convex losses to a neighborhood of a stationary point with rate O(1/K). We demonstrate the effectiveness of KARULA on synthetic and real federated data sets.