FedHiP: Heterogeneity-Invariant Personalized Federated Learning Through Closed-Form Solutions

TL;DR

FedHiP introduces a heterogeneity-invariant personalized federated learning method that uses closed-form solutions and a foundation model to achieve robust personalization regardless of data non-IIDness.

Contribution

It proposes a novel gradient-free, closed-form solution approach for PFL that is invariant to data heterogeneity, addressing a key challenge in federated learning.

Findings

Outperforms state-of-the-art methods by 5.79%-20.97% in accuracy

Achieves heterogeneity invariance in personalized models

Utilizes foundation models for gradient-free feature extraction

Abstract

Lately, Personalized Federated Learning (PFL) has emerged as a prevalent paradigm to deliver personalized models by collaboratively training while simultaneously adapting to each client's local applications. Existing PFL methods typically face a significant challenge due to the ubiquitous data heterogeneity (i.e., non-IID data) across clients, which severely hinders convergence and degrades performance. We identify that the root issue lies in the long-standing reliance on gradient-based updates, which are inherently sensitive to non-IID data. To fundamentally address this issue and bridge the research gap, in this paper, we propose a Heterogeneity-invariant Personalized Federated learning scheme, named FedHiP, through analytical (i.e., closed-form) solutions to avoid gradient-based updates. Specifically, we exploit the trend of self-supervised pre-training, leveraging a foundation model as a frozen backbone for gradient-free feature extraction. Following the feature extractor, we further develop an analytic classifier for gradient-free training. To support both collective generalization and individual personalization, our FedHiP scheme incorporates three phases: analytic local training, analytic global aggregation, and analytic local personalization. The closed-form solutions of our FedHiP scheme enable its ideal property of heterogeneity invariance, meaning that each personalized model remains identical regardless of how non-IID the data are distributed across all other clients. Extensive experiments on benchmark datasets validate the superiority of our FedHiP scheme, outperforming the state-of-the-art baselines by at least 5.79%-20.97% in accuracy.