Fine-Tuning Impairs the Balancedness of Foundation Models in Long-tailed Personalized Federated Learning

TL;DR

This paper introduces FedPuReL, a novel federated learning method that preserves class balance in foundation models and improves personalization in long-tailed, heterogeneous data scenarios.

Contribution

It proposes gradient purification and residual learning techniques to maintain balanced global models and unbiased personalization in long-tailed federated learning.

Findings

FedPuReL outperforms existing methods in long-tailed scenarios.

Gradient purification maintains class balance in global models.

Residual learning enhances personalized model performance.

Abstract

Personalized federated learning (PFL) with foundation models has emerged as a promising paradigm enabling clients to adapt to heterogeneous data distributions. However, real-world scenarios often face the co-occurrence of non-IID data and long-tailed class distributions, presenting unique challenges that remain underexplored in PFL. In this paper, we investigate this long-tailed personalized federated learning and observe that current methods suffer from two limitations: (i) fine-tuning degrades performance below zero-shot baselines due to the erosion of inherent class balance in foundation models; (ii) conventional personalization techniques further transfer this bias to local models through parameter or feature-level fusion. To address these challenges, we propose Federated Learning via Gradient Purification and Residual Learning (FedPuReL), which preserves balanced knowledge in the global model while enabling unbiased personalization. Specifically, we purify local gradients using zero-shot predictions to maintain a class-balanced global model, and model personalization as residual correction atop the frozen global model. Extensive experiments demonstrate that FedPuReL consistently outperforms state-of-the-art methods, achieving superior performance on both global and personalized models across diverse long-tailed scenarios. The code is available at https://github.com/shihaohou/FedPuReL.