A Closer Look at Personalized Fine-Tuning in Heterogeneous Federated Learning

TL;DR

This paper introduces a new personalized fine-tuning method for federated learning called LP-FT, which improves the balance between global generalization and local personalization by mitigating feature distortion.

Contribution

It adapts Linear Probing and full Fine-Tuning to federated learning, providing a principled approach and theoretical insights into reducing feature distortion and enhancing personalization.

Findings

LP-FT outperforms existing PFT methods across multiple datasets.

Federated feature distortion destabilizes local models and is mitigated by LP-FT.

Theoretical analysis explains how phased parameter updates improve stability.

Abstract

Federated Learning (FL) enables decentralized, privacy-preserving model training but struggles to balance global generalization and local personalization due to non-identical data distributions across clients. Personalized Fine-Tuning (PFT), a popular post-hoc solution, fine-tunes the final global model locally but often overfits to skewed client distributions or fails under domain shifts. We propose adapting Linear Probing followed by full Fine-Tuning (LP-FT), a principled centralized strategy for alleviating feature distortion (Kumar et al., 2022), to the FL setting. Through systematic evaluation across seven datasets and six PFT variants, we demonstrate LP-FT's superiority in balancing personalization and generalization. Our analysis uncovers federated feature distortion, a phenomenon where local fine-tuning destabilizes globally learned features, and theoretically characterizes how LP-FT mitigates this via phased parameter updates. We further establish conditions (e.g., partial feature overlap, covariate-concept shift) under which LP-FT outperforms standard fine-tuning, offering actionable guidelines for deploying robust personalization in FL.