HEART-PFL: Stable Personalized Federated Learning under Heterogeneity with Hierarchical Directional Alignment and Adversarial Knowledge Transfer

TL;DR

HEART-PFL introduces a dual-sided framework for personalized federated learning that enhances client-specific model accuracy and global stability through hierarchical alignment and adversarial knowledge transfer, demonstrating state-of-the-art results on multiple datasets.

Contribution

The paper proposes HEART-PFL, a novel framework combining hierarchical directional alignment and adversarial knowledge transfer to improve personalization and robustness in federated learning.

Findings

Achieves state-of-the-art accuracy on CIFAR-100, Flowers-102, and Caltech-101.

Maintains robustness against out-of-domain proxy data.

Provides ablation evidence of complementary effects of HDA and AKT.

Abstract

Personalized Federated Learning (PFL) aims to deliver effective client-specific models under heterogeneous distributions, yet existing methods suffer from shallow prototype alignment and brittle server-side distillation. We propose HEART-PFL, a dual-sided framework that (i) performs depth-aware Hierarchical Directional Alignment (HDA) using cosine similarity in the early stage and MSE matching in the deep stage to preserve client specificity, and (ii) stabilizes global updates through Adversarial Knowledge Transfer (AKT) with symmetric KL distillation on clean and adversarial proxy data. Using lightweight adapters with only 1.46M trainable parameters, HEART-PFL achieves state-of-the-art personalized accuracy on CIFAR-100, Flowers-102, and Caltech-101 (63.42%, 84.23%, and 95.67%, respectively) under Dirichlet non-IID partitions, and remains robust to out-of-domain proxy data. Ablation studies further confirm that HDA and AKT provide complementary gains in alignment, robustness, and optimization stability, offering insights into how the two components mutually reinforce effective personalization. Overall, these results demonstrate that HEART-PFL simultaneously enhances personalization and global stability, highlighting its potential as a strong and scalable solution for PFL(code available at https://github.com/danny0628/HEART-PFL).