FedHPL: Efficient Heterogeneous Federated Learning with Prompt Tuning and Logit Distillation

TL;DR

FedHPL introduces a unified federated learning framework that uses prompt tuning and logit distillation to effectively handle heterogeneity in model architectures, data distributions, and resource constraints, achieving superior performance with less computation.

Contribution

It proposes a novel, parameter-efficient federated learning method combining prompt tuning and logit distillation to address heterogeneity challenges in a unified framework.

Findings

Outperforms state-of-the-art FL methods on benchmark datasets.

Reduces computation overhead and training rounds.

Improves model performance under resource constraints.

Abstract

Federated learning (FL) is a popular privacy-preserving paradigm that enables distributed clients to collaboratively train models with a central server while keeping raw data locally. In practice, distinct model architectures, varying data distributions, and limited resources across local clients inevitably cause model performance degradation and a slowdown in convergence speed. However, existing FL methods can only solve some of the above heterogeneous challenges and have obvious performance limitations. Notably, a unified framework has not yet been explored to overcome these challenges. Accordingly, we propose FedHPL, a parameter-efficient unified $\textbf{Fed}$erated learning framework for $\textbf{H}$eterogeneous settings based on $\textbf{P}$rompt tuning and $\textbf{L}$ogit distillation. Specifically, we employ a local prompt tuning scheme that leverages a few learnable visual prompts to efficiently fine-tune the frozen pre-trained foundation model for downstream tasks, thereby accelerating training and improving model performance under limited local resources and data heterogeneity. Moreover, we design a global logit distillation scheme to handle the model heterogeneity and guide the local training. In detail, we leverage logits to implicitly capture local knowledge and design a weighted knowledge aggregation mechanism to generate global client-specific logits. We provide a theoretical guarantee on the generalization error bound for FedHPL. The experiments on various benchmark datasets under diverse settings of models and data demonstrate that our framework outperforms state-of-the-art FL approaches, with less computation overhead and training rounds.