Multi-level Personalized Federated Learning on Heterogeneous and Long-Tailed Data

TL;DR

This paper introduces MuPFL, a hierarchical personalized federated learning framework that effectively handles data heterogeneity and long-tailed distributions, improving accuracy and training efficiency.

Contribution

The paper proposes MuPFL, a novel multi-level personalized FL framework with modules to mitigate overfitting, refine models, and incorporate prior knowledge, advancing federated learning under challenging data conditions.

Findings

MuPFL outperforms state-of-the-art methods in accuracy by up to 7.39%.

Training speed is improved by up to 80%.

Effective under extreme non-i.i.d. and long-tail data distributions.

Abstract

Federated learning (FL) offers a privacy-centric distributed learning framework, enabling model training on individual clients and central aggregation without necessitating data exchange. Nonetheless, FL implementations often suffer from non-i.i.d. and long-tailed class distributions across mobile applications, e.g., autonomous vehicles, which leads models to overfitting as local training may converge to sub-optimal. In our study, we explore the impact of data heterogeneity on model bias and introduce an innovative personalized FL framework, Multi-level Personalized Federated Learning (MuPFL), which leverages the hierarchical architecture of FL to fully harness computational resources at various levels. This framework integrates three pivotal modules: Biased Activation Value Dropout (BAVD) to mitigate overfitting and accelerate training; Adaptive Cluster-based Model Update (ACMU) to refine local models ensuring coherent global aggregation; and Prior Knowledge-assisted Classifier Fine-tuning (PKCF) to bolster classification and personalize models in accord with skewed local data with shared knowledge. Extensive experiments on diverse real-world datasets for image classification and semantic segmentation validate that MuPFL consistently outperforms state-of-the-art baselines, even under extreme non-i.i.d. and long-tail conditions, which enhances accuracy by as much as 7.39% and accelerates training by up to 80% at most, marking significant advancements in both efficiency and effectiveness.