A Survey on Heterogeneous Federated Learning

TL;DR

This survey comprehensively reviews heterogeneous federated learning, addressing data, statistical, system, and model heterogeneity, and discusses methodologies, applications, challenges, and future directions in the field.

Contribution

It provides a detailed taxonomy of heterogeneous FL settings, summarizes transfer learning approaches, and highlights future research opportunities.

Findings

Taxonomy of heterogeneous FL settings based on problem and learning objectives.

Survey of transfer learning methods for heterogeneity in FL.

Identification of challenges and promising future research directions.

Abstract

Federated learning (FL) has been proposed to protect data privacy and virtually assemble the isolated data silos by cooperatively training models among organizations without breaching privacy and security. However, FL faces heterogeneity from various aspects, including data space, statistical, and system heterogeneity. For example, collaborative organizations without conflict of interest often come from different areas and have heterogeneous data from different feature spaces. Participants may also want to train heterogeneous personalized local models due to non-IID and imbalanced data distribution and various resource-constrained devices. Therefore, heterogeneous FL is proposed to address the problem of heterogeneity in FL. In this survey, we comprehensively investigate the domain of heterogeneous FL in terms of data space, statistical, system, and model heterogeneity. We first give an overview of FL, including its definition and categorization. Then, We propose a precise taxonomy of heterogeneous FL settings for each type of heterogeneity according to the problem setting and learning objective. We also investigate the transfer learning methodologies to tackle the heterogeneity in FL. We further present the applications of heterogeneous FL. Finally, we highlight the challenges and opportunities and envision promising future research directions toward new framework design and trustworthy approaches.