Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning

TL;DR

This paper investigates how different training strategies for client models in heterogeneous federated learning affect global model performance, proposing a new combined approach to improve personalization and robustness.

Contribution

It provides insights into the impact of client model training methods on global model aggregation and introduces a novel KD and Learning without Forgetting combination for better personalization.

Findings

Global models can effectively leverage Knowledge Distillation with heterogeneous data.

The proposed combined approach improves personalized model performance.

Heterogeneous FL can match the performance of homogeneous FL like FedAvg in realistic scenarios.

Abstract

The heterogeneity of hardware and data is a well-known and studied problem in the community of Federated Learning (FL) as running under heterogeneous settings. Recently, custom-size client models trained with Knowledge Distillation (KD) has emerged as a viable strategy for tackling the heterogeneity challenge. However, previous efforts in this direction are aimed at client model tuning rather than their impact onto the knowledge aggregation of the global model. Despite performance of global models being the primary objective of FL systems, under heterogeneous settings client models have received more attention. Here, we provide more insights into how the chosen approach for training custom client models has an impact on the global model, which is essential for any FL application. We show the global model can fully leverage the strength of KD with heterogeneous data. Driven by empirical observations, we further propose a new approach that combines KD and Learning without Forgetting (LwoF) to produce improved personalised models. We bring heterogeneous FL on pair with the mighty FedAvg of homogeneous FL, in realistic deployment scenarios with dropping clients.