Enhancing Heterogeneous Federated Learning with Knowledge Extraction and Multi-Model Fusion

TL;DR

This paper introduces a resource-aware federated learning approach that effectively manages model heterogeneity and reduces communication costs by aggregating and distilling local knowledge into a robust global model, enhancing performance on edge devices.

Contribution

It proposes a novel federated learning method combining knowledge extraction and multi-model fusion to handle heterogeneity and reduce communication overhead.

Findings

Reduces communication cost of ResNet-32 by up to 50%.

Reduces communication cost of VGG-11 by up to 10 times.

Improves performance on heterogeneous data and models.

Abstract

Concerned with user data privacy, this paper presents a new federated learning (FL) method that trains machine learning models on edge devices without accessing sensitive data. Traditional FL methods, although privacy-protective, fail to manage model heterogeneity and incur high communication costs due to their reliance on aggregation methods. To address this limitation, we propose a resource-aware FL method that aggregates local knowledge from edge models and distills it into robust global knowledge through knowledge distillation. This method allows efficient multi-model knowledge fusion and the deployment of resource-aware models while preserving model heterogeneity. Our method improves communication cost and performance in heterogeneous data and models compared to existing FL algorithms. Notably, it reduces the communication cost of ResNet-32 by up to 50\% and VGG-11 by up to 10$\times$ while delivering superior performance.