HierarchyFL: Heterogeneous Federated Learning via Hierarchical Self-Distillation

TL;DR

HierarchyFL is a scalable federated learning framework that employs hierarchical self-distillation and an ensemble library to enhance knowledge sharing and model accuracy across diverse heterogeneous AIoT devices using limited public data.

Contribution

It introduces a novel hierarchical self-distillation approach with an ensemble library to effectively address model heterogeneity in large-scale federated AIoT systems.

Findings

Significantly improves model accuracy across heterogeneous AIoT devices.

Enhances knowledge sharing efficiency in federated learning.

Demonstrates scalability on various datasets.

Abstract

Federated learning (FL) has been recognized as a privacy-preserving distributed machine learning paradigm that enables knowledge sharing among various heterogeneous artificial intelligence (AIoT) devices through centralized global model aggregation. FL suffers from model inaccuracy and slow convergence due to the model heterogeneity of the AIoT devices involved. Although various existing methods try to solve the bottleneck of the model heterogeneity problem, most of them improve the accuracy of heterogeneous models in a coarse-grained manner, which makes it still a great challenge to deploy large-scale AIoT devices. To alleviate the negative impact of this problem and take full advantage of the diversity of each heterogeneous model, we propose an efficient framework named HierarchyFL, which uses a small amount of public data for efficient and scalable knowledge across a variety of differently structured models. By using self-distillation and our proposed ensemble library, each hierarchical model can intelligently learn from each other on cloud servers. Experimental results on various well-known datasets show that HierarchyFL can not only maximize the knowledge sharing among various heterogeneous models in large-scale AIoT systems, but also greatly improve the model performance of each involved heterogeneous AIoT device.