Resource-Aware Aggregation and Sparsification in Heterogeneous Ensemble Federated Learning

TL;DR

This paper introduces SHEFL, a resource-aware ensemble federated learning framework that dynamically allocates models and adjusts sparsification to improve accuracy and efficiency across heterogeneous clients.

Contribution

The paper proposes a novel ensemble-based FL framework that adapts to client resource variability and reduces training bias through dynamic sparsification.

Findings

Significantly improves accuracy over existing methods.

Enhances training stability across diverse client resources.

Effectively reduces computational burden for deep ensembles.

Abstract

Federated learning (FL) enables distributed training with private client data, but its convergence is hindered by system heterogeneity under realistic communication scenarios. Most FL schemes addressing system heterogeneity utilize global pruning or ensemble distillation, yet often overlook typical constraints required for communication efficiency. Meanwhile, deep ensembles can aggregate predictions from individually trained models to improve performance, but current ensemble-based FL methods fall short in fully capturing diversity of model predictions. In this work, we propose \textbf{SHEFL}, a global ensemble-based FL framework suited for clients with diverse computational capacities. We allocate different numbers of global models to clients based on their available resources. We introduce a novel aggregation scheme that mitigates the training bias between clients and dynamically adjusts the sparsification ratio across clients to reduce the computational burden of training deep ensembles. Extensive experiments demonstrate that our method effectively addresses computational heterogeneity, significantly improving accuracy and stability compared to existing approaches.