Efficient Federated Learning with Heterogeneous Data and Adaptive Dropout

TL;DR

This paper introduces FedDHAD, a federated learning framework that adaptively manages data heterogeneity and device variability, significantly improving accuracy and efficiency over existing methods.

Contribution

The paper proposes two novel methods, FedDH and FedAD, to dynamically handle data heterogeneity and device differences in federated learning, enhancing performance and efficiency.

Findings

Up to 6.7% higher accuracy compared to state-of-the-art

Up to 2.02 times faster convergence

15.0% reduction in computation cost

Abstract

Federated Learning (FL) is a promising distributed machine learning approach that enables collaborative training of a global model using multiple edge devices. The data distributed among the edge devices is highly heterogeneous. Thus, FL faces the challenge of data distribution and heterogeneity, where non-Independent and Identically Distributed (non-IID) data across edge devices may yield in significant accuracy drop. Furthermore, the limited computation and communication capabilities of edge devices increase the likelihood of stragglers, thus leading to slow model convergence. In this paper, we propose the FedDHAD FL framework, which comes with two novel methods: Dynamic Heterogeneous model aggregation (FedDH) and Adaptive Dropout (FedAD). FedDH dynamically adjusts the weights of each local model within the model aggregation process based on the non-IID degree of heterogeneous data to deal with the statistical data heterogeneity. FedAD performs neuron-adaptive operations in response to heterogeneous devices to improve accuracy while achieving superb efficiency. The combination of these two methods makes FedDHAD significantly outperform state-of-the-art solutions in terms of accuracy (up to 6.7% higher), efficiency (up to 2.02 times faster), and computation cost (up to 15.0% smaller).