Dataset Distillation-based Hybrid Federated Learning on Non-IID Data

TL;DR

This paper introduces HFLDD, a hybrid federated learning framework using dataset distillation to mitigate non-IID data issues, especially label imbalance, resulting in improved accuracy and reduced communication costs.

Contribution

The paper proposes a novel hybrid federated learning framework that employs dataset distillation to handle non-IID data, particularly label skew, by clustering clients and training on distilled data.

Findings

HFLDD improves test accuracy on non-IID datasets.

HFLDD reduces communication overhead compared to baseline methods.

HFLDD effectively handles label imbalance in federated learning.

Abstract

In federated learning, the heterogeneity of client data has a great impact on the performance of model training. Many heterogeneity issues in this process are raised by non-independently and identically distributed (non-IID) data. To address the issue of label distribution skew, we propose a hybrid federated learning framework called HFLDD, which integrates dataset distillation to generate approximately independent and equally distributed (IID) data, thereby improving the performance of model training. In particular, we partition the clients into heterogeneous clusters, where the data labels among different clients within a cluster are unbalanced while the data labels among different clusters are balanced. The cluster heads collect distilled data from the corresponding cluster members, and conduct model training in collaboration with the server. This training process is like traditional federated learning on IID data, and hence effectively alleviates the impact of non-IID data on model training. We perform a comprehensive analysis of the convergence behavior, communication overhead, and computational complexity of the proposed HFLDD. Extensive experimental results based on multiple public datasets demonstrate that when data labels are severely imbalanced, the proposed HFLDD outperforms the baseline methods in terms of both test accuracy and communication cost.