Federated Distillation on Edge Devices: Efficient Client-Side Filtering for Non-IID Data

TL;DR

This paper introduces EdgeFD, a resource-efficient federated distillation method that improves client-side data filtering using KMeans clustering, enhancing accuracy and scalability on edge devices with non-IID data.

Contribution

EdgeFD simplifies density ratio estimation and eliminates server filtering, enabling effective knowledge sharing on resource-constrained edge devices with diverse data distributions.

Findings

EdgeFD achieves near-IID accuracy under challenging non-IID conditions.

The KMeans-based estimator reduces computational overhead significantly.

EdgeFD outperforms state-of-the-art methods in diverse scenarios.

Abstract

Federated distillation has emerged as a promising collaborative machine learning approach, offering enhanced privacy protection and reduced communication compared to traditional federated learning by exchanging model outputs (soft logits) rather than full model parameters. However, existing methods employ complex selective knowledge-sharing strategies that require clients to identify in-distribution proxy data through computationally expensive statistical density ratio estimators. Additionally, server-side filtering of ambiguous knowledge introduces latency to the process. To address these challenges, we propose a robust, resource-efficient EdgeFD method that reduces the complexity of the client-side density ratio estimation and removes the need for server-side filtering. EdgeFD introduces an efficient KMeans-based density ratio estimator for effectively filtering both in-distribution and out-of-distribution proxy data on clients, significantly improving the quality of knowledge sharing. We evaluate EdgeFD across diverse practical scenarios, including strong non-IID, weak non-IID, and IID data distributions on clients, without requiring a pre-trained teacher model on the server for knowledge distillation. Experimental results demonstrate that EdgeFD outperforms state-of-the-art methods, consistently achieving accuracy levels close to IID scenarios even under heterogeneous and challenging conditions. The significantly reduced computational overhead of the KMeans-based estimator is suitable for deployment on resource-constrained edge devices, thereby enhancing the scalability and real-world applicability of federated distillation. The code is available online for reproducibility.