FedeKD: Energy-Based Gating for Robust Federated Knowledge Distillation under Heterogeneous Settings

TL;DR

FedeKD introduces an energy-based gating mechanism in federated knowledge distillation to improve robustness by estimating sample-wise trust without relying on public data, effectively reducing negative transfer in heterogeneous environments.

Contribution

The paper proposes a novel reliability-aware FKD framework with an energy-based gating mechanism for sample-wise trust estimation, enhancing robustness without needing public datasets.

Findings

FedeKD significantly reduces negative transfer in heterogeneous federated settings.

The energy-based gating mechanism improves sample-wise knowledge transfer.

FedeKD maintains strong predictive performance across six real-world datasets.

Abstract

Federated learning (FL) operates in heterogeneous environments, where variations in data distributions and asymmetric model design often result in negative transfer. While federated knowledge distillation (FKD) avoids direct model parameter sharing, existing methods typically rely on public datasets or assume that transferred knowledge is uniformly reliable, which limits their robustness in practice. This paper presents FedeKD, a reliability-aware FKD framework that makes sample-wise trust estimation an explicit component of knowledge transfer, without relying on additional public data. Each client maintains a high-capacity private model for local learning and a lightweight shared proxy model for cross-client knowledge exchange. During training, proxy models are aggregated on the server to form a global proxy, which is then used to guide updates of the private models. At the core of FedeKD is an energy-based gating mechanism that converts task-specific private-proxy disagreement into sample-wise trust weights for backward distillation. This mechanism enables sample-wise weighting of knowledge transfer, where the proxy model contributes more to reliable samples while down-weighting unreliable ones. Extensive experiments on six real-world datasets demonstrate that FedeKD significantly reduces negative transfer under heterogeneous settings while maintaining strong predictive performance.