Stragglers Can Contribute More: Uncertainty-Aware Distillation for Asynchronous Federated Learning

TL;DR

FedEcho introduces an uncertainty-aware distillation method in asynchronous federated learning, effectively balancing the contributions of straggler and faster clients to improve model performance amid delays and data heterogeneity.

Contribution

The paper presents FedEcho, a novel framework that dynamically assesses and adjusts client prediction reliability to enhance asynchronous federated learning.

Findings

FedEcho outperforms existing baselines in diverse experiments.

It effectively mitigates outdated update impacts and client bias.

The method maintains robust performance without accessing private data.

Abstract

Asynchronous federated learning (FL) has recently gained attention for its enhanced efficiency and scalability, enabling local clients to send model updates to the server at their own pace without waiting for slower participants. However, such a design encounters significant challenges, such as the risk of outdated updates from straggler clients degrading the overall model performance and the potential bias introduced by faster clients dominating the learning process, especially under heterogeneous data distributions. Existing methods typically address only one of these issues, creating a conflict where mitigating the impact of outdated updates can exacerbate the bias created by faster clients, and vice versa. To address these challenges, we propose FedEcho, a novel framework that incorporates uncertainty-aware distillation to enhance the asynchronous FL performances under large asynchronous delays and data heterogeneity. Specifically, uncertainty-aware distillation enables the server to assess the reliability of predictions made by straggler clients, dynamically adjusting the influence of these predictions based on their estimated uncertainty. By prioritizing more certain predictions while still leveraging the diverse information from all clients, FedEcho effectively mitigates the negative impacts of outdated updates and data heterogeneity. Through extensive experiments, we demonstrate that FedEcho consistently outperforms existing asynchronous federated learning baselines, achieving robust performance without requiring access to private client data.