Edge Bias in Federated Learning and its Solution by Buffered Knowledge Distillation

TL;DR

This paper identifies the 'edge bias' problem in federated learning with knowledge distillation and proposes 'buffered distillation' to mitigate it, improving model performance and addressing straggler issues.

Contribution

The paper introduces 'buffered distillation', a novel method to reduce edge bias and improve federated learning effectiveness, especially in scenarios with delayed edge devices.

Findings

Buffered distillation effectively reduces edge bias.

The method mitigates straggler problems caused by delayed edges.

Experimental results show improved model accuracy and robustness.

Abstract

Federated learning (FL), which utilizes communication between the server (core) and local devices (edges) to indirectly learn from more data, is an emerging field in deep learning research. Recently, Knowledge Distillation-based FL methods with notable performance and high applicability have been suggested. In this paper, we choose knowledge distillation-based FL method as our baseline and tackle a challenging problem that ensues from using these methods. Especially, we focus on the problem incurred in the server model that tries to mimic different datasets, each of which is unique to an individual edge device. We dub the problem 'edge bias', which occurs when multiple teacher models trained on different datasets are used individually to distill knowledge. We introduce this nuisance that occurs in certain scenarios of FL, and to alleviate it, we propose a simple yet effective distillation scheme named 'buffered distillation'. In addition, we also experimentally show that this scheme is effective in mitigating the straggler problem caused by delayed edges.