Federated Knowledge Distillation

TL;DR

This paper provides a comprehensive analysis of federated distillation (FD), highlighting its communication efficiency and versatility across tasks, through theoretical insights and practical implementations in distributed learning.

Contribution

It offers a novel asymptotic analysis of FD algorithms using neural tangent kernel theory and demonstrates FD's effectiveness in various distributed learning scenarios.

Findings

FD reduces communication costs compared to traditional federated learning.

FD achieves comparable accuracy with significantly less data exchange.

FD is applicable to classification, wireless channels, and reinforcement learning environments.

Abstract

Distributed learning frameworks often rely on exchanging model parameters across workers, instead of revealing their raw data. A prime example is federated learning that exchanges the gradients or weights of each neural network model. Under limited communication resources, however, such a method becomes extremely costly particularly for modern deep neural networks having a huge number of model parameters. In this regard, federated distillation (FD) is a compelling distributed learning solution that only exchanges the model outputs whose dimensions are commonly much smaller than the model sizes (e.g., 10 labels in the MNIST dataset). The goal of this chapter is to provide a deep understanding of FD while demonstrating its communication efficiency and applicability to a variety of tasks. To this end, towards demystifying the operational principle of FD, the first part of this chapter provides a novel asymptotic analysis for two foundational algorithms of FD, namely knowledge distillation (KD) and co-distillation (CD), by exploiting the theory of neural tangent kernel (NTK). Next, the second part elaborates on a baseline implementation of FD for a classification task, and illustrates its performance in terms of accuracy and communication efficiency compared to FL. Lastly, to demonstrate the applicability of FD to various distributed learning tasks and environments, the third part presents two selected applications, namely FD over asymmetric uplink-and-downlink wireless channels and FD for reinforcement learning.