Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data

TL;DR

This paper introduces federated distillation and augmentation techniques for on-device machine learning, significantly reducing communication costs and improving model performance on non-IID data compared to traditional federated learning.

Contribution

It proposes federated distillation for low-overhead model training and federated augmentation to mitigate non-IID data issues, advancing privacy-preserving on-device ML methods.

Findings

26x reduction in communication overhead

Achieves 95-98% test accuracy

Effective handling of non-IID data

Abstract

On-device machine learning (ML) enables the training process to exploit a massive amount of user-generated private data samples. To enjoy this benefit, inter-device communication overhead should be minimized. With this end, we propose federated distillation (FD), a distributed model training algorithm whose communication payload size is much smaller than a benchmark scheme, federated learning (FL), particularly when the model size is large. Moreover, user-generated data samples are likely to become non-IID across devices, which commonly degrades the performance compared to the case with an IID dataset. To cope with this, we propose federated augmentation (FAug), where each device collectively trains a generative model, and thereby augments its local data towards yielding an IID dataset. Empirical studies demonstrate that FD with FAug yields around 26x less communication overhead while achieving 95-98% test accuracy compared to FL.