Communication-Efficient Federated Learning via Predictive Coding

TL;DR

This paper introduces a predictive coding compression scheme for federated learning that significantly reduces communication overhead while maintaining or improving learning performance, especially suited for wireless mobile devices.

Contribution

The paper proposes a novel predictive coding based compression method with shared predictors and residual transmission, optimizing rate-distortion and entropy coding for federated learning.

Findings

Communication cost reduced by up to 99%

Achieves better learning performance than baseline methods

Effective in wireless mobile device scenarios

Abstract

Federated learning can enable remote workers to collaboratively train a shared machine learning model while allowing training data to be kept locally. In the use case of wireless mobile devices, the communication overhead is a critical bottleneck due to limited power and bandwidth. Prior work has utilized various data compression tools such as quantization and sparsification to reduce the overhead. In this paper, we propose a predictive coding based compression scheme for federated learning. The scheme has shared prediction functions among all devices and allows each worker to transmit a compressed residual vector derived from the reference. In each communication round, we select the predictor and quantizer based on the rate-distortion cost, and further reduce the redundancy with entropy coding. Extensive simulations reveal that the communication cost can be reduced up to 99% with even better learning performance when compared with other baseline methods.