Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theory

TL;DR

This paper investigates the communication-accuracy trade-off in federated learning, proposing a rate-distortion theory-based method that reduces communication costs while maintaining model performance, outperforming existing compression techniques.

Contribution

It introduces a novel rate-distortion framework for optimizing communication efficiency in federated learning, demonstrating near-optimal performance and empirical consistency across various settings.

Findings

Proposed method outperforms existing compression techniques on benchmark tasks.

Rate-distortion frontier remains consistent despite non-i.i.d. data.

Distortion effectively proxies model accuracy for optimization.

Abstract

A significant bottleneck in federated learning (FL) is the network communication cost of sending model updates from client devices to the central server. We present a comprehensive empirical study of the statistics of model updates in FL, as well as the role and benefits of various compression techniques. Motivated by these observations, we propose a novel method to reduce the average communication cost, which is near-optimal in many use cases, and outperforms Top-K, DRIVE, 3LC and QSGD on Stack Overflow next-word prediction, a realistic and challenging FL benchmark. This is achieved by examining the problem using rate-distortion theory, and proposing distortion as a reliable proxy for model accuracy. Distortion can be more effectively used for optimizing the trade-off between model performance and communication cost across clients. We demonstrate empirically that in spite of the non-i.i.d. nature of federated learning, the rate-distortion frontier is consistent across datasets, optimizers, clients and training rounds.