Optimal Rate Adaption in Federated Learning with Compressed Communications

TL;DR

This paper systematically analyzes the tradeoff between compression and accuracy in federated learning, proposing an adaptive compression rate framework that improves efficiency while maintaining high accuracy.

Contribution

It introduces the first systematic examination of compression-accuracy tradeoff in FL and proposes an adaptive rate framework based on convergence analysis.

Findings

Adaptive compression improves network efficiency.

The framework maintains high accuracy with reduced communication.

Experimental results validate effectiveness on MNIST and CIFAR-10.

Abstract

Federated Learning (FL) incurs high communication overhead, which can be greatly alleviated by compression for model updates. Yet the tradeoff between compression and model accuracy in the networked environment remains unclear and, for simplicity, most implementations adopt a fixed compression rate only. In this paper, we for the first time systematically examine this tradeoff, identifying the influence of the compression error on the final model accuracy with respect to the learning rate. Specifically, we factor the compression error of each global iteration into the convergence rate analysis under both strongly convex and non-convex loss functions. We then present an adaptation framework to maximize the final model accuracy by strategically adjusting the compression rate in each iteration. We have discussed the key implementation issues of our framework in practical networks with representative compression algorithms. Experiments over the popular MNIST and CIFAR-10 datasets confirm that our solution effectively reduces network traffic yet maintains high model accuracy in FL.