Communication Compression for Distributed Learning without Control Variates

TL;DR

This paper introduces CAFe, a new distributed learning framework that enables highly compressible updates without control variates, improving convergence and privacy in federated learning scenarios.

Contribution

CAFe is a novel framework that leverages past aggregated updates to allow aggressive compression without control variates, addressing privacy and statefulness issues.

Findings

CAFe outperforms existing schemes in experiments.

CAFe guarantees convergence without control variates.

Theoretical analysis shows CAFe's superiority in non-convex settings.

Abstract

Distributed learning algorithms, such as the ones employed in Federated Learning (FL), require communication compression to reduce the cost of client uploads. The compression methods used in practice are often biased, making error feedback necessary both to achieve convergence under aggressive compression and to provide theoretical convergence guarantees. However, error feedback requires client-specific control variates, creating two key challenges: it violates privacy-preserving principles and demands stateful clients. In this paper, we propose Compressed Aggregate Feedback (CAFe), a novel distributed learning framework that allows highly compressible client updates by exploiting past aggregated updates, and does not require control variates. We consider Distributed Gradient Descent (DGD) as a representative algorithm and analytically prove CAFe's superiority to Distributed Compressed Gradient Descent (DCGD) with biased compression in the non-convex regime with bounded gradient dissimilarity. Experimental results confirm that CAFe outperforms existing distributed learning compression schemes.