BICompFL: Stochastic Federated Learning with Bi-Directional Compression

TL;DR

BICompFL introduces a bi-directional compression method for stochastic federated learning, significantly reducing communication costs while maintaining high accuracy, supported by theoretical analysis and experimental validation.

Contribution

The paper proposes BICompFL, a novel approach to bi-directional compression in stochastic federated learning, addressing inherent challenges and improving communication efficiency.

Findings

Reduces communication cost by an order of magnitude.

Maintains state-of-the-art accuracy.

Provides theoretical analysis of communication interplay.

Abstract

We address the prominent communication bottleneck in federated learning (FL). We specifically consider stochastic FL, in which models or compressed model updates are specified by distributions rather than deterministic parameters. Stochastic FL offers a principled approach to compression, and has been shown to reduce the communication load under perfect downlink transmission from the federator to the clients. However, in practice, both the uplink and downlink communications are constrained. We show that bi-directional compression for stochastic FL has inherent challenges, which we address by introducing BICompFL. Our BICompFL is experimentally shown to reduce the communication cost by an order of magnitude compared to multiple benchmarks, while maintaining state-of-the-art accuracies. Theoretically, we study the communication cost of BICompFL through a new analysis of an importance-sampling based technique, which exposes the interplay between uplink and downlink communication costs.