E-3SFC: Communication-Efficient Federated Learning with Double-way Features Synthesizing

TL;DR

This paper introduces E-3SFC, a novel gradient compression method for federated learning that significantly reduces communication costs while maintaining high accuracy, by using synthetic features and double-way compression.

Contribution

The paper proposes a new gradient compression algorithm, 3SFC, utilizing model-based synthetic features and error feedback, extended to E-3SFC with double-way compression and dynamic scheduling.

Findings

Outperforms state-of-the-art methods by up to 13.4% in accuracy.

Reduces communication costs by 111.6 times.

Achieves linear and sub-linear convergence rates under various conditions.

Abstract

The exponential growth in model sizes has significantly increased the communication burden in Federated Learning (FL). Existing methods to alleviate this burden by transmitting compressed gradients often face high compression errors, which slow down the model's convergence. To simultaneously achieve high compression effectiveness and lower compression errors, we study the gradient compression problem from a novel perspective. Specifically, we propose a systematical algorithm termed Extended Single-Step Synthetic Features Compressing (E-3SFC), which consists of three sub-components, i.e., the Single-Step Synthetic Features Compressor (3SFC), a double-way compression algorithm, and a communication budget scheduler. First, we regard the process of gradient computation of a model as decompressing gradients from corresponding inputs, while the inverse process is considered as compressing the gradients. Based on this, we introduce a novel gradient compression method termed 3SFC, which utilizes the model itself as a decompressor, leveraging training priors such as model weights and objective functions. 3SFC compresses raw gradients into tiny synthetic features in a single-step simulation, incorporating error feedback to minimize overall compression errors. To further reduce communication overhead, 3SFC is extended to E-3SFC, allowing double-way compression and dynamic communication budget scheduling. Our theoretical analysis under both strongly convex and non-convex conditions demonstrates that 3SFC achieves linear and sub-linear convergence rates with aggregation noise. Extensive experiments across six datasets and six models reveal that 3SFC outperforms state-of-the-art methods by up to 13.4% while reducing communication costs by 111.6 times. These findings suggest that 3SFC can significantly enhance communication efficiency in FL without compromising model performance.