Communication-Efficient Federated Learning by Exploiting Spatio-Temporal Correlations of Gradients

TL;DR

This paper introduces GradESTC, a novel gradient compression method for federated learning that leverages spatial and temporal correlations in gradients to significantly reduce communication overhead without sacrificing accuracy.

Contribution

GradESTC is the first method to exploit both spatial and temporal gradient correlations for communication-efficient federated learning.

Findings

Reduces uplink communication by 39.79% on average.

Maintains comparable convergence speed and final accuracy to uncompressed FedAvg.

Effectively leverages spatio-temporal gradient structures.

Abstract

Communication overhead is a critical challenge in federated learning, particularly in bandwidth-constrained networks. Although many methods have been proposed to reduce communication overhead, most focus solely on compressing individual gradients, overlooking the temporal correlations among them. Prior studies have shown that gradients exhibit spatial correlations, typically reflected in low-rank structures. Through empirical analysis, we further observe a strong temporal correlation between client gradients across adjacent rounds. Based on these observations, we propose GradESTC, a compression technique that exploits both spatial and temporal gradient correlations. GradESTC exploits spatial correlations to decompose each full gradient into a compact set of basis vectors and corresponding combination coefficients. By exploiting temporal correlations, only a small portion of the basis vectors need to be dynamically updated in each round. GradESTC significantly reduces communication overhead by transmitting lightweight combination coefficients and a limited number of updated basis vectors instead of the full gradients. Extensive experiments show that, upon reaching a target accuracy level near convergence, GradESTC reduces uplink communication by an average of 39.79% compared to the strongest baseline, while maintaining comparable convergence speed and final accuracy to uncompressed FedAvg. By effectively leveraging spatio-temporal gradient structures, GradESTC offers a practical and scalable solution for communication-efficient federated learning.