Exploiting Correlations in Federated Learning: Opportunities and Practical Limitations

TL;DR

This paper classifies and analyzes correlation-based compression techniques in federated learning, proposing adaptive methods that improve data efficiency based on correlation measurements.

Contribution

It introduces a unified correlation-based framework for gradient and model compression in federated learning and proposes adaptive compression schemes.

Findings

Correlation degrees vary with task complexity and model architecture.

Adaptive compression methods outperform non-adaptive approaches.

Correlation metrics guide effective compression mode selection.

Abstract

The communication bottleneck in federated learning (FL) has spurred extensive research into techniques to reduce the volume of data exchanged between client devices and the central parameter server. In this paper, we systematically classify gradient and model compression schemes into three categories based on the type of correlations they exploit: structural, temporal, and spatial. We examine the sources of such correlations, propose quantitative metrics for measuring their magnitude, and reinterpret existing compression methods through this unified correlation-based framework. Our experimental studies demonstrate that the degrees of structural, temporal, and spatial correlations vary significantly depending on task complexity, model architecture, and algorithmic configurations. These findings suggest that algorithm designers should carefully evaluate correlation assumptions under specific deployment scenarios rather than assuming that they are always present. Motivated by these findings, we propose two adaptive compression designs that actively switch between different compression modes based on the measured correlation strength, and we evaluate their performance gains relative to conventional non-adaptive approaches. In summary, our unified taxonomy provides a clean and principled foundation for developing more effective and application-specific compression techniques for FL systems.