FedSparQ: Adaptive Sparse Quantization with Error Feedback for Robust & Efficient Federated Learning

TL;DR

FedSparQ introduces an adaptive, error-feedback-based sparse quantization method for federated learning, significantly reducing communication costs while maintaining or improving model accuracy across diverse data distributions.

Contribution

It proposes a novel adaptive sparse quantization framework with error feedback that requires no manual tuning and is effective across various models and data distributions.

Findings

Reduces communication by 90% compared to FedAvg.

Improves model accuracy by up to 6%.

Enhances convergence robustness by 50%.

Abstract

Federated Learning (FL) enables collaborative model training across decentralized clients while preserving data privacy by keeping raw data local. However, FL suffers from significant communication overhead due to the frequent exchange of high-dimensional model updates over constrained networks. In this paper, we present FedSparQ, a lightweight compression framework that dynamically sparsifies the gradient of each client through an adaptive threshold, applies half-precision quantization to retained entries and integrates residuals from error feedback to prevent loss of information. FedSparQ requires no manual tuning of sparsity rates or quantization schedules, adapts seamlessly to both homogeneous and heterogeneous data distributions, and is agnostic to model architecture. Through extensive empirical evaluation on vision benchmarks under independent and identically distributed (IID) and non-IID data, we show that FedSparQ substantially reduces communication overhead (reducing by 90% of bytes sent compared to FedAvg) while preserving or improving model accuracy (improving by 6% compared to FedAvg non-compressed solution or to state-of-the-art compression models) and enhancing convergence robustness (by 50%, compared to the other baselines). Our approach provides a practical, easy-to-deploy solution for bandwidth-constrained federated deployments and lays the groundwork for future extensions in adaptive precision and privacy-preserving protocols.