$\gamma$-FedHT: Stepsize-Aware Hard-Threshold Gradient Compression in Federated Learning

TL;DR

This paper introduces $oldsymbol{ ext{ extgamma}- ext{FedHT}}$, a stepsize-aware gradient compression method with Error-Feedback for federated learning, improving accuracy and convergence under communication constraints and non-IID data.

Contribution

It proposes a novel stepsize-aware hard-threshold compressor with Error-Feedback, providing convergence guarantees and better accuracy in federated learning with non-IID datasets.

Findings

Achieves up to 7.42% accuracy improvement over Top-$k$ compression.

Maintains convergence rates comparable to FedAvg under convex and non-convex settings.

Effectively balances communication efficiency and model accuracy in non-IID federated learning.

Abstract

Gradient compression can effectively alleviate communication bottlenecks in Federated Learning (FL). Contemporary state-of-the-art sparse compressors, such as Top-$k$, exhibit high computational complexity, up to $\mathcal{O}(d\log_2{k})$, where $d$ is the number of model parameters. The hard-threshold compressor, which simply transmits elements with absolute values higher than a fixed threshold, is thus proposed to reduce the complexity to $\mathcal{O}(d)$. However, the hard-threshold compression causes accuracy degradation in FL, where the datasets are non-IID and the stepsize $\gamma$ is decreasing for model convergence. The decaying stepsize reduces the updates and causes the compression ratio of the hard-threshold compression to drop rapidly to an aggressive ratio. At or below this ratio, the model accuracy has been observed to degrade severely. To address this, we propose $\gamma$-FedHT, a stepsize-aware low-cost compressor with Error-Feedback to guarantee convergence. Given that the traditional theoretical framework of FL does not consider Error-Feedback, we introduce the fundamental conversation of Error-Feedback. We prove that $\gamma$-FedHT has the convergence rate of $\mathcal{O}(\frac{1}{T})$ ($T$ representing total training iterations) under $\mu$-strongly convex cases and $\mathcal{O}(\frac{1}{\sqrt{T}})$ under non-convex cases, \textit{same as FedAVG}. Extensive experiments demonstrate that $\gamma$-FedHT improves accuracy by up to $7.42\%$ over Top-$k$ under equal communication traffic on various non-IID image datasets.