Data-Aware Gradient Compression for FL in Communication-Constrained Mobile Computing

TL;DR

This paper introduces data-aware gradient compression methods for federated learning in mobile environments, optimizing communication efficiency and convergence by assigning adaptive compression ratios based on data volume, especially in non-IID scenarios.

Contribution

It derives the convergence rate for non-uniform compression in distributed SGD and proposes DAGC-R and DAGC-A algorithms for adaptive gradient compression in communication-constrained FL.

Findings

DAGC-R speeds up training by up to 25.43%

DAGC-A improves robustness and reduces training time by up to 16.65%

Adaptive compression outperforms uniform approaches in non-IID data settings

Abstract

Federated Learning (FL) in mobile environments faces significant communication bottlenecks. Gradient compression has proven as an effective solution to this issue, offering substantial benefits in environments with limited bandwidth and metered data. Yet, it encounters severe performance drops in non-IID environments due to a one-size-fits-all compression approach, which does not account for the varying data volumes across workers. Assigning varying compression ratios to workers with distinct data distributions and volumes is therefore a promising solution. This work derives the convergence rate of distributed SGD with non-uniform compression, which reveals the intricate relationship between model convergence and the compression ratios applied to individual workers. Accordingly, we frame the relative compression ratio assignment as an $n$-variable chi-squared nonlinear optimization problem, constrained by a limited communication budget. We propose DAGC-R, which assigns conservative compression to workers handling larger data volumes. Recognizing the computational limitations of mobile devices, we propose the DAGC-A, which is computationally less demanding and enhances the robustness of compression in non-IID scenarios. Our experiments confirm that the DAGC-R and DAGC-A can speed up the training speed by up to $25.43\%$ and $16.65\%$ compared to the uniform compression respectively, when dealing with highly imbalanced data volume distribution and restricted communication.