Gradient Projection onto Historical Descent Directions for Communication-Efficient Federated Learning

TL;DR

This paper introduces two algorithms, ProjFL and ProjFL+EF, that improve communication efficiency in federated learning by projecting gradients onto historical descent directions, with proven convergence and comparable accuracy to existing methods.

Contribution

The paper proposes novel gradient projection algorithms for federated learning that reduce communication costs while maintaining convergence guarantees across various convexity settings.

Findings

Achieve similar accuracy to baseline methods

Significantly reduce communication overhead

Convergence guarantees under multiple convexity conditions

Abstract

Federated Learning (FL) enables decentralized model training across multiple clients while optionally preserving data privacy. However, communication efficiency remains a critical bottleneck, particularly for large-scale models. In this work, we introduce two complementary algorithms: ProjFL, designed for unbiased compressors, and ProjFL+EF, tailored for biased compressors through an Error Feedback mechanism. Both methods rely on projecting local gradients onto a shared client-server subspace spanned by historical descent directions, enabling efficient information exchange with minimal communication overhead. We establish convergence guarantees for both algorithms under strongly convex, convex, and non-convex settings. Empirical evaluations on standard FL classification benchmarks with deep neural networks show that ProjFL and ProjFL+EF achieve accuracy comparable to existing baselines while substantially reducing communication costs.