FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projection

TL;DR

FedSLoP introduces a low-rank gradient projection method for federated learning, significantly reducing communication and memory costs while maintaining competitive accuracy, supported by theoretical convergence guarantees.

Contribution

The paper proposes FedSLoP, a novel federated optimization algorithm using stochastic low-rank gradient projections, with proven convergence and improved efficiency over existing methods.

Findings

FedSLoP reduces communication volume compared to FedAvg.

FedSLoP achieves similar or better accuracy in heterogeneous data settings.

Theoretical analysis guarantees convergence to a stationary point at rate O(1/√NT).

Abstract

Federated learning enables a population of clients to collaboratively train machine learning models without exchanging their raw data, but standard algorithms such as FedAvg suffer from slow convergence and high communication and memory costs in heterogeneous, resource-constrained environments. We introduce FedSLoP, a federated optimization algorithm that combines stochastic low-rank subspace projections of gradients, thereby reducing the dimension of communicated and stored updates while preserving optimization progress. On the theoretical side, we develop a detailed nonconvex convergence analysis under standard smoothness and bounded-variance assumptions, showing that FedSLoP is guaranteed to converge to a first-order stationary point at a rate of $O(1/\sqrt{NT})$. On the empirical side, we conduct extensive experiments on federated MNIST classification with heterogeneous data partitions, showing that FedSLoP substantially reduces communication volume and client-side memory while achieving competitive or better accuracy compared with FedAvg and representative sparse or low-rank baselines. Together, our results demonstrate that random subspace momentum methods such as FedSLoP provide a principled and effective approach to communication- and memory-efficient federated learning. Codes are available at: https://github.com/pkumelon/FedSLoP.git.