Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularization

TL;DR

This paper introduces FedLoRU, a low-rank update framework for federated learning that reduces communication costs, provides theoretical insights into rank properties, and maintains competitive performance in heterogeneous environments.

Contribution

It presents the first theoretical analysis of rank properties in FL and proposes a low-rank update method that offers implicit regularization and communication efficiency.

Findings

FedLoRU achieves comparable accuracy to full-rank algorithms.

Theoretical analysis shows higher-rank structures in client losses.

FedLoRU is robust to heterogeneity and large client populations.

Abstract

Federated Learning (FL) faces significant challenges related to communication efficiency and performance reduction when scaling to many clients. To address these issues, we explore the potential of using low-rank updates and provide the first theoretical study of rank properties in FL. Our theoretical analysis shows that a client's loss exhibits a higher-rank structure (i.e., gradients span higher-rank subspaces of the Hessian) compared to the server's loss, and that low-rank approximations of the clients' gradients have greater similarity. Based on this insight, we hypothesize that constraining client-side optimization to a low-rank subspace could provide an implicit regularization effect while reducing communication costs. Consequently, we propose FedLoRU, a general low-rank update framework for FL. Our framework enforces low-rank client-side updates and accumulates these updates to form a higher-rank model. We are able to establish convergence of the algorithm; the convergence rate matches FedAvg. Additionally, variants of FedLoRU can adapt to environments with statistical and model heterogeneity by employing multiple or hierarchical low-rank updates. Experimental results demonstrate that FedLoRU performs comparably to full-rank algorithms and exhibits robustness to heterogeneous and large numbers of clients.