Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees

TL;DR

This paper introduces FeDLRT, a federated learning method that uses dynamical low-rank training to significantly reduce client compute and communication costs while maintaining accuracy, with proven convergence guarantees.

Contribution

The paper presents a novel federated low-rank training scheme with convergence guarantees, improving efficiency in client computation and communication.

Findings

Reduces client compute and communication costs by up to tenfold.

Maintains high global accuracy across computer vision benchmarks.

Proves global loss descent and convergence to stationary points.

Abstract

In this work, we propose a federated dynamical low-rank training (FeDLRT) scheme to reduce client compute and communication costs - two significant performance bottlenecks in horizontal federated learning. Our method builds upon dynamical low-rank splitting schemes for manifold-constrained optimization to create a global low-rank basis of network weights, which enables client training on a small coefficient matrix. A consistent global low-rank basis allows us to incorporate a variance correction scheme and prove global loss descent and convergence to a stationary point. Dynamic augmentation and truncation of the low-rank bases automatically optimizes computing and communication resource utilization. We demonstrate the efficiency of FeDLRT in an array of computer vision benchmarks and show a reduction of client compute and communication costs by up to an order of magnitude with minimal impacts on global accuracy.