Sparse Incremental Aggregation in Multi-Hop Federated Learning

TL;DR

This paper explores multi-hop federated learning with in-network model aggregation, proposing novel sparsification methods that significantly enhance communication efficiency without sacrificing convergence, achieving up to 15x improvements.

Contribution

It introduces new correlated sparsification techniques for incremental aggregation in multi-hop FL, demonstrating their effectiveness in maintaining convergence and improving communication efficiency.

Findings

Achieved up to 15x communication efficiency improvement

Proposed novel correlated sparsification methods for IA

Maintained convergence with sparsification in multi-hop FL

Abstract

This paper investigates federated learning (FL) in a multi-hop communication setup, such as in constellations with inter-satellite links. In this setup, part of the FL clients are responsible for forwarding other client's results to the parameter server. Instead of using conventional routing, the communication efficiency can be improved significantly by using in-network model aggregation at each intermediate hop, known as incremental aggregation (IA). Prior works [1] have indicated diminishing gains for IA under gradient sparsification. Here we study this issue and propose several novel correlated sparsification methods for IA. Numerical results show that, for some of these algorithms, the full potential of IA is still available under sparsification without impairing convergence. We demonstrate a 15x improvement in communication efficiency over conventional routing and a 11x improvement over state-of-the-art (SoA) sparse IA.