Bandwidth-Aware Network Topology Optimization for Decentralized Learning

TL;DR

This paper introduces a bandwidth-aware network topology optimization framework for decentralized learning, improving consensus speed and training efficiency under bandwidth constraints using an ADMM-based approach.

Contribution

It proposes a novel optimization framework that accounts for bandwidth limitations, utilizing a Mixed-Integer SDP reformulation and an ADMM-based solution for scalable topology design.

Findings

Outperforms benchmark topologies in consensus speed.

Reduces training time by over 1.11x in homogeneous bandwidth settings.

Reduces training time by over 1.21x in heterogeneous bandwidth settings.

Abstract

Network topology is critical for efficient parameter synchronization in distributed learning over networks. However, most existing studies do not account for bandwidth limitations in network topology design. In this paper, we propose a bandwidth-aware network topology optimization framework to maximize consensus speed under edge cardinality constraints. For heterogeneous bandwidth scenarios, we introduce a maximum bandwidth allocation strategy for the edges to ensure efficient communication among nodes. By reformulating the problem into an equivalent Mixed-Integer SDP problem, we leverage a computationally efficient ADMM-based method to obtain topologies that yield the maximum consensus speed. Within the ADMM substep, we adopt the conjugate gradient method to efficiently solve large-scale linear equations to achieve better scalability. Experimental results demonstrate that the resulting network topologies outperform the benchmark topologies in terms of consensus speed, and reduce the training time required for decentralized learning tasks on real-world datasets to achieve the target test accuracy, exhibiting speedups of more than $1.11\times$ and $1.21\times$ for homogeneous and heterogeneous bandwidth settings, respectively.