Pick your Neighbor: Local Gauss-Southwell Rule for Fast Asynchronous Decentralized Optimization

TL;DR

This paper introduces an optimization-aware neighbor selection rule for asynchronous decentralized optimization, leveraging a Gauss-Southwell inspired approach to improve convergence speed by prioritizing the most impactful communication, validated through theoretical analysis and experiments.

Contribution

It proposes a novel set-wise Gauss-Southwell rule for neighbor selection in asynchronous decentralized optimization, achieving significant speedup over random selection methods.

Findings

Achieves up to maximum degree speedup in network communication.

Provides a new analytical framework for set-wise coordinate descent.

Validated improvements through numerical experiments.

Abstract

In decentralized optimization environments, each agent $i$ in a network of $n$ nodes has its own private function $f_i$, and nodes communicate with their neighbors to cooperatively minimize the aggregate objective $\sum_{i=1}^n f_i$. In this setting, synchronizing the nodes' updates incurs significant communication overhead and computational costs, so much of the recent literature has focused on the analysis and design of asynchronous optimization algorithms, where agents activate and communicate at arbitrary times without needing a global synchronization enforcer. However, most works assume that when a node activates, it selects the neighbor to contact based on a fixed probability (e.g., uniformly at random), a choice that ignores the optimization landscape at the moment of activation. Instead, in this work we introduce an optimization-aware selection rule that chooses the neighbor providing the highest dual cost improvement (a quantity related to a dualization of the problem based on consensus). This scheme is related to the coordinate descent (CD) method with the Gauss-Southwell (GS) rule for coordinate updates; in our setting however, only a subset of coordinates is accessible at each iteration (because each node can communicate only with its neighbors), so the existing literature on GS methods does not apply. To overcome this difficulty, we develop a new analytical framework for smooth and strongly convex $f_i$ that covers the class of set-wise CD algorithms -- a class that directly applies to decentralized scenarios, but is not limited to them -- and we show that the proposed set-wise GS rule achieves a speedup factor of up to the maximum degree in the network (which is in the order of $\Theta(n)$ for highly connected graphs). The speedup predicted by our analysis is validated in numerical experiments with synthetic data.