Towards accelerated rates for distributed optimization over time-varying networks

TL;DR

This paper introduces an accelerated decentralized optimization algorithm for time-varying networks that improves communication and computation efficiency by combining multi-step gossip with Nesterov's scheme, achieving faster convergence.

Contribution

It proposes a novel accelerated method for decentralized optimization over dynamic networks, integrating multi-step gossip with Nesterov acceleration for improved rates.

Findings

Achieves $ ilde{O}( ext{sqrt}( ext{kappa}_g) ext{chi} ext{log}^2(1/ε))$ communication complexity.

Requires $O( ext{sqrt}( ext{kappa}_g) ext{log}(1/ε))$ gradient computations per node.

Performs well over time-varying networks with improved convergence rates.

Abstract

We study the problem of decentralized optimization over time-varying networks with strongly convex smooth cost functions. In our approach, nodes run a multi-step gossip procedure after making each gradient update, thus ensuring approximate consensus at each iteration, while the outer loop is based on accelerated Nesterov scheme. The algorithm achieves precision $\varepsilon > 0$ in $O(\sqrt{\kappa_g}\chi\log^2(1/\varepsilon))$ communication steps and $O(\sqrt{\kappa_g}\log(1/\varepsilon))$ gradient computations at each node, where $\kappa_g$ is the global function number and $\chi$ characterizes connectivity of the communication network. In the case of a static network, $\chi = 1/\gamma$ where $\gamma$ denotes the normalized spectral gap of communication matrix $\mathbf{W}$. The complexity bound includes $\kappa_g$, which can be significantly better than the worst-case condition number among the nodes.