Automated Worst-Case Performance Analysis of Decentralized Gradient Descent

TL;DR

This paper introduces an automated method to precisely determine the worst-case performance of decentralized gradient algorithms, improving bounds and understanding network effects through SDP-based analysis.

Contribution

It extends the Performance Estimation Problem approach to decentralized optimization, providing exact and relaxed bounds for various network configurations.

Findings

Nearly tight worst-case performance bounds for decentralized (sub)gradient methods

Improved over previous literature in performance bounds

Insights into the impact of network spectral properties on performance

Abstract

We develop a methodology to automatically compute worst-case performance bounds for a class of decentralized algorithms that optimize the average of local functions distributed across a network. We extend the recently proposed PEP approach to decentralized optimization. This approach allows computing the exact worst-case performance and worst-case instance of centralized algorithms by solving an SDP. We obtain an exact formulation when the network matrix is given, and a relaxation when considering entire classes of network matrices characterized by their spectral range. We apply our methodology to the decentralized (sub)gradient method, obtain a nearly tight worst-case performance bound that significantly improves over the literature, and gain insights into the worst communication networks for a given spectral range.