Communication/Computation Tradeoffs in Consensus-Based Distributed Optimization

TL;DR

This paper analyzes how the number of processors and communication frequency affect the efficiency of consensus-based distributed optimization, revealing that optimal strategies depend on a problem-specific tradeoff parameter and can improve speedups.

Contribution

It introduces a theoretical framework for understanding communication/computation tradeoffs in distributed optimization and demonstrates how network topology and communication frequency influence scalability.

Findings

Expander graph communication yields speedups.

Optimal number of processors depends on the tradeoff parameter r.

Less frequent communication can still achieve speedups.

Abstract

We study the scalability of consensus-based distributed optimization algorithms by considering two questions: How many processors should we use for a given problem, and how often should they communicate when communication is not free? Central to our analysis is a problem-specific value $r$ which quantifies the communication/computation tradeoff. We show that organizing the communication among nodes as a $k$-regular expander graph (Reingold, Vadhan, and Wigderson, 2002) yields speedups, while when all pairs of nodes communicate (as in a complete graph), there is an optimal number of processors that depends on $r$. Surprisingly, a speedup can be obtained, in terms of the time to reach a fixed level of accuracy, by communicating less and less frequently as the computation progresses. Experiments on a real cluster solving metric learning and non-smooth convex minimization tasks demonstrate strong agreement between theory and practice.