Communication Complexity of Distributed Convex Learning and Optimization

TL;DR

This paper investigates the fundamental communication limits in distributed convex learning, revealing scenarios where current algorithms are optimal and others where improvements are possible, especially when local functions lack similarity.

Contribution

It provides a comprehensive analysis of the communication complexity in distributed convex optimization, identifying optimality conditions and highlighting the impact of local function similarity.

Findings

Certain algorithms are proven to be worst-case optimal.

Without similarity between local functions, many communication rounds are necessary.

The results depend on assumptions about local data and function types.

Abstract

We study the fundamental limits to communication-efficient distributed methods for convex learning and optimization, under different assumptions on the information available to individual machines, and the types of functions considered. We identify cases where existing algorithms are already worst-case optimal, as well as cases where room for further improvement is still possible. Among other things, our results indicate that without similarity between the local objective functions (due to statistical data similarity or otherwise) many communication rounds may be required, even if the machines have unbounded computational power.