Distributed Stochastic Variance Reduced Gradient Methods and A Lower Bound for Communication Complexity

TL;DR

This paper introduces a distributed stochastic variance reduced gradient algorithm that optimally balances runtime, communication, and rounds of communication for large-scale convex optimization, and establishes a lower bound for communication rounds.

Contribution

The paper presents a novel distributed stochastic variance reduced gradient method that achieves optimal communication complexity and rounds, matching a new lower bound for distributed first-order algorithms.

Findings

Achieves optimal parallel runtime and communication efficiency.

Outperforms existing methods in rounds of communication when the condition number is moderate.

Proves a lower bound for communication rounds and shows the proposed method attains it.

Abstract

We study distributed optimization algorithms for minimizing the average of convex functions. The applications include empirical risk minimization problems in statistical machine learning where the datasets are large and have to be stored on different machines. We design a distributed stochastic variance reduced gradient algorithm that, under certain conditions on the condition number, simultaneously achieves the optimal parallel runtime, amount of communication and rounds of communication among all distributed first-order methods up to constant factors. Our method and its accelerated extension also outperform existing distributed algorithms in terms of the rounds of communication as long as the condition number is not too large compared to the size of data in each machine. We also prove a lower bound for the number of rounds of communication for a broad class of distributed first-order methods including the proposed algorithms in this paper. We show that our accelerated distributed stochastic variance reduced gradient algorithm achieves this lower bound so that it uses the fewest rounds of communication among all distributed first-order algorithms.