Communication-Efficient Projection-Free Algorithm for Distributed Optimization

TL;DR

This paper introduces DCGS, a communication-efficient distributed optimization algorithm that improves linear oracle complexity and performs well on Lasso and matrix completion tasks.

Contribution

The paper proposes DCGS, a novel distributed projection-free algorithm with improved linear oracle complexity and comparable communication complexity to existing methods.

Findings

DCGS matches state-of-the-art communication complexity.

DCGS reduces linear oracle complexity to near the order of communication complexity.

Experimental results show significant performance improvements on Lasso and matrix completion.

Abstract

Distributed optimization has gained a surge of interest in recent years. In this paper we propose a distributed projection free algorithm named Distributed Conditional Gradient Sliding(DCGS). Compared to the state-of-the-art distributed Frank-Wolfe algorithm, our algorithm attains the same communication complexity under much more realistic assumptions. In contrast to the consensus based algorithm, DCGS is based on the primal-dual algorithm, yielding a modular analysis that can be exploited to improve linear oracle complexity whenever centralized Frank-Wolfe can be improved. We demonstrate this advantage and show that the linear oracle complexity can be reduced to almost the same order of magnitude as the communication complexity, when the feasible set is polyhedral. Finally we present experimental results on Lasso and matrix completion, demonstrating significant performance improvement compared to the existing distributed Frank-Wolfe algorithm.