Parallel and Distributed Block-Coordinate Frank-Wolfe Algorithms

TL;DR

This paper introduces parallel and distributed Frank-Wolfe algorithms that leverage asynchronous computations and expected delays to achieve significant speedups in large-scale optimization tasks.

Contribution

It develops novel asynchronous parallel and distributed Frank-Wolfe algorithms with robustness to delays, extending the BCFW method for improved scalability and efficiency.

Findings

Achieved significant speedups on structural SVM and Group Fused Lasso tasks.

Algorithms are robust to stragglers and faulty worker threads.

Demonstrated advantages over state-of-the-art synchronous methods.

Abstract

We develop parallel and distributed Frank-Wolfe algorithms; the former on shared memory machines with mini-batching, and the latter in a delayed update framework. Whenever possible, we perform computations asynchronously, which helps attain speedups on multicore machines as well as in distributed environments. Moreover, instead of worst-case bounded delays, our methods only depend (mildly) on \emph{expected} delays, allowing them to be robust to stragglers and faulty worker threads. Our algorithms assume block-separable constraints, and subsume the recent Block-Coordinate Frank-Wolfe (BCFW) method~\citep{lacoste2013block}. Our analysis reveals problem-dependent quantities that govern the speedups of our methods over BCFW. We present experiments on structural SVM and Group Fused Lasso, obtaining significant speedups over competing state-of-the-art (and synchronous) methods.