Communication-Efficient Asynchronous Stochastic Frank-Wolfe over Nuclear-norm Balls

TL;DR

This paper introduces an asynchronous stochastic Frank-Wolfe algorithm tailored for nuclear-norm constrained problems in distributed systems, effectively reducing synchronization delays and communication costs while maintaining convergence rates.

Contribution

The paper presents the first asynchronous SFW method that addresses both synchronization and communication challenges in distributed nuclear-norm constrained optimization.

Findings

Achieves near-linear speed-up with increasing machines.

Maintains convergence rate comparable to vanilla SFW.

Demonstrates effectiveness on Amazon EC2.

Abstract

Large-scale machine learning training suffers from two prior challenges, specifically for nuclear-norm constrained problems with distributed systems: the synchronization slowdown due to the straggling workers, and high communication costs. In this work, we propose an asynchronous Stochastic Frank Wolfe (SFW-asyn) method, which, for the first time, solves the two problems simultaneously, while successfully maintaining the same convergence rate as the vanilla SFW. We implement our algorithm in python (with MPI) to run on Amazon EC2, and demonstrate that SFW-asyn yields speed-ups almost linear to the number of machines compared to the vanilla SFW.