Scalable Projection-Free Optimization

TL;DR

This paper introduces scalable, projection-free optimization algorithms, including a sample-efficient Frank-Wolfe variant, a communication-efficient distributed framework, and a derivative-free method for submodular maximization, advancing large-scale machine learning optimization.

Contribution

It presents novel scalable Frank-Wolfe variants, notably 1-SFW requiring only one sample per iteration, and extends to distributed and derivative-free settings for submodular maximization.

Findings

1-SFW achieves optimal complexity bounds with minimal sampling.

QFW enables efficient distributed optimization with reduced communication.

Black-Box Continuous Greedy effectively maximizes submodular functions without derivatives.

Abstract

As a projection-free algorithm, Frank-Wolfe (FW) method, also known as conditional gradient, has recently received considerable attention in the machine learning community. In this dissertation, we study several topics on the FW variants for scalable projection-free optimization. We first propose 1-SFW, the first projection-free method that requires only one sample per iteration to update the optimization variable and yet achieves the best known complexity bounds for convex, non-convex, and monotone DR-submodular settings. Then we move forward to the distributed setting, and develop Quantized Frank-Wolfe (QFW), a general communication-efficient distributed FW framework for both convex and non-convex objective functions. We study the performance of QFW in two widely recognized settings: 1) stochastic optimization and 2) finite-sum optimization. Finally, we propose Black-Box Continuous Greedy, a derivative-free and projection-free algorithm, that maximizes a monotone continuous DR-submodular function over a bounded convex body in Euclidean space.