Flexible block-iterative analysis for the Frank-Wolfe algorithm

TL;DR

This paper establishes convergence rates for the block-coordinate Frank-Wolfe algorithm in both convex and nonconvex settings, allowing for more flexible, efficient, and parallelizable optimization strategies.

Contribution

It provides the first convergence analysis of BCFW for nonconvex functions with Lipschitz gradients, enabling partial and parallel updates without full LMO activation.

Findings

Convergence rates match state-of-the-art in convex and nonconvex cases.

Partial and parallel updates improve efficiency.

Demonstrated speedup with FrankWolfe.jl implementation.

Abstract

We prove that the block-coordinate Frank-Wolfe (BCFW) algorithm converges with state-of-the-art rates in both convex and nonconvex settings under a very mild "block-iterative" assumption. This appears to be the first result on BCFW addressing the setting of nonconvex objective functions with Lipschitz-continuous gradients and no additional assumptions. This analysis newly allows for (I) progress without activating the most-expensive linear minimization oracle(s), LMO(s), at every iteration, (II) parallelized updates that do not require all LMOs, and therefore (III) deterministic parallel update strategies that take into account the numerical cost of the problem's LMOs. Our results apply for short-step BCFW as well as an adaptive method for convex functions. New relationships between updated coordinates and primal progress are proven, and a favorable speedup is demonstrated using FrankWolfe.jl.