Continuous Time Frank-Wolfe Does Not Zig-Zag, But Multistep Methods Do Not Accelerate

TL;DR

This paper analyzes the Frank-Wolfe algorithm's zig-zagging behavior, showing it is an artifact of discretization, and explores multistep variants that stabilize the method but do not improve convergence rates.

Contribution

It introduces multistep Frank-Wolfe variants based on continuous flow discretizations and demonstrates their limitations in accelerating convergence.

Findings

Zig-zagging is due to discretization, not the continuous flow.

Multistep methods stabilize the algorithm but do not improve convergence rates.

Runge-Kutta schemes cannot outperform vanilla Frank-Wolfe in convergence speed.

Abstract

The Frank-Wolfe algorithm has regained much interest in its use in structurally constrained machine learning applications. However, one major limitation of the Frank-Wolfe algorithm is the slow local convergence property due to the zig-zagging behavior. We observe that this zig-zagging phenomenon can be viewed as an artifact of discretization, as when the method is viewed as an Euler discretization of a continuous time flow, that flow does not zig-zag. For this reason, we propose multistep Frank-Wolfe variants based on discretizations of the same flow whose truncation errors decay as $O(\Delta^p)$, where $p$ is the method's order. This strategy "stabilizes" the method, and allows tools like line search and momentum to have more benefit. However, in terms of a convergence rate, our result is ultimately negative, suggesting that no Runge-Kutta-type discretization scheme can achieve a better convergence rate than the vanilla Frank-Wolfe method. We believe that this analysis adds to the growing knowledge of flow analysis for optimization methods, and is a cautionary tale on the ultimate usefulness of multistep methods.