Restarting Frank-Wolfe: Faster Rates Under H\"olderian Error Bounds

TL;DR

This paper introduces a new variant of Frank-Wolfe algorithms that adaptively accelerates convergence by leveraging geometric properties and error bounds, bridging the gap between sublinear and linear rates.

Contribution

The authors propose a restart-based Frank-Wolfe method that adapts to problem geometry using strong Wolfe primal bounds, achieving faster convergence rates.

Findings

Achieves interpolated convergence rates between sublinear and linear regimes.

Utilizes strong Wolfe primal bounds combining geometric and H"olderian error bounds.

Demonstrates improved convergence behavior under new error bound conditions.

Abstract

Conditional Gradient algorithms (aka Frank-Wolfe algorithms) form a classical set of methods for constrained smooth convex minimization due to their simplicity, the absence of projection steps, and competitive numerical performance. While the vanilla Frank-Wolfe algorithm only ensures a worst-case rate of $\mathcal{O}(1/\epsilon)$, various recent results have shown that for strongly convex functions on polytopes, the method can be slightly modified to achieve linear convergence. However, this still leaves a huge gap between sublinear $\mathcal{O}(1/\epsilon)$ convergence and linear $\mathcal{O}(\log 1/\epsilon)$ convergence to reach an $\epsilon$-approximate solution. Here, we present a new variant of Conditional Gradient algorithms, that can dynamically adapt to the function's geometric properties using restarts and smoothly interpolates between the sublinear and linear regimes. These interpolated convergence rates are obtained when the optimization problem satisfies a new type of error bounds, which we call \textit{strong Wolfe primal bounds}. They combine geometric information on the constraint set with H\"olderian Error Bounds on the objective function.