Adaptive Open-Loop Step-Sizes for Accelerated Convergence Rates of the Frank-Wolfe Algorithm

TL;DR

This paper introduces a log-adaptive open-loop step-size rule for the Frank-Wolfe algorithm, achieving convergence rates comparable or superior to fixed-step strategies, and extends theoretical convergence guarantees for a broad class of step-sizes.

Contribution

It proposes a novel log-adaptive step-size scheme for FW, extending convergence analysis to more general open-loop step-sizes, and implements this in the FrankWolfe.jl package.

Findings

Log-adaptive step-sizes attain at least as fast convergence as fixed-parameter ones.

Theoretical extension to general non-decreasing functions g(t) for step-sizes.

Implementation of the adaptive rule in the FrankWolfe.jl software.

Abstract

Recent work has shown that in certain settings, the Frank-Wolfe algorithm (FW) with open-loop step-sizes $\eta_t = \frac{\ell}{t+\ell}$ for a fixed parameter $\ell \in \mathbb{N},\, \ell \geq 2$, attains a convergence rate faster than the traditional $O(t^{-1})$ rate. In particular, when a strong growth property holds, the convergence rate attainable with open-loop step-sizes $\eta_t = \frac{\ell}{t+\ell}$ is $O(t^{-\ell})$. In this setting there is no single value of the parameter $\ell$ that prevails as superior. This paper shows that FW with log-adaptive open-loop step-sizes $\eta_t = \frac{2+\log(t+1)}{t+2+\log(t+1)}$ attains a convergence rate that is at least as fast as that attainable with fixed-parameter open-loop step-sizes $\eta_t = \frac{\ell}{t+\ell}$ for any value of $\ell \in \mathbb{N},\,\ell\geq 2$. To establish our main convergence results, we extend our previous affine-invariant accelerated convergence results for FW to more general open-loop step-sizes of the form $\eta_t = g(t)/(t+g(t))$, where $g:\mathbb{N}\to\mathbb{R}_{\geq 0}$ is any non-decreasing function such that the sequence of step-sizes $(\eta_t)$ is non-increasing. This covers in particular the fixed-parameter case by choosing $g(t) = \ell$ and the log-adaptive case by choosing $g(t) = 2+ \log(t+1)$. To facilitate adoption of log-adaptive open-loop step-sizes, we have incorporated this rule into the {\tt FrankWolfe.jl} software package.