Approximate Frank-Wolfe Algorithms over Graph-structured Support Sets

TL;DR

This paper develops approximate Frank-Wolfe algorithms for convex optimization over graph-structured sets, introducing dual maximization oracles and analyzing their convergence, with empirical evidence of fast real-world image recovery.

Contribution

It proposes dual maximization oracles for approximate FW over graph-structured sets and provides convergence analysis under various conditions, addressing limitations of previous approximation assumptions.

Findings

Convergence rate of . . . in worst case with \u03b4-approximate DMO.

Convergence rate of . (/t^2) when solution is on the boundary.

Empirical results show fast convergence in real-world image recovery.

Abstract

In this paper, we consider approximate Frank-Wolfe (FW) algorithms to solve convex optimization problems over graph-structured support sets where the linear minimization oracle (LMO) cannot be efficiently obtained in general. We first demonstrate that two popular approximation assumptions (additive and multiplicative gap errors) are not applicable in that no cheap gap-approximate LMO oracle exists. Thus, approximate dual maximization oracles (DMO) are proposed, which approximate the inner product rather than the gap. We prove that the standard FW method using a $\delta$-approximate DMO converges as $\mathcal{O}((1-\delta) \sqrt{s}/\delta)$ in the worst case, and as $\mathcal{O}(L/(\delta^2 t))$ over a $\delta$-relaxation of the constraint set. Furthermore, when the solution is on the boundary, a variant of FW converges as $\mathcal{O}(1/t^2)$ under the quadratic growth assumption. Our empirical results suggest that even these improved bounds are pessimistic, showing fast convergence in recovering real-world images with graph-structured sparsity.