Accelerated Frank-Wolfe Algorithms: Complementarity Conditions and Sparsity

TL;DR

This paper introduces new accelerated Frank-Wolfe algorithms that leverage complementarity conditions to efficiently find sparse solutions in convex optimization problems over polytopes and matrix domains, achieving optimal complexity.

Contribution

The paper develops two novel accelerated Frank-Wolfe algorithms that exploit solution sparsity, providing optimal complexity bounds and reducing dependence on ambient dimension.

Findings

Algorithms achieve optimal first-order oracle complexity.

Methods effectively exploit solution sparsity (face dimension and rank).

Results close a gap in accelerating FW algorithms without dimension dependence.

Abstract

We develop new accelerated first-order algorithms in the Frank-Wolfe (FW) family for minimizing smooth convex functions over compact convex sets, with a focus on two prominent constraint classes: (1) polytopes and (2) matrix domains given by the spectrahedron and the unit nuclear-norm ball. A key technical ingredient is a complementarity condition that captures solution sparsity -- face dimension for polytopes and rank for matrices. We present two algorithms: (1) a purely linear optimization oracle (LOO) method for polytopes that has optimal worst-case first-order (FO) oracle complexity and, aside of a finite \emph{burn-in} phase and up to a logarithmic factor, has LOO complexity that scales with $r/\sqrt{\epsilon}$, where $\epsilon$ is the target accuracy and $r$ is the solution sparsity $r$ (independently of the ambient dimension), and (2) a hybrid scheme that combines FW with a sparse projection oracle (e.g., low-rank SVDs for matrix domains with low-rank solutions), which also has optimal FO oracle complexity, and after a finite burn-in phase, only requires $O(1/\sqrt{\epsilon})$ sparse projections and LOO calls (independently of both the ambient dimension and the rank of optimal solutions). Our results close a gap on how to accelerate recent advancements in linearly-converging FW algorithms for strongly convex optimization, without paying the price of the dimension.