$k$FW: A Frank-Wolfe style algorithm with stronger subproblem oracles

TL;DR

This paper introduces $k$FW, a Frank-Wolfe variant using stronger subproblem oracles to improve convergence speed, especially for sparse solutions, outperforming existing methods in various convex optimization problems.

Contribution

The paper develops $k$FW, a novel Frank-Wolfe algorithm with enhanced subproblem oracles that achieve faster convergence and better performance on sparse convex optimization problems.

Findings

Achieves finite convergence rate on polytopes and group norm balls.

Attains linear convergence on spectrahedra and nuclear norm balls.

Demonstrates an order-of-magnitude speedup in numerical experiments.

Abstract

This paper proposes a new variant of Frank-Wolfe (FW), called $k$FW. Standard FW suffers from slow convergence: iterates often zig-zag as update directions oscillate around extreme points of the constraint set. The new variant, $k$FW, overcomes this problem by using two stronger subproblem oracles in each iteration. The first is a $k$ linear optimization oracle ($k$LOO) that computes the $k$ best update directions (rather than just one). The second is a $k$ direction search ($k$DS) that minimizes the objective over a constraint set represented by the $k$ best update directions and the previous iterate. When the problem solution admits a sparse representation, both oracles are easy to compute, and $k$FW converges quickly for smooth convex objectives and several interesting constraint sets: $k$FW achieves finite $\frac{4L_f^3D^4}{\gamma\delta^2}$ convergence on polytopes and group norm balls, and linear convergence on spectrahedra and nuclear norm balls. Numerical experiments validate the effectiveness of $k$FW and demonstrate an order-of-magnitude speedup over existing approaches.