Block-Coordinate Frank-Wolfe Optimization for Structural SVMs

TL;DR

This paper introduces a randomized block-coordinate Frank-Wolfe algorithm for convex optimization with block-separable constraints, achieving similar convergence rates as the classic method but with lower iteration costs, and demonstrating superior performance on structural SVMs.

Contribution

It presents a novel block-coordinate Frank-Wolfe algorithm that is efficient for structural SVMs, providing optimal step-size computation and duality gap guarantees.

Findings

Achieves similar convergence rate as full Frank-Wolfe with lower iteration cost

Yields an online algorithm with low iteration complexity for structural SVMs

Outperforms existing structural SVM solvers in experiments

Abstract

We propose a randomized block-coordinate variant of the classic Frank-Wolfe algorithm for convex optimization with block-separable constraints. Despite its lower iteration cost, we show that it achieves a similar convergence rate in duality gap as the full Frank-Wolfe algorithm. We also show that, when applied to the dual structural support vector machine (SVM) objective, this yields an online algorithm that has the same low iteration complexity as primal stochastic subgradient methods. However, unlike stochastic subgradient methods, the block-coordinate Frank-Wolfe algorithm allows us to compute the optimal step-size and yields a computable duality gap guarantee. Our experiments indicate that this simple algorithm outperforms competing structural SVM solvers.