Lazifying Conditional Gradient Algorithms

TL;DR

This paper introduces a method to significantly accelerate conditional gradient algorithms by replacing the costly linear optimization oracle with a faster separation oracle, resulting in substantial speedups in practical computations.

Contribution

It proposes a general framework to lazify conditional gradient algorithms, reducing oracle calls and computational cost while maintaining convergence.

Findings

Achieves several orders of magnitude speedup in wall-clock time.

Reduces the number of linear optimization oracle calls needed.

Applicable to various conditional gradient algorithms.

Abstract

Conditional gradient algorithms (also often called Frank-Wolfe algorithms) are popular due to their simplicity of only requiring a linear optimization oracle and more recently they also gained significant traction for online learning. While simple in principle, in many cases the actual implementation of the linear optimization oracle is costly. We show a general method to lazify various conditional gradient algorithms, which in actual computations leads to several orders of magnitude of speedup in wall-clock time. This is achieved by using a faster separation oracle instead of a linear optimization oracle, relying only on few linear optimization oracle calls.