The Complexity of Large-scale Convex Programming under a Linear Optimization Oracle

TL;DR

This paper analyzes the complexity of large-scale convex programming using linear-optimization-based methods, establishing lower bounds, optimality results, and introducing new algorithms that improve upon classic methods like Frank-Wolfe.

Contribution

It provides the first comprehensive complexity bounds for LCP methods, proves their optimality, and proposes new variants inspired by Nesterov's acceleration for better large-scale performance.

Findings

Established lower complexity bounds for LCP methods.

Proved the optimality of the Frank-Wolfe method and variants.

Introduced new accelerated LCP algorithms with advantages over classic methods.

Abstract

This paper considers a general class of iterative optimization algorithms, referred to as linear-optimization-based convex programming (LCP) methods, for solving large-scale convex programming (CP) problems. The LCP methods, covering the classic conditional gradient (CG) method (a.k.a., Frank-Wolfe method) as a special case, can only solve a linear optimization subproblem at each iteration. In this paper, we first establish a series of lower complexity bounds for the LCP methods to solve different classes of CP problems, including smooth, nonsmooth and certain saddle-point problems. We then formally establish the theoretical optimality or nearly optimality, in the large-scale case, for the CG method and its variants to solve different classes of CP problems. We also introduce several new optimal LCP methods, obtained by properly modifying Nesterov's accelerated gradient method, and demonstrate their possible advantages over the classic CG for solving certain classes of large-scale CP problems.