Universal Conditional Gradient Sliding for Convex Optimization

TL;DR

This paper introduces a universal, projection-free optimization method called UCGS that efficiently solves convex problems with H"older continuous gradients without prior smoothness knowledge, improving complexity bounds.

Contribution

The paper presents the first sliding-type algorithm that improves both gradient and overall complexity for approximate solutions in convex optimization with H"older continuous gradients.

Findings

UCGS achieves optimal complexity bounds for weakly smooth convex functions.

UCGS does not require prior knowledge of smoothness parameters.

The method matches state-of-the-art complexity in the smooth case and improves it in the weakly smooth case.

Abstract

In this paper, we present a first-order projection-free method, namely, the universal conditional gradient sliding (UCGS) method, for solving $\varepsilon$-approximate solutions to convex differentiable optimization problems. For objective functions with H\"older continuous gradients, we show that UCGS is able to terminate with $\varepsilon$-solutions with at most $O((M_\nu D_X^{1+\nu}/{\varepsilon})^{2/(1+3\nu)})$ gradient evaluations and $O((M_\nu D_X^{1+\nu}/{\varepsilon})^{4/(1+3\nu)})$ linear objective optimizations, where $\nu\in (0,1]$ and $M_\nu>0$ are the exponent and constant of the H\"older condition. Furthermore, UCGS is able to perform such computations without requiring any specific knowledge of the smoothness information $\nu$ and $M_\nu$. In the weakly smooth case when $\nu\in (0,1)$, both complexity results improve the current state-of-the-art $O((M_\nu D_X^{1+\nu}/{\varepsilon})^{1/\nu})$ results on first-order projection-free method achieved by the conditional gradient method. Within the class of sliding-type algorithms, to the best of our knowledge, this is the first time a sliding-type algorithm is able to improve not only the gradient complexity but also the overall complexity for computing an approximate solution. In the smooth case when $\nu=1$, UCGS matches the state-of-the-art complexity result but adds more features allowing for practical implementation.