Non-convex Conditional Gradient Sliding

TL;DR

This paper introduces Non-Convex Conditional Gradient Sliding (NCGS), a projection-free optimization method that improves efficiency over existing non-convex Frank-Wolfe algorithms in various stochastic and finite-sum settings.

Contribution

It proposes NCGS, a novel non-convex extension of CGS that outperforms non-convex Frank-Wolfe methods in multiple problem settings.

Findings

NCGS outperforms existing non-convex Frank-Wolfe algorithms.

NCGS reduces gradient computations in non-convex optimization.

The method is effective in batched, stochastic, and finite-sum scenarios.

Abstract

We investigate a projection free method, namely conditional gradient sliding on batched, stochastic and finite-sum non-convex problem. CGS is a smart combination of Nesterov's accelerated gradient method and Frank-Wolfe (FW) method, and outperforms FW in the convex setting by saving gradient computations. However, the study of CGS in the non-convex setting is limited. In this paper, we propose the non-convex conditional gradient sliding (NCGS) which surpasses the non-convex Frank-Wolfe method in batched, stochastic and finite-sum setting.