Improved Complexities for Stochastic Conditional Gradient Methods under Interpolation-like Conditions

TL;DR

This paper improves the theoretical understanding of stochastic conditional gradient methods in over-parametrized machine learning, demonstrating better oracle complexities under interpolation-like conditions for convex objectives.

Contribution

It introduces improved complexity bounds for stochastic conditional gradient methods leveraging interpolation-like conditions, including a gradient sliding technique for further efficiency.

Findings

Convex case requires O(ε^{-2}) stochastic gradient calls

Gradient sliding reduces calls to O(ε^{-1.5})

Leverages interpolation-like conditions for complexity improvements

Abstract

We analyze stochastic conditional gradient methods for constrained optimization problems arising in over-parametrized machine learning. We show that one could leverage the interpolation-like conditions satisfied by such models to obtain improved oracle complexities. Specifically, when the objective function is convex, we show that the conditional gradient method requires $\mathcal{O}(\epsilon^{-2})$ calls to the stochastic gradient oracle to find an $\epsilon$-optimal solution. Furthermore, by including a gradient sliding step, we show that the number of calls reduces to $\mathcal{O}(\epsilon^{-1.5})$.