# Approximate and Stochastic Greedy Optimization

## TL;DR

This paper analyzes approximate and stochastic greedy algorithms for convex optimization, establishing convergence conditions, rates, and equivalences, and demonstrating their effectiveness on smooth and nonsmooth functions.

## Contribution

It provides a unified convergence analysis for approximate greedy algorithms, introduces stochastic variants with proven convergence, and compares their performance on different convex functions.

## Key findings

- Approximate greedy algorithms converge under certain conditions.
- Stochastic variants can fail with full gradients but succeed with stochastic gradients.
- New stochastic FW algorithm converges for nonsmooth convex functions.

## Abstract

We consider two greedy algorithms for minimizing a convex function in a bounded convex set: an algorithm by Jones [1992] and the Frank-Wolfe (FW) algorithm. We first consider approximate versions of these algorithms. For smooth convex functions, we give sufficient conditions for convergence, a unified analysis for the well-known convergence rate of O(1/k) together with a result showing that this rate is the best obtainable from the proof technique, and an equivalence result for the two algorithms. We also consider approximate stochastic greedy algorithms for minimizing expectations. We show that replacing the full gradient by a single stochastic gradient can fail even on smooth convex functions. We give a convergent approximate stochastic Jones algorithm and a convergent approximate stochastic FW algorithm for smooth convex functions. In addition, we give a convergent approximate stochastic FW algorithm for nonsmooth convex functions. Convergence rates for these algorithms are given and proved.