Projection-Free Non-Smooth Convex Programming

TL;DR

This paper introduces a projection-free sub-gradient algorithm capable of solving both smooth and non-smooth constrained convex optimization problems with competitive convergence rates, offering an efficient alternative to traditional projected methods.

Contribution

It presents a novel sub-gradient based algorithm that handles non-smooth convex problems without projections, extending the scope of projection-free optimization methods.

Findings

Achieves an $O(1/ oot{T}{}$ convergence rate for both smooth and non-smooth problems.

Performs similarly in expectation with stochastic sub-gradients.

Provides a projection-free alternative to PGD and SGD algorithms.

Abstract

In this paper, we provide a sub-gradient based algorithm to solve general constrained convex optimization without taking projections onto the domain set. The well studied Frank-Wolfe type algorithms also avoid projections. However, they are only designed to handle smooth objective functions. The proposed algorithm treats both smooth and non-smooth problems and achieves an $O(1/\sqrt{T})$ convergence rate (which matches existing lower bounds). The algorithm yields similar performance in expectation when the deterministic sub-gradients are replaced by stochastic sub-gradients. Thus, the proposed algorithm is a projection-free alternative to the Projected sub-Gradient Descent (PGD) and Stochastic projected sub-Gradient Descent (SGD) algorithms.