Sparse Accelerated Exponential Weights

TL;DR

This paper introduces SAEW, an accelerated exponential weights method that achieves optimal convergence rates for sparse parameter estimation in stochastic convex optimization, combining acceleration and sparsity promotion.

Contribution

SAEW is a novel procedure that accelerates exponential weights with successive averaging and hard thresholding, achieving fast convergence and sparsity in online stochastic optimization.

Findings

Achieves the optimal $1/T$ convergence rate for strongly convex functions.

Produces sparse estimators through hard thresholding.

Accelerates the traditional $1/ oot{T}{}$ rate to $1/T$ in stochastic optimization.

Abstract

We consider the stochastic optimization problem where a convex function is minimized observing recursively the gradients. We introduce SAEW, a new procedure that accelerates exponential weights procedures with the slow rate $1/\sqrt{T}$ to procedures achieving the fast rate $1/T$. Under the strong convexity of the risk, we achieve the optimal rate of convergence for approximating sparse parameters in $\mathbb{R}^d$. The acceleration is achieved by using successive averaging steps in an online fashion. The procedure also produces sparse estimators thanks to additional hard threshold steps.