A simpler approach to obtaining an O(1/t) convergence rate for the projected stochastic subgradient method

TL;DR

This paper introduces a new weighted averaging technique for the projected stochastic subgradient method that achieves an O(1/t) convergence rate with simple proof and implementation, and demonstrates comparable empirical performance.

Contribution

It proposes a novel weighted averaging scheme that simplifies analysis and implementation while maintaining optimal convergence rates.

Findings

Achieves O(1/t) convergence rate with new averaging method.

Simplifies proof and implementation of stochastic subgradient convergence.

Empirically comparable performance to existing techniques.

Abstract

In this note, we present a new averaging technique for the projected stochastic subgradient method. By using a weighted average with a weight of t+1 for each iterate w_t at iteration t, we obtain the convergence rate of O(1/t) with both an easy proof and an easy implementation. The new scheme is compared empirically to existing techniques, with similar performance behavior.