Stochastic Optimization of Smooth Loss

TL;DR

This paper provides a high probability bound for stochastic optimization with smooth loss and introduces a strategy to tune step size without knowing the optimal classifier, improving theoretical guarantees.

Contribution

It establishes a high probability bound for stochastic optimization with smooth loss and proposes a new step size tuning strategy that does not require prior knowledge of the optimal classifier.

Findings

High probability bound for stochastic optimization with smooth loss

A new step size tuning strategy without needing the optimal classifier

Improved theoretical guarantees for stochastic optimization

Abstract

In this paper, we first prove a high probability bound rather than an expectation bound for stochastic optimization with smooth loss. Furthermore, the existing analysis requires the knowledge of optimal classifier for tuning the step size in order to achieve the desired bound. However, this information is usually not accessible in advanced. We also propose a strategy to address the limitation.