High Probability Convergence for Accelerated Stochastic Mirror Descent

TL;DR

This paper presents a generic approach to establish high probability convergence guarantees for stochastic convex optimization algorithms, with bounds depending on initial distance rather than domain diameter, applicable to various function classes.

Contribution

It introduces a universal method for high probability convergence analysis of stochastic mirror descent, improving upon expectation-based bounds and domain-dependent bounds in prior work.

Findings

High probability convergence bounds depending on initial distance

Applicable to Lipschitz, smooth functions, and their linear combinations

Step sizes similar to standard settings

Abstract

In this work, we describe a generic approach to show convergence with high probability for stochastic convex optimization. In previous works, either the convergence is only in expectation or the bound depends on the diameter of the domain. Instead, we show high probability convergence with bounds depending on the initial distance to the optimal solution as opposed to the domain diameter. The algorithms use step sizes analogous to the standard settings and are universal to Lipschitz functions, smooth functions, and their linear combinations.