Stochastic Mirror Descent under Iterate-Dependent Markov Noise: Analysis in the Asymptotic and Finite Time Regimes

TL;DR

This paper analyzes stochastic mirror descent algorithms under iterate-dependent Markov noise, establishing convergence and finite-time bounds in both convex and non-convex settings, applicable to decision-dependent uncertainty problems.

Contribution

It provides the first unified convergence analysis for stochastic mirror descent with state-dependent Markov noise, covering both asymptotic and finite-time regimes.

Findings

Almost sure convergence for convex and non-convex problems.

Finite-time concentration bounds for smooth objectives.

Sample complexity matches classical rates under i.i.d. noise.

Abstract

We study a stochastic optimization problem in which the sampling distribution depends on the decision variable, and the available samples are generated through an iterate-dependent Markov chain. Such settings arise naturally in problems with decision-dependent uncertainty; however, they introduce bias and temporal dependence, which render standard techniques developed for i.i.d.\ noise inapplicable. In this work, we analyze the stochastic mirror descent algorithm under iterate-dependent Markov noise. We first establish almost sure convergence for both convex and non-convex problems under the mild assumption of Lipschitz continuity of the objective function, without requiring differentiability. We then derive finite-time concentration bounds for smooth objectives. In the convex setting, the resulting sample complexity matches the classical rate of stochastic mirror descent under i.i.d.\ noise. In the non-convex setting, we obtain a sample complexity bound in terms of the norm of the Riemannian gradient over the probability simplex. Overall, our results establish a unified convergence framework for stochastic mirror descent with state-dependent Markov noise, and highlight its behavior in both convex and non-convex regimes.