Revisiting the Constant Stepsize Stochastic Approximation with Decision-Dependent Markovian Noise

TL;DR

This paper analyzes the convergence and stationary bias of constant stepsize stochastic approximation algorithms under decision-dependent Markovian noise, introducing new regularity conditions and establishing geometric convergence and CLTs.

Contribution

It introduces a local regularity condition called Poisson--Gateaux differentiability, enabling analysis of non-smooth kernels and broadening the understanding of SA convergence under decision-dependent noise.

Findings

Stationary bias is of order O(α) under broad settings.

Established finite-time p-th moment bounds for SA iterates.

Proved geometric weak convergence and a functional CLT for the SA process.

Abstract

We revisit the convergence analysis of constant stepsize stochastic approximation (SA) with decision-dependent Markovian noise, with a focus on characterizing the stationary bias against the root of the mean-field equation. We first establish the finite-time $p$-th moment bounds for the SA iterates in a general decision-dependent setting, which serve as a stability foundation for the subsequent analysis. Building on this foundation, and leveraging a local regularity condition termed Poisson--Gateaux differentiability (WD$^\ast$) for the solution to Poisson equation induced by the decision-dependent Markov kernel, we show that the stationary bias is of the order $\mathcal{O}(\alpha)$ for a broad class of decision-dependent settings. Additionally, we establish geometric weak convergence of the joint SA process towards a unique stationary distribution, and a functional central limit theorem. Our relaxed regularity condition enables us to cover cases of non-smooth kernels such as acceptance--rejection mechanisms, projected Langevin dynamics, and clipped state dynamics.