Stochastic Recursive Inclusions in two timescales with non-additive iterate dependent Markov noise

TL;DR

This paper analyzes the asymptotic behavior of a two-timescale stochastic approximation scheme with set-valued drifts and non-additive Markov noise, extending previous work to more complex noise dependencies.

Contribution

It introduces a framework for analyzing two-timescale stochastic approximations with non-additive Markov noise, generalizing prior models and applicable to constrained optimization.

Findings

Recursion tracks differential inclusions averaged over stationary measures.

Framework accommodates non-additive, iterate-dependent Markov noise.

Applicable to optimization problems without requiring differentiability or knowledge of measures.

Abstract

In this paper we study the asymptotic behavior of a stochastic approximation scheme on two timescales with set-valued drift functions and in the presence of non-additive iterate-dependent Markov noise. It is shown that the recursion on each timescale tracks the flow of a differential inclusion obtained by averaging the set-valued drift function in the recursion with respect to a set of measures which take into account both the averaging with respect to the stationary distributions of the Markov noise terms and the interdependence between the two recursions on different timescales. The framework studied in this paper builds on the works of \it{A. Ramaswamy et al. }\rm by allowing for the presence of non-additive iterate-dependent Markov noise. As an application, we consider the problem of computing the optimum in a constrained convex optimization problem where the objective function and the constraints are averaged with respect to the stationary distribution of an underlying Markov chain. Further the proposed scheme neither requires the differentiability of the objective function nor the knowledge of the averaging measure.