Stochastic Approximation with Random Step Sizes and Urn Models with Random Replacement Matrices Having Finite Mean

TL;DR

This paper develops a new stochastic approximation framework with random step sizes and drifts, and applies it to urn models with random replacement matrices, relaxing many traditional assumptions.

Contribution

It introduces a novel stochastic approximation result with random step sizes and drifts, and applies it to analyze urn models with minimal assumptions on replacement matrices.

Findings

Proved convergence of urn model vectors in $L^1$ and probability.

Established stochastic approximation results with only finite first moments.

Analyzed Lotka-Volterra type differential equations directly.

Abstract

Stochastic approximation algorithm is a useful technique which has been exploited successfully in probability theory and statistics for a long time. The step sizes used in stochastic approximation are generally taken to be deterministic and same is true for the drift. However, the specific application of urn models with random replacement matrices motivates us to consider stochastic approximation in a setup where both the step sizes and the drift are random, but the sequence is uniformly bounded. The problem becomes interesting when the negligibility conditions on the errors hold only in probability. We first prove a result on stochastic approximation in this setup, which is new in the literature. Then, as an application, we study urn models with random replacement matrices. In the urn model, the replacement matrices need neither be independent, nor identically distributed. We assume that the replacement matrices are only independent of the color drawn in the same round conditioned on the entire past. We relax the usual second moment assumption on the replacement matrices in the literature and require only first moment to be finite. We require the conditional expectation of the replacement matrix given the past to be close to an irreducible matrix, in an appropriate sense. We do not require any of the matrices to be balanced or nonrandom. We prove convergence of the proportion vector, the composition vector and the count vector in $L^1$, and hence in probability. It is to be noted that the related differential equation is of Lotka-Volterra type and can be analyzed directly.