Analysis of stochastic approximation schemes with set-valued maps in the absence of a stability guarantee and their stabilization

TL;DR

This paper investigates stochastic approximation schemes with set-valued maps, showing that entering an attracting set leads to high-probability convergence, and introduces a feedback stabilization method when a globally attracting set exists.

Contribution

It provides new convergence criteria and stabilization techniques for stochastic approximation schemes with set-valued drifts without requiring stability guarantees.

Findings

High-probability locking into attracting sets after many iterations

New criteria for convergence with locally attracting sets

Feedback stabilization guarantees convergence with globally attracting sets

Abstract

In this paper, we analyze the behavior of stochastic approximation schemes with set-valued maps in the absence of a stability guarantee. We prove that after a large number of iterations if the stochastic approximation process enters the domain of attraction of an attracting set it gets locked into the attracting set with high probability. We demonstrate that the above result is an effective instrument for analyzing stochastic approximation schemes in the absence of a stability guarantee, by using it obtain an alternate criteria for convergence in the presence of a locally attracting set for the mean field and by using it to show that a feedback mechanism, which involves resetting the iterates at regular time intervals, stabilizes the scheme when the mean field possesses a globally attracting set, thereby guaranteeing convergence. The results in this paper build on the works of V.S. Borkar, C. Andrieu and H. F. Chen , by allowing for the presence of set-valued drift functions.

Equations

$$X_{n+1}-X_n-a(n)M_{n+1}=a(n)f(X_n),$$

$$\frac{dx}{dt}=f(x).$$

$$X_{n+1}-X_n-a(n)M_{n+1}\in a(n)F(X_n),$$

$$\frac{dx}{dt}\in F(x).$$

$$\mathbb{P}\left(X_n\to A\mathrm{as}n\to \infty |X_{n_0}\in {\mathcal{O}}^{\prime }\right)\ge 1-2d{e}^{-\tilde{K}/b(n_0)},$$

$$X_{n+1}-X_n-a(n)M_{n+1}\in a(n)F(X_n),$$

$$\Vert M_{n+1}\Vert \le K(1+\Vert x_n\Vert )a.s.,$$

$$\left\{(x,y)\in {\mathbb{R}}^{2d}:x\in {\mathbb{R}}^d,y\in F(x)\right\},$$

$$\frac{dx}{dt}\in F(x),$$

$$X_{n+1}-X_n-a(n)M_{n+1}-a(n){\eta }_{n+1}=a(n)f(X_n),n\ge 0,$$

$$\Phi (t,x):=\left\{\mathrm{x}(t)\in {\mathbb{R}}^d:\mathrm{x}(\cdot )\text{ is a solution of DI }\text{eqref}mfld\text{with }\mathrm{x}(0)=x\right\}.$$

$$B(A):=\left\{x\in {\mathbb{R}}^d:{\cap }_{t\ge 0}\overline{\left\{\Phi (q,x):q\ge t\right\}}\subseteq A\right\}.$$

$$\bar{X}(\omega ,t):=\left(\frac{t-t(n)}{t(n+1)-t(n)}\right)X_{n+1}(\omega )+\left(\frac{t(n+1)-t}{t(n+1)-t(n)}\right)X_n(\omega ),$$

$$\lambda (\omega ):={\cap }_{t\ge 0}\overline{\left\{\bar{X}(\omega ,q):q\ge t\right\}}.$$

$$\mathrm{D}(\mathrm{z},{\mathrm{z}}^{\prime })=\sum _{k=1}^{\infty }\frac{1}{2^k}\min \{\Vert \mathrm{z}-{\mathrm{z}}^{\prime }{\Vert }_{[-k,k]},1\},$$

$$\mathbb{P}(X_n\to A\text{as }n\to \infty |E)\ge 1-2d{e}^{-\tilde{K}/b(n_0)},$$

$$\underset{n_0\to \infty }{\lim }\mathbb{P}(X_n\to A\text{as }n\to \infty |X_{n_0}\in {\mathcal{O}}^{\prime })=1.$$

$${\cup }_{n\ge N}\{X_n\in {\mathcal{O}}^{\prime }\}=\{X_N\in {\mathcal{O}}^{\prime }\}\cup \left({\cup }_{n>N}\{X_k\notin {\mathcal{O}}^{\prime },\text{for }N\le k\le n-1,X_n\in {\mathcal{O}}^{\prime }\}\right),$$

$$\mathbb{P}(\{X_n\to A\text{as }n\to \infty \}$$

$$\ge \sum _{n\ge N}\mathbb{P}\left(\{X_n\to A\text{as }n\to \infty \}\cap \{X_k\notin {\mathcal{O}}^{\prime },\text{for }N\le k\le n-1,X_n\in {\mathcal{O}}^{\prime }\}\right)$$

$$\displaystyle=\sum_{n\geq N}\bigg{[}\mathbb{P}\left(\{X_{n}\to A\ \text{as }n\to\infty|\{X_{k}\notin\mathcal{O}^{\prime},\ \text{for }N\leq k\leq n-1,X_{n}\in\mathcal{O}^{\prime}\}\right)$$

$$\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ \mathbb{P}\left(\{X_{k}\notin\mathcal{O}^{\prime},\ \text{for }N\leq k\leq n-1,X_{n}\in\mathcal{O}^{\prime}\}\right)\bigg{]}$$

$$\ge \sum _{n\ge N}\left(1-2d{e}^{-\tilde{K}/b(n)}\right)\mathbb{P}\left(\{X_k\notin {\mathcal{O}}^{\prime },\text{for }N\le k\le n-1,X_n\in {\mathcal{O}}^{\prime }\}\right)$$

$$\ge \left(1-2d{e}^{-\tilde{K}/b(N)}\right)\sum _{n\ge N}\mathbb{P}\left(\{X_k\notin {\mathcal{O}}^{\prime },\text{for }N\le k\le n-1,X_n\in {\mathcal{O}}^{\prime }\}\right)$$

$$=\left(1-2d{e}^{-\tilde{K}/b(N)}\right)\mathbb{P}\left({\cup }_{n\ge N}\{X_n\in {\mathcal{O}}^{\prime }\}\right)$$

$$\ge \left(1-2d{e}^{-\tilde{K}/b(N)}\right)\mathbb{P}\left({\cap }_{N\ge 0}{\cup }_{n\ge N}\{X_n\in {\mathcal{O}}^{\prime }\}\right).$$

$$\mathbb{P}(\{X_n\to A\text{as }n\to \infty \}\cap ({\cap }_{N\ge 0}{\cup }_{n\ge N}\{X_n\in {\mathcal{O}}^{\prime }\}))=\mathbb{P}\left({\cap }_{N\ge 0}{\cup }_{n\ge N}\{X_n\in {\mathcal{O}}^{\prime }\}\right).$$

$$\{\underset{n\ge 0}{\sup }\Vert X_n\Vert <\infty \}\cap \{\lambda (\cdot )\cap B(A)\ne \varnothing \}\subseteq {\cap }_{N\ge 0}{\cup }_{n\ge N}\{X_n\in {\mathcal{O}}^{\prime }\}.$$

$$\mathbb{P}(\{\underset{n\ge 0}{\sup }\Vert X_n\Vert <\infty \}\cap \{\lambda (\cdot )\cap B(A)\ne \varnothing \})=\mathbb{P}({\cap }_{N\ge 0}{\cup }_{n\ge N}\{X_n\in {\mathcal{O}}^{\prime }\}),$$

$${\chi }_n(\omega )=\{\begin{array}{ll}0& \mathrm{if}{X}_n(\omega )=X_n^{\prime }(\omega ),\\ 1& \mathrm{if}{X}_n(\omega )\ne X_n^{\prime }(\omega ).\end{array}$$

$$\{\omega \in \Omega :\sum _{n\ge 1}{\chi }_n(\omega )<\infty \}\subseteq \{\omega \in \Omega :\sum _{n=0}^{\infty }\mathbb{E}[(a(n){)}^2\Vert M_{n+1}{\Vert }^2|{\mathcal{F}}_n](\omega )<\infty \}.$$

$$\mathrm{H}(S_1,S_2):=\max \left\{\underset{s_1\in S_1}{\sup }\underset{s_2\in S_2}{\inf }\Vert s_1-s_2\Vert ,\underset{s_2\in S_2}{\sup }\underset{s_1\in S_1}{\inf }\Vert s_1-s_2\Vert \right\}.$$

$$\Vert f(x)c-f(x^{\prime })c\Vert$$

$$\le r{L}_{x_0}^f\Vert x-x^{\prime }\Vert .$$

Full text

1 Introduction

It is well known that several optimization and control tasks can be cast as a root finding problem. That is, given $f:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$, one needs to find $x^{*}\in\mathbb{R}^{d}$, such that $f(x^{*})=0$ (given such a point exists). Due to practical considerations, one usually has access to noisy measurements/estimations of the function whose root needs to be determined. An approach to solving such a problem with noisy measurements of $f$, is given by the recursion,

[TABLE]

where $\{M_{n}\}_{n\geq 1}$, denotes the noise arising in the measurement of $f$ and having fixed an initial condition ($X_{0}\in\mathbb{R}^{d}$), the iterates $\{X_{n}\}_{n\geq 1}$ are generated according to recursion $\eqref{std_rec}$. [1] under certain assumptions which include the Lipschitz continuity of the function $f$, boundedness of the iterates along almost every sample path (that is $\mathbb{P}(\sup_{n\geq 0}\|X_{n}\|<\infty)=1$) and a condition which ensures that the eventual contribution of the additive noise terms is negligible, showed that the linearly interpolated trajectory of recursion $\eqref{std_rec}$ tracks the flow of the ordinary differential equation (o.d.e.) given by,

[TABLE]

Such a trajectory is called an *asymptotic pseudotrajectory *for the flow of o.d.e.$\eqref{std_ode}$ (for a precise definition see [1]). Suppose the set of zeros of $f$ is a globally asymptotically stable set for the flow of o.d.e. $\eqref{std_ode}$, then it was shown that the limit set of an asymptotic pseudotrajectory was contained in such a set and hence the iterates $\{X_{n}\}_{n\geq 0}$ converge in the limit to a root of the function $f$.

In order to analyze recursion $\eqref{std_rec}$ when the function $f$ is no longer Lipschitz continuous or even continuous, but is just measurable satisfying the linear growth property, that is for every $x\in\mathbb{R}^{d}$, $\|f(x)\|\leq K(1+\|x\|)$ for some $K>0$, or when there is a non-additive noise/control component taking values in a compact set whose law is not known (in which case the recursion $\eqref{std_rec}$ takes the form $X_{n+1}-X_{n}-a(n)M_{n+1}=a(n)f(X_{n},U_{n})$, where $U_{n}$ denotes the noise/control), the above mentioned o.d.e. method needed to be extended to recursions with much weaker requirements on the function $f$. This was accomplished in [2], where the asymptotic behavior of the recursion given by,

[TABLE]

was studied, where $F$ is a set-valued map satisfying some conditions (while the other quantities have same interpretation as in $\eqref{std_rec}$). Under the assumption of stability of iterates (that is $\mathbb{P}(\sup_{n\geq 0}\|X_{n}\|<\infty)=1$) and appropriate conditions on the additive noise terms, in [2], it was shown that the linearly interpolated trajectory of recursion $\eqref{std_rec_1}$ tracks the flow of the differential inclusion (d.i.) given by,

[TABLE]

We refer the reader to [3, Ch. 5.3] for a detailed argument as to how the measurable case and the case with unknown noise/control be recast in the form of recursion $\eqref{std_rec_1}$. For a brief summary of the convergence analysis of recursion $\eqref{std_rec_1}$ we refer the reader to section 3.1 of this paper.

Common to the analysis of both recursion $\eqref{std_rec}$ and $\eqref{std_rec_1}$ is the assumption on the stability of the iterates, that is $\mathbb{P}(\sup_{n\geq 0}\|X_{n}\|<\infty)=1$. The condition of stability is highly non-trivial and difficult to verify. Over the years significant effort has gone into providing sufficient conditions for stability (see [4], [5]). In [6], it was shown that for recursion $\eqref{std_rec}$, in the absence of stability guarantee, the probability of converging to an attracting set of o.d.e. $\eqref{std_ode}$ given that the iterates lie in a neighborhood of it converged to one as the index ($n$) in which the iterate entered the neighborhood of the attracting set increased to infinity. This probability of the iterates converging to an attracting set given that the iterate lies in a neighborhood of it is called the *lock-in probability *and in [6] a lower bound for the same was used to obtain sample complexity bounds for recursion $\eqref{std_rec}$. Further a tighter lower bound for the lock-in probability was derived in [7] under a slightly stronger noise assumption and used to obtain convergence guarantee when the law of the iterates are tight. In this paper we extend the results in [6] to the case of stochastic approximation schemes with set-valued maps as in recursion $\eqref{std_rec_1}$.

1.1 Contributions and organization of the paper

We first provide a lower bound for the lock-in probability of stochastic approximation schemes with set-valued maps as in recursion $\eqref{std_rec_1}$. The bound is derived under an assumption on the additive noise terms which is stronger than the corresponding in [6], which is necessitated due to the lack of Lipschitz continuity of the drift function $F$. We establish that,

[TABLE]

for $n_{0}$ large, where, $A\subseteq\mathbb{R}^{d}$, denotes an attracting set of DI 4, $\mathcal{O}^{\prime}$ is an open neighborhood of $A$ with compact closure, $\tilde{K}$ is some positive constant and $\{b(n)\}_{n\geq 0}$ is a sequence of reals converging to zero, which are step size dependent.

Having summarized the convergence analysis under stability in section 3.1, we state the lock-in probability bound in section 3.2 and provide a few implications of the same. Using the lock-in probability result we provide an alternate criteria for convergence in the presence of a locally attracting set which removes the need to verify stability. A detailed comparison between the obtained convergence guarantee and the corresponding in the presence of stability is also provided.

Proof of the lock-in probability result is presented in section 5. The proof relies heavily on the insights obtained from the analysis in [6] for single-valued maps. From the analysis in [6], it is evident that the Lipschitz continuity of the drift function $f$ plays a crucial role in obtaining events and decoupling error contributions which in turn are necessary to obtain the bound in the inequality above. But in the recursion studied in this paper (that is recursion (3)), the drift function $F$ is set-valued and the assumptions under which we study the said recursion (which are summarized in section 2), the drift function $F$ is not even continuous. We overcome this problem by first obtaining a sequence of locally Lipschitz continuous set-valued maps which approximate the drift function $F$ from above and then parameterizing them using the Stiener selection procedure. The associated results are summarized in section 5.1. This enables us to write recursion $\eqref{std_rec_1}$ in the form of recursion $\eqref{std_rec}$, but with locally Lipschitz continuous drift functions. Further the relation between the solutions of differential inclusions with the approximating set-valued maps as their vector field and those of DI $\eqref{std_di}$, is established in section 5.2. Having written recursion $\eqref{std_rec_1}$ in the form of recursion $\eqref{std_rec}$, we then collect sample paths of interest in section 5.3. Along the sample paths that are collected the iterates are such that, having entered a neighborhood of the attracting set at iteration $n_{0}$, the iterates will infinitely often enter the said neighborhood and the time elapsed between successive visits to the neighborhood of the attracting set can be upper bounded by a constant which is mean field dependent. Further we show that the probability of occurrence of such sample paths can be lower bounded by error contributions due to additive noise terms alone after a large number of iterations. Using the concentration inequality for martingale sequences we obtain the lock-in probability bound in section 5.4.

Using the lock-in probability result we design a feedback mechanism which enables us to stabilize the stochastic approximation scheme in the presence of a globally attracting set for DI $\eqref{std_di}$. The feedback mechanism involves resetting the iterates at regular time intervals if they are found to be lying outside a certain compact set. This approach to stabilization has been studied in various forms for stochastic approximation schemes with single-valued drift functions as in recursion $\eqref{std_rec}$, in [8], [9], [10] and [11] to name a few. We extend the same to the case of set-valued drift functions. The main idea in the analysis of such a scheme is to show that along almost every sample path of the modified recursion, the number of resets that are performed is finite, thereby guaranteeing that eventually the iterates lie within a compact set. We observe that the lock-in probability result (to be precise the approach adopted to obtain the lock-in probability result) plays a central role in showing that the number of resets performed remain finite. Having shown that the iterates eventually lie within a compact set, we use the convergence arguments from [2] to argue that the iterates generated by the modified scheme converge to the globally attracting set of DI $\eqref{std_di}$. The modified scheme is presented and explained in detail in section 4. The proof of the finite resets theorem is presented in section 6. The procedure employed to collect sample paths in the proof of the lock-in probability result can be used to collect sample paths where only finite number of resets have occurred in the modified scheme and this in turn enables us show that the number of resets are finite almost surely.

Finally, we conclude by providing a few directions for future work in section 7.

2 Recursion and assumptions

Let $\left(\Omega,\mathscr{F},\mathbb{P}\right)$ be a probability space and $\left\{X_{n}\right\}_{n\geq 0}$ be a sequence of $\mathbb{R}^{d}$-valued random variables on $\Omega$, such that for every $n\geq 0$,

[TABLE]

where,

• (A1)

$F:\mathbb{R}^{d}\rightarrow\left\{\text{subsets of }\mathbb{R}^{d}\right\}$ is a set-valued map which for every $x\in\mathbb{R}^{d}$ satisfies the following:

• (i)

$F(x)$ is a convex and compact subset of $\mathbb{R}^{d}$,

• (ii)

there exists $K>0$ (independent of $x$) such that $\sup_{y\in F(x)}\left\|y\right\|\leq K(1+\left\|x\right\|)$,

• (iii)

for every $\mathbb{R}^{d}$-valued sequence $\{x_{n}\}_{n\geq 1}$ converging to $x$ and for every sequence $\{y_{n}\in F(x_{n})\}_{n\geq 1}$ converging to $y\in\mathbb{R}^{d}$, we have that $y\in F(x)$.

• (A2)

$\{a(n)\}_{n\geq 0}$ is a sequence of positive real numbers satisfying,

• (i)

$\sum_{n=0}^{\infty}a(n)=\infty$,

• (ii)

$\sum_{n=0}^{\infty}\left(a(n)\right)^{2}<\infty$.

• (A3)

$\{M_{n}\}_{n\geq 1}$ is a $\mathbb{R}^{d}$-valued, martingale difference sequence with respect to the filtration $\{\mathscr{F}_{n}:=\sigma(X_{m},M_{m},\ m\leq n)\}$. Furthermore, $\{M_{n}\}_{n\geq 1}$ are such that,

[TABLE]

for every $n\geq 0$, for some constant $K>0$.

Assumption $(A1)$ ensures that the set-valued map $F$ is a Marchaud map. The condition $(A1)$(ii) is called the *linear growth property *since it ensures that the size of the sets $F(x)$ grow linearly with respect to the distance from the origin. The condition $(A1)$(iii) is called the *closed graph property *since it states that the graph of the set-valued map $F$, defined as,

[TABLE]

is a closed subset of $\mathbb{R}^{2d}$. The map $F$ being a Marchaud map ensures that the differential inclusion (DI) given by,

[TABLE]

possesses at least one solution through every initial condition. By a solution of DI $\eqref{mfld}$ with initial condition $x_{0}\in\mathbb{R}^{d}$, we mean an absolutely continuous function $\bm{\mathrm{x}}:\mathbb{R}\rightarrow\mathbb{R}^{d}$ such that $\bm{\mathrm{x}}(0)=x_{0}$ and for almost every $t\in\mathbb{R}$, $\frac{d\bm{\mathrm{x}}(t)}{dt}\in F(\bm{\mathrm{x}}(t))$. DI $\eqref{mfld}$ is the *mean field *of recursion $\eqref{rec}$ and its dynamics play an important role in describing the asymptotic behavior of recursion $\eqref{rec}$.

Assumption $(A2)$ states the conditions to be satisfied by the step size sequence $\{a(n)\}_{n\geq 0}$. Square summability (that is $(A2)$(ii)) is needed later in the analysis for obtaining a probability bound on certain tail events associated with the additive noise terms $\{M_{n}\}_{n\geq 1}$.

Assumption $(A3)$, defines the martingale noise model. These terms denote the noise arising in the measurement of $F(\cdot)$. This condition holds in several reinforcement learning applications (see [3, Ch. 10])

• Remark

Clearly when $\{M_{n}\}_{n\geq 1}$ are i.i.d. zero mean and bounded, assumption $(A3)$ is satisfied. Further, since the drift function in recursion $\eqref{rec}$ is a set-valued map, scenarios where the measurement noise terms possess a bounded bias can be recast in the form of recursion $\eqref{rec}$ as explained below.

Consider the recursion given by,

[TABLE]

where $f:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$ is a single-valued Lipschitz continuous map, for every $n\geq 0$, $\eta_{n+1}$ denotes the bias in the measurement noise. Let the bias terms $\{\eta_{n}\}_{n\geq 1}$ be bounded by a positive constant, say $\epsilon>0$ (that is, for every $n\geq 1$, $\|\eta_{n}\|\leq\epsilon$). Then, recursion $\eqref{apd}$ can be written in the form of recursion $\eqref{rec}$ with set-valued map $F$, given by, $F(x)=\{f(x)+\eta:\|\eta\|\leq\epsilon\}$, for every $x\in\mathbb{R}^{d}$. We refer the reader to [3, ch. 5.3] for several other variants of the standard stochastic approximation scheme which can be analyzed with the help of recursion $\eqref{rec}$.

3 Lock-in probability for stochastic recursive inclusions

In order to state the main result of this paper, definition of the flow of a DI, an attracting set for such a dynamical system are needed. We recall these notions below and we state them with respect to the mean field of recursion $\eqref{rec}$ (for a detailed description and associated results see [2]).

The *flow *of DI $\eqref{mfld}$ is given by the set-valued map $\Phi:\mathbb{R}\times\mathbb{R}^{d}\rightarrow\{\text{subsets of }\mathbb{R}^{d}\}$, where for every $(t,x)\in\mathbb{R}\times\mathbb{R}^{d}$,

[TABLE]

A compact set $A\subset\mathbb{R}^{d}$ is an *attracting set *for the flow of DI $\eqref{mfld}$, if there exists an open neighborhood of $A$, say $\mathcal{O}$, with the property that for every $\epsilon>0$, there exists a time $T>0$ (depending on $\epsilon$ and $\mathcal{O}$) such that for every $t\geq T$ and for every $x\in U$, $\Phi(t,x)\in N^{\epsilon}(A)$, where $N^{\epsilon}(A)$ denotes the $\epsilon$-neighborhood of $A$. Such a neighborhood $\mathcal{O}$ of an attracting set $A$ is called the *fundamental neighborhood *of $A$.

The set of initial conditions in $\mathbb{R}^{d}$ from which the flow is attracted to an attracting set $A$ is called the *basin of attraction *and is denoted by $B(A)$. Formally,

[TABLE]

An attracting set $A$ is said to be *globally attracting *if, $B(A)=\mathbb{R}^{d}$.

3.1 Summary of the asymptotic analysis under stability

Let $t(0):=0$ and for every $n\geq 1$, $t(n):=\sum_{k=0}^{n-1}a(k)$. The linearly interpolated trajectory of recursion $\eqref{rec}$, is given by the stochastic process $\bar{X}:\Omega\times\mathbb{R}\rightarrow\mathbb{R}^{d}$, where for every $(\omega,t)\in\Omega\times[0,\infty)$,

[TABLE]

where $n$ is such that $t\in[t(n),t(n+1))$ and for every $(\omega,t)\in\Omega\times(-\infty,0)$, $\bar{X}(\omega,t):=X_{0}(\omega)$.

For $\omega\in\Omega$, the limit set map of $\bar{X}$ is given by, $\lambda:\Omega\rightarrow\{\text{subsets of }\mathbb{R}^{d}\}$ where for every $\omega\in\Omega$,

[TABLE]

In [2], under assumptions $(A1)-(A3)$ along with the additional assumption of stability of the iterates (that is $\mathbb{P}(\sup_{n\geq 0}\|X_{n}\|<\infty)=1$), it was shown that for almost every $\omega\in\Omega$, the linearly interpolated trajectory of recursion $\eqref{rec}$, $\bar{X}(\omega,\cdot)$, is an *asymptotic pseudotrajectory *for the flow of DI $\eqref{mfld}$. More precisely, for almost every $\omega\in\Omega$, $\bar{X}(\omega,\cdot)$ was shown to satisfy the following:

• (a)

The family of shifted trajectories given by $\{\bar{X}(\omega,\cdot+t)\}_{t\geq 0}$ is relatively compact in $\mathcal{C}(\mathbb{R},\mathbb{R}^{d})$ where $\mathcal{C}(\mathbb{R},\mathbb{R}^{d})$ denotes the metric space of all continuous functions on $\mathbb{R}$ taking values in $\mathbb{R}^{d}$ with metric $\bm{\mathrm{D}}$, which for every $\bm{\mathrm{z}},\ \bm{\mathrm{z}}^{\prime}\in\mathcal{C}(\mathbb{R},\mathbb{R}^{d})$ is given by

[TABLE]

where $\|\bm{\mathrm{z}}-\bm{\mathrm{z}}^{\prime}\|_{[-k,k]}:=\sup_{t\in[-k,k]}\|\bm{\mathrm{z}}(t)-\bm{\mathrm{z}}^{\prime}(t)\|$.

• (b)

Every limit point of the shifted trajectories $\{\bar{X}(\omega,\cdot+t)\}_{t\geq 0}$ is a solution of the DI $\eqref{mfld}$.

From [2, Thm. 4.3], it follows that for almost every $\omega\in\Omega$, the limit set of the linearly interpolated trajectory $\bar{X}(\omega,\cdot)$, $\lambda(\omega)$, is a non-empty, compact and an internally chain transitive (ICT) set for the flow of DI $\eqref{mfld}$ (see [2, Defn. VI] for definition of an ICT set). Now using [2, Thm. 3.23] the main convergence result of [2] follows and is stated below.

Theorem 3.1

Let $A\subseteq\mathbb{R}^{d}$ be an attracting set for the flow of DI $\eqref{mfld}$. Under assumptions $(A1)-(A3)$,

• (a)

for almost every $\omega\in\{\omega\in\Omega:\sup_{n\geq 0}\|X_{n}(\omega)\|<\infty\}\cap\{\omega\in\Omega:\lambda(\omega)\cap B(A)\neq\emptyset\}$, $\lambda(\omega)\subseteq A$ and therefore as $n\to\infty$, $X_{n}(\omega)\to A$.

• (b)

if $B(A)=\mathbb{R}^{d}$ (that is $A$ is a globally attracting set), then for almost every $\omega\in\{\omega\in\Omega:\sup_{n\geq 0}\|X_{n}(\omega)\|<\infty\}$, $\lambda(\omega)\subseteq A$ and therefore as $n\to\infty$, $X_{n}(\omega)\to A$.

The assumption of stability of the iterates used to obtain the above convergence result is highly non-trivial and difficult to verify. Moreover the proof method used to prove the above convergence result cannot be modified in a straight forward manner to obtain a similar convergence guarantee. This warrants an alternate approach to study the behavior of recursion $\eqref{rec}$ in the absence of stability guarantee and we accomplish this by extending the lock-in probability result from [6] to the set-valued case. Using the obtained lock-in probability bound we recover convergence guarantee similar to Theorem 3.1 while eliminating the need to verify stability.

3.2 Main result and its implications

Before we state the main result, we state an assumption which fixes the attracting set of interest.

• (A4)

Let $A\subseteq\mathbb{R}^{d}$, be an attracting set of DI $\eqref{mfld}$ (the mean field of recursion $\eqref{rec}$) with $\mathcal{O}\subseteq\mathbb{R}^{d}$ as its fundamental neighborhood of attraction.

Let $\mathcal{O}^{\prime}$ be an open neighborhood of the attracting set $A$ (as in $(A4)$) such that $\bar{\mathcal{O}^{\prime}}$ is compact and $\bar{\mathcal{O}^{\prime}}\subseteq\mathcal{O}$. Then the main result of the paper can be stated as follows.

Theorem 3.2

(Lock-in probability)* Under assumptions $(A1)-(A4)$, there exists a constant $\tilde{K}>0$ (depending on the attracting set $A$ and $\mathcal{O}^{\prime}$) and an $N_{0}\geq 1$ such that, for every $n_{0}\geq N_{0}$, for every $E\in\mathscr{F}_{n_{0}}$ satisfying $E\subseteq\{\omega\in\Omega:X_{n_{0}}(\omega)\in\mathcal{O}^{\prime}\}$ and $\mathbb{P}(E)>0$, we have that,*

[TABLE]

where, for every $n\geq 0$, $b(n):=\sum_{k=n}^{\infty}(a(k))^{2}$.

There are two immediate implications of the above result and are stated below, one of which serves as an alternate convergence result in the absence of stability guarantees, that is it allows us to obtain the convergence guarantee in Theorem 3.1$(a)$ without the need to verify whether a given sample path satisfies $\sup_{n\geq 0}\|X_{n}(\omega)\|<\infty$.

• (1)

As a consequence of assumption $(A2)$(ii), we have that $\lim_{n\to\infty}b(n)=0$. Therefore from Theorem 3.2, if the observation that iterate lies in a neighborhood of the attracting set is made later in time ($n_{0}$), the probability of converging to the attracting set increases and converges to one as $n_{0}\to\infty$. Formally,

[TABLE]

• (2)

Suppose $\mathbb{P}(\cap_{N\geq 0}\cup_{n\geq N}\{X_{n}\in\mathcal{O}^{\prime}\})>0$ (if $\mathbb{P}(\cap_{N\geq 0}\cup_{n\geq N}\{X_{n}\in\mathcal{O}^{\prime}\})=0$ then the iterates almost surely do not converge to the attracting set $A$). Then for every $N\geq 0$, $\mathbb{P}(\cup_{n\geq N}\{X_{n}\in\mathcal{O}^{\prime}\})>0$ and

[TABLE]

where, the union in the R.H.S. is disjoint. Then by Theorem 3.2, for every $N\geq N_{0}$,

[TABLE]

The above inequality is true for every $N\geq N_{0}$. Taking limit and using the fact that $\lim_{n\to\infty}b(n)=0$, we get that,

[TABLE]

Therefore from the above we can conclude that,

Corollary 3.3

Under assumptions $(A1)-(A4)$, for almost every $\omega\in\cap_{N\geq 0}\cup_{n\geq N}\{X_{n}\in\mathcal{O}^{\prime}\}$, $X_{n}(\omega)\to A$ as $n\to\infty$.

• Remark

In comparison with Theorem 3.1$(a)$, the condition that $\omega\in\cap_{N\geq 0}\cup_{n\geq N}\{X_{n}\in\mathcal{O}^{\prime}\}$ is stronger than the requirement that $\omega\in\{\lambda(\omega)\cap B(A)\neq\emptyset\}$ because the former requires the iterate sequence to enter an open neighborhood of $A$ with compact closure infinitely often while the latter requires the iterates to enter the basin of attraction of $A$ infinitely often which is larger than $\mathcal{O}^{\prime}$. But in the presence of stability we have that,

[TABLE]

Further, as a consequence of Corollary 3.3, we have that,

[TABLE]

or in other words, the sample paths which visit $\mathcal{O}^{\prime}$ infinitely often and are unstable, occur with zero probability.

4 Application: Stabilization via resetting

In this section we modify recursion $\eqref{rec}$ in such a way that the modified procedure yields sample paths which are stable (that is lie in a compact set almost surely) which in turn allows us to recover the convergence result as in Theorem 3.1$(b)$ without the need to verify stability, in the presence of a globally attracting set for the mean field. That is, we replace assumption $(A4)$ with the following stronger requirement.

• (A4)’

Let $A\subseteq\mathbb{R}^{d}$ be a globally attracting set for the flow of DI $\eqref{mfld}$.

The modification that we propose involves resetting the iterates at regular time intervals if they are found to be lying outside a certain compact set. Let the initial condition $X_{0}(\omega)=x_{0}\in\mathbb{R}^{d}$ for every $\omega\in\Omega$ and $\{r_{n}\in(0,\infty)\}_{n\geq 0}$ be such that,

• (1)

$\|x_{0}\|<r_{0}$,

• (2)

for every $n\geq 0$, $r_{n}<r_{n+1}$,

• (3)

$\lim_{n\to\infty}r_{n}=\infty$.

The modified scheme, henceforth referred to as *stabilized stochastic recursive inclusion *(SSRI) is where every sample path is generated as outlined in Algorithm 1.

A flowchart depicting the flow of control in Algorithm 1 is presented in Figure 1. In order to understand the algorithm let us consider the scenario where the $k^{th}$ reset has been performed at iteration index $n_{0}$. Then the algorithm checks whether the iterate lies in the compact set $r_{k}U$ ( closed ball of radius $r_{k}$ centered at the origin) after approximately $2^{k}T_{W}$ amount of time has elapsed (for the relation between time and iteration index see section 3.1). Now either a reset occurs or the iterate is left unchanged.

• (a)

If the iterate is left unchanged then the next reset check is performed after $2^{k}T_{W}$ amount of time has elapsed.

• (b)

If the iterate is reset, then, the next check is performed after $2^{k+1}T_{W}$ amount of time has elapsed.

In fact it would suffice if the time between successive reset checks were set to be greater than a certain threshold which is determined by the minimum time needed by the flow of the mean field (that is DI $\eqref{mfld}$) to reach the attracting set $A$ from any initial condition in a compact neighborhood of it. But in practical scenarios one may not be able to compute such a time and hence may not be able to determine the required threshold. This approach of increasing time duration between successive reset checks with increasing reset count allows us to bypass this problem. The choice of exponentially increasing durations is one of convenience as it simplifies notations involved in proving certain results later.

For every $n\geq 1$, define the indicator random variable $\chi_{n}:\Omega\rightarrow\{0,1\}$ such that, for every $\omega\in\Omega$,

[TABLE]

We assume that the noise terms $\{M_{n}\}_{n\geq 1}$ satisfy the following version of assumption $(A3)$.

• (A3)’

$\{M_{n}\}$ is a martingale difference sequence with respect to the filtration $\{\mathscr{F}_{n}\}_{n\geq 1}$, where, for every $n\geq 1$, $\mathscr{F}_{n}$ denotes the smallest $\sigma$-algebra generated by the iterates $X_{m}$ (that is the iterates before the reset operation) and noise terms $M_{m}$, for $0\leq m\leq n$ (then it is easy to show that for every $n\geq 1$, $X_{n}^{\prime}$ and hence $\chi_{n}$ are $\mathscr{F}_{n}$ measurable). Since for every $n\geq 1$, $M_{n}$ denotes the noise arising in the estimation (or measurement) of $F$ at $X_{n-1}^{\prime}$, we assume that the energy of the noise depends on $X_{n-1}^{\prime}$. That is for every $n\geq 0$, $\|M_{n+1}\|\leq K(1+\|X_{n}^{\prime}\|)\ a.s.$.

The next theorem says that, for almost every sample path generated by Algorithm 1, the total number of resets is finite, thereby guaranteeing stability. The proof of this theorem (provided in section 6) crucially hinges on a lower bound for the probability of the event that there are no future resets given that there are a certain number of resets up until iteration $n_{0}$ for some large $n_{0}$. Specifically it requires the probability of the above mentioned event to converge to one as $n_{0}$ tends to infinity and this is guaranteed by Theorem 3.2.

Theorem 4.1

(Finite resets)* Under assumptions $(A1),(A2),(A3)^{\prime}$ and $(A4)^{\prime}$, $\mathbb{P}\left(\left\{\omega\in\Omega:\sum_{n=1}^{\infty}\chi_{n}(\omega)<\infty\right\}\right)\\ =1$.*

As a consequence of the above theorem, we have the following.

• (a)

Let $\omega\in\{\omega\in\Omega:\sum_{n=1}^{\infty}\chi_{n}(\omega)<\infty\}$. Then there exists an $N\geq 1$ and $R>0$ (depending on $\omega$) such that, for every $n\geq N$, $X_{n}(\omega)=X_{n}^{\prime}(\omega)$ and $\sup_{n\geq N}\|X_{n}(\omega)\|\leq R$. Therefore $\sum_{n\geq N}\mathbb{E}[(a(n))^{2}\|M_{n+1}\|^{2}|\mathscr{F}_{n}](\omega)\leq\sum_{n\geq N}(a(n))^{2}K^{2}\left(1+\|X_{n}^{\prime}(\omega)\|\right)^{2}\leq K^{2}(1+R)^{2}\sum_{n\geq N}(a(n))^{2}\\ <\infty$, where the last inequality follows from assumption $(A2)$(ii). Therefore,

[TABLE]

Therefore by Theorem 4.1, we have that the $\mathbb{P}(\sum_{n=0}^{\infty}\mathbb{E}[(a(n))^{2}\|M_{n+1}\|^{2}|\mathscr{F}_{n}]<\infty)=1$ and by martingale convergence theorem (see [3, Section 11.3, Thm. 11]) we have that, the square integrable martingale $\{\sum_{m=0}^{n-1}a(m)M_{m+1},\mathscr{F}_{n}\}_{n\geq 1}$ converges almost surely.

• (b)

Thus for $\omega$ lying in a probability one set, there exists $N\geq 1$ and $R>0$ (depending on $\omega$) such that along this sample path the iterates $\{X_{n}(\omega)\}_{n\geq N}$, can be viewed as being generated by recursion $\eqref{rec}$ with initial condition $X_{N}(\omega)$, their norms are bounded by $R$ uniformly and the additive noise terms $\{M_{n}(\omega)\}_{n\geq N}$ satisfy the hypothesis of [2, Prop. 1.3]. Then by arguments similar to those of Theorem 3.1$(b)$ we have that,

Corollary 4.2

Under assumptions $(A1)-(A3)$ and $(A4)^{\prime}$, for almost every $\omega$, the iterates generated by Algorithm 1, $\{X_{n}^{\prime}(\omega)\}_{n\geq 0}$, are such that $X_{n}^{\prime}(\omega)\to A$ as $n\to\infty$.

5 Proof of the lock-in probability theorem (Thm. 3.2)

Proof of the lock-in probability result follows as a consequence of a series of lemmas. The overall structure can be summarized as follows.

• (a)

Our first aim is to replace the set-valued map in recursion $\eqref{rec}$ with an equivalent single-valued locally Lipschitz continuous function with an additional parameter. In order to accomplish this, we first embed the graph of the set-valued map $F$ in the graph of a sequence of locally Lipschitz continuous set-valued maps. These maps are then parametrized using the Stiener selection procedure which preserves the modulus of continuity.

• (b)

The relation between the solutions of DI $\eqref{mfld}$ and that of differential inclusions with continuous set-valued maps which approximate $F$ (as in (a) above) is established.

• (c)

An ordinary differential equation(o.d.e) is defined using an appropriate single-valued parametrization of $F$ (as in (a) above). The existence of solutions to such an o.d.e. and further its uniqueness follow from Caratheodary’s existence theorem and locally Lipschitz nature of the vector field respectively. The solutions of this o.d.e. aide in separating the probability contributions due to the additive noise terms and the set-valued nature of the drift function. Using the results from part (b) above, we conclude that after a large number of iterations, the probability contribution is only due to the additive noise terms.

• (d)

We finally review the standard probability lower bounding procedure for the additive noise terms from [3, Ch. 4]. Using this bound in the result obtained in part (c) above gives us the desired lock-in probability bound.

Throughout, we use $U$ to denote the closed unit ball in $\mathbb{R}^{d}$ centered at the origin. Further, for every $Y_{1},Y_{2}\subseteq\mathbb{R}^{d}$ and $r\in\mathbb{R}$, define,

• •

$Y_{1}+Y_{2}:=\{y_{1}+y_{2}:y_{1}\in Y_{1}\ \mathrm{and}\ y_{2}\in Y_{2}\}$,

• •

$rY_{1}:=\{ry_{1}:y_{1}\in Y_{1}\}$.

5.1 Upper semicontinuous set-valued maps and their approximation

First we recall definitions of continuous set-valued maps and locally Lipschitz continuous set-valued maps. These notions are taken from [12, Ch. 1].

• Definition

A set-valued map $F:\mathbb{R}^{d}\rightarrow\{\text{compact subsets of }\mathbb{R}^{d}\}$ is,

• –

*upper semicontinuous *(u.s.c.) if, for every $x\in\mathbb{R}^{d}$, for every $\epsilon>0$, there exists a $\delta>0$ (depending on $x$ and $\epsilon$) such that, for every $x^{\prime}\in\mathbb{R}^{d}$ satisfying $\|x^{\prime}-x\|<\delta$, we have that $F(x^{\prime})\subseteq F(x)+\epsilon U$, where $F(x)+\epsilon U:=\{y+\epsilon u:y\in F(x),\ u\in U\}$.

• –

*lower semicontinuous *(l.s.c.) if, for every $x\in\mathbb{R}^{d}$, for every $\mathbb{R}^{d}$-valued sequence $\{x_{n}\}_{n\geq 1}$ converging to $x$, for every $y\in F(x)$, there exists a sequence $\{y_{n}\in F(x_{n})\}_{n\geq 1}$ converging to $y$.

• –

*continuous *if, it is both u.s.c. and l.s.c.

• –

*locally Lipschitz continuous *if, for every $x_{0}\in\mathbb{R}^{d}$, there exists $\delta>0$ and $L>0$ (depending on $x_{0}$) such that for every $x,x^{\prime}\in x_{0}+\delta U$, we have that $F(x)\subseteq F(x^{\prime})+L\|x-x^{\prime}\|U$.

Let $\mathcal{K}(\mathbb{R}^{d})$ denote the family of all non-empty compact subsets of $\mathbb{R}^{d}$. Let $\bm{\mathrm{H}}:\mathcal{K}(\mathbb{R}^{d})\times\mathcal{K}(\mathbb{R}^{d})\rightarrow[0,\infty)$ be defined such that, for every $S_{1},\ S_{2}\in\mathcal{K}(\mathbb{R}^{d})$,

[TABLE]

With $\bm{\mathrm{H}}$ as defined above, $(\mathcal{K}(\mathbb{R}^{d}),\bm{\mathrm{H}})$ is a complete metric space (for a proof see [13, Thm. 1.1.2]). The notions of continuity and local Lipschitz continuity of a set-valued map can be restated using the metric defined above and is stated as a lemma below for easy reference (for a proof see [12, Ch. 1, section 5, Cor. 1]).

Lemma 5.1

A set-valued map $F:\mathbb{R}^{d}\rightarrow\mathcal{K}(\mathbb{R}^{d})$ is

• (a)

Continuous, if and only if, for every $x_{0}\in\mathbb{R}^{d}$, for every $\epsilon>0$, there exists $\delta>0$ (depending on $x_{0}$ and $\epsilon$), such that for every $x\in x_{0}+\delta U$, $\bm{\mathrm{H}}(F(x),F(x_{0}))<\epsilon$.

• (b)

locally Lipschitz continuous, if and only if, for every $x_{0}\in\mathbb{R}^{d}$, there exists $\delta>0$ and $L>0$ (depending on $x_{0}$), such that for every $x,\ x^{\prime}\in x_{0}+\delta U$, $\bm{\mathrm{H}}(F(x),F(x^{\prime}))\leq L\|x-x^{\prime}\|$.

Before we proceed further we look at a certain form of locally Lipschitz continuous set-valued maps that arise later. The next lemma defines such maps and also states that the sum of two locally Lipschitz continuous set-valued maps is again a locally Lipschitz continuous set-valued map, a result needed later to obtain locally Lipschitz continuous single-valued parametrization of map $F$ in recursion $\eqref{rec}$.

Lemma 5.2

• (a)

If $f:\mathbb{R}^{d}\rightarrow\mathbb{R}$ is a locally Lipschitz continuous map and $C\in\mathcal{K}(\mathbb{R}^{d})$, then the set-valued map $F:\mathbb{R}^{d}\rightarrow\mathcal{K}(\mathbb{R}^{d})$, given by $F(x):=f(x)C$ for every $x\in\mathbb{R}^{d}$, is a locally Lipschitz continuous set-valued map.

• (b)

If for every $i\in\{1,2\}$, $F_{i}:\mathbb{R}^{d}\rightarrow\mathcal{K}(\mathbb{R}^{d})$ is a locally Lipschitz continuous set-valued map, then the set-valued map $F:\mathbb{R}^{d}\rightarrow\mathcal{K}(\mathbb{R}^{d})$, given by $F(x):=F_{1}(x)+F_{2}(x)$ for every $x\in\mathbb{R}^{d}$, is a locally Lipschitz continuous set-valued map.

• Proof

• (a)

Fix $x_{0}\in\mathbb{R}^{d}$ and let $r:=\sup_{c\in C}\|c\|$. Since $f$ is locally Lipschitz continuous, there exists $\delta^{f}_{x_{0}}>0$ and $L^{f}_{x_{0}}>0$ such that for every $x,x^{\prime}\in x_{0}+\delta^{f}_{x_{0}}U$, $|f(x)-f(x^{\prime})|\leq L^{f}_{x_{0}}\|x-x^{\prime}\|$. Let $x,x^{\prime}\in x_{0}+\delta^{f}_{x_{0}}U$. Then for any $c\in C$,

[TABLE]

Therefore for every $x,x^{\prime}\in x_{0}+\delta^{f}_{x_{0}}U$, for every $c\in C$, $f(x^{\prime})c-f(x)c\in rL^{f}_{x_{0}}\|x-x^{\prime}\|U$. Thus for every $x,x^{\prime}\in x_{0}+\delta^{f}_{x_{0}}U$, $F(x^{\prime})\subseteq F(x)+rL^{f}_{x_{0}}\|x-x^{\prime}\|U$, from which it follows that the set-valued map $F$ is locally Lipschitz continuous at $x_{0}$ with $\delta:=\delta^{f}_{x_{0}}$ and $L:=rL^{f}_{x_{0}}$. Since $x_{0}\in\mathbb{R}^{d}$ is arbitrary, the above argument gives us that $F$ is locally Lipschitz continuous at every $x_{0}$.

• (b)

Fix $x_{0}\in\mathbb{R}^{d}$. Since for every $i\in\{1,2\}$, $F_{i}$ are locally Lipschitz continuous, there exists $\delta_{i}>0$ and $L_{i}>0$ such that for every $x,x^{\prime}\in x_{0}+\delta_{i}U$, $F_{i}(x)\subseteq F_{i}(x^{\prime})+L_{i}\|x-x^{\prime}\|U$. Let $\delta:=\min\{\delta_{1},\delta_{2}\}$, $L:=L_{1}+L_{2}$ and $x,x^{\prime}\in x_{0}+\delta U$. For any $y\in F(x)$, there exists $y_{1}\in F_{1}(x)$ and $y_{2}\in F_{2}(x)$ such that $y=y_{1}+y_{2}$ . By our choice of $\delta$, we have $y^{\prime}_{1}\in F_{1}(x^{\prime})$, $y^{\prime}_{2}\in F_{2}(x^{\prime})$ and $u_{1},u_{2}\in U$, such that for every $i\in\{1,2\}$, $y_{i}=y^{\prime}_{i}+L_{i}\|x-x^{\prime}\|u_{i}$. Therefore,

[TABLE]

Clearly $y^{\prime}_{1}+y^{\prime}_{2}\in F(x^{\prime})$ and since $U$ is a convex subset of $\mathbb{R}^{d}$, $\frac{L_{1}u_{1}+L_{2}u_{2}}{L_{1}+L_{2}}\in U$. From $\eqref{tmp0}$ we get that, $F(x)\subseteq F(x^{\prime})+(L_{1}+L_{2})\|x-x^{\prime}\|U$, for every $x,x^{\prime}\in x_{0}+\delta U$. Therefore $F$ is locally Lipschitz continuous at $x_{0}$. Since $x_{0}$ is arbitrary, the above argument gives us that $F$ is locally Lipschitz continuous.

Consider a set-valued map $F$ satisfying assumption $(A1)$. A simple contradiction argument gives us that $F$ is u.s.c. It is not possible to represent such u.s.c. set-valued maps with a single-valued continuous map with an additional parameter. But instead one can approximate them from above as explained next. The first step is to embed the graph of the map $F$ in that of a sequence of continuous set-valued maps as stated in the lemma below. For the proof of the lemma below notions of a paracompact topological space, an open covering, its locally finite refinement and partition of unity subordinated to a locally finite covering are needed, which are summarized in Appendix A for easy reference.

Lemma 5.3

Let $F:\mathbb{R}^{d}\rightarrow\mathcal{K}(\mathbb{R}^{d})$ be a set-valued map satisfying $(A1)$. Then, there exists a sequence of continuous set-valued maps $\{F^{(l)}:\mathbb{R}^{d}\rightarrow\mathcal{K}(\mathbb{R}^{d})\}_{l\geq 1}$, such that for every $l\geq 1$,

• (a)

for every $x\in\mathbb{R}^{d}$, $F^{(l)}(x)$ is a non-empty, convex and compact subset of $\mathbb{R}^{d}$,

• (b)

for every $x\in\mathbb{R}^{d}$, $F(x)\subseteq F^{(l+1)}(x)\subseteq F^{(l)}(x)$,

• (c)

there exists $K^{(l)}>0$, such that for every $x\in\mathbb{R}^{d}$, $\sup_{y\in F^{(l)}(x)}\|y\|\leq K^{(l)}(1+\|x\|)$,

• (d)

$F^{(l)}$* is a locally Lipschitz continuous set valued map.*

Furthermore,

• (e)

for every $x\in\mathbb{R}^{d}$, $F(x)=\cap_{l\geq 1}F^{(l)}(x)$.

• Proof

For any $\epsilon>0$, for every $x_{0}\in\mathbb{R}^{d}$, let $B(\epsilon,x_{0}):=\left\{x:\ \parallel x-x_{0}\parallel<\epsilon\right\}$. Let $\left\{\epsilon_{l}:=\frac{1}{3^{l}}\right\}_{l\geq 1}$. Then for every $l\geq 1$, $\mathscr{C}_{l}:=\left\{B(\epsilon_{l},x_{0}):x_{0}\in\mathbb{R}^{d}\right\}$ is an open covering of $\mathbb{R}^{d}$. Since $\mathbb{R}^{d}$ is a metric space, it is paracompact (see [12, Ch. 0, Sec. 1, Thm. 1]). Therefore for every $l\geq 1$, there exists a locally finite open refinement of the covering $\mathscr{C}_{l}$ and let it be denoted by $\tilde{\mathscr{C}}_{l}:=\left\{C_{i}^{l}\right\}_{i\in I^{l}}$ where $I^{l}$ is an arbitrary index set. By [12, Ch. 0, Sec. 1, Thm. 2], there exists a locally Lipschitz continuous partition of unity, $\left\{\psi_{i}^{l}\right\}_{i\in I^{l}}$, subordinated to the covering $\tilde{\mathscr{C}}_{l}$. Therefore, for every $l\geq 1$, for every $i\in I^{l}$, there exists $x_{i}^{l}$, such that $\mathrm{support}(\psi_{i}^{l})\subseteq C_{i}^{l}\subseteq B(\epsilon_{l},x_{i}^{l})$. For every $l\geq 1$, for every $x\in\mathbb{R}^{d}$, let $I^{l}(x):=\left\{i\in I^{l}:\psi_{i}^{l}(x)>0\right\}$ and by definition of $\psi_{i}^{l}$, we have that $0<|I^{l}(x)|<\infty$ and $\sum_{i\in I^{l}(x)}\psi_{i}^{l}(x)=1$.

For every $l\geq 1$, define the set valued map $F^{(l)}:\mathbb{R}^{d}\rightarrow\left\{\text{subsets of }\mathbb{R}^{d}\right\}$, such that for every $x\in\mathbb{R}^{d}$, $F^{(l)}(x):=\sum_{i\in I^{l}(x)}\psi_{i}^{l}(x)A_{i}^{l}$, where $A_{i}^{l}:=\bar{co}\left(F\left(B\left(2\epsilon_{l},x_{i}^{l}\right)\right)\right)$.

The proofs of parts $(a),\ (b),\ (c)$ and $(e)$ of the lemma are exactly the same as that of [14, Lemma 3.2]. We shall provide a proof of part $(d)$ of the lemma above from which continuity of the set-valued maps $F^{(l)}$ follows.

• (d)

Fix $l\geq 1$ and $x\in\mathbb{R}^{d}$. Since $\tilde{\mathscr{C}}_{l}$ is a locally finite open covering of $\mathbb{R}^{d}$, there exists $\delta>0$ (depending on $x$), such that $I^{l}(x,\delta):=\left\{i\in I^{l}:\ B(x,\delta)\cap C^{l}_{i}\neq\emptyset\right\}$ is finite. Since $\{\psi_{i}^{l}\}_{i\in I^{l}}$ is a locally Lipschitz continuous partition of unity subordinated to the covering $\tilde{\mathscr{C}}_{l}$, we have that for every $i\in I^{l}$, $\mathrm{support}(\psi_{i}^{l})\subseteq C_{i}^{l}$. Therefore, for every $x^{\prime}\in B(x,\delta)$, $F^{(l)}(x^{\prime})=\sum_{i\in I^{l}(x,\delta)}\psi^{l}_{i}(x^{\prime})A_{i}^{l}$.

From the proof of part $(a)$ of this lemma we know that for every $i\in I^{l}$, $A_{i}^{l}$ is a compact and convex subset of $\mathbb{R}^{d}$. Therefore from Lemma 5.2$(a)$, we get that , for every $i\in I^{l}(x,\delta)$, the set-valued map given by $y\rightarrow\psi_{i}^{l}(y)A_{i}^{l}$ is locally Lipschitz continuous. Further since $|I^{l}(x,\delta)|<\infty$, from Lemma 5.2$(b)$, we get that the set-valued map given by $y\rightarrow\sum_{i\in I^{l}(x,\delta)}\psi^{l}(y)A_{i}^{l}$ is locally Lipschitz continuous. Since the set-valued map $y\rightarrow\sum_{i\in I^{l}(x,\delta)}\psi^{l}_{i}(y)A_{i}^{l}$ restricted to $B(x,\delta)$ is the same as $F^{(l)}$ on $B(x,\delta)$, we get that $F^{(l)}$ is locally Lipschitz continuous at $x$. Since $x$ is arbitrary, the above argument gives us that $F^{(l)}$ is a locally Lipschitz continuous set-valued map.

The continuous set-valued maps $F^{(l)}$ as obtained above can be now parametrized (that is represented with a single-valued continuous function with an additional parameter). Key to parametrization is a continuous selection procedure by which we mean a function $\sigma:\mathcal{K}(\mathbb{R}^{d})\rightarrow\mathbb{R}^{d}$ which is continuous and is such that for every $Y\in\mathcal{K}(\mathbb{R}^{d})$, $\sigma(Y)\in Y$. Since the maps $F^{(l)}$ are convex set-valued, it suffices to look for a selection procedure which is continuous restricted to the family of compact and convex subsets of $\mathbb{R}^{d}$. Further we want a selection procedure which would preserve the local Lipschitz continuity of the set-valued map $F^{(l)}$ in the parametrization as well. In order to accomplish this we shall use the Stiener selection procedure (for a definition see [15, Thm. 9.4.1]). The next lemma summarizes some properties of the Stiener selection procedure and an intersection lemma which form the central tools for parameterizing the set-valued maps $F^{(l)}$ (for a proof we refer the reader to [15, Thm. 9.4.1] and [15, Lemma 9.4.2]). Before we state the lemma we introduce some notation needed. Let $\mathcal{K}_{c}(\mathbb{R}^{d})$ denote the family of all non-empty compact and convex subsets of $\mathbb{R}^{d}$. For any set $Y\subseteq\mathbb{R}^{d}$ and for any $x\in\mathbb{R}^{d}$, define $\bm{d}(x,Y):=\inf_{y\in Y}\|x-y\|$.

Lemma 5.4

• (a)

There exists a function $\sigma:\mathcal{K}_{c}(\mathbb{R}^{d})\rightarrow\mathbb{R}^{d}$, such that for every $Y,Y_{1},Y_{2}\in\mathcal{K}_{c}(\mathbb{R}^{d})$,

[TABLE]

• (b)

The map $\Pi:\mathcal{K}_{c}(\mathbb{R}^{d})\times\mathbb{R}^{d}\rightarrow\mathcal{K}_{c}(\mathbb{R}^{d})$, defined such that for every $Y\in\mathcal{K}_{c}(\mathbb{R}^{d})$ and $x\in\mathbb{R}^{d}$, $\Pi(Y,u):=Y\cap\left(x+2\bm{d}(x,Y)U\right)$, is such that for every $Y_{1},Y_{2}\in\mathcal{K}_{c}(\mathbb{R}^{d})$ and for every $x_{1},x_{2}\in\mathbb{R}^{d}$,

[TABLE]

We now use the results stated in the above lemma to parametrize the set-valued maps $F^{(l)}$.

Lemma 5.5

Let $\{F^{(l)}\}_{l\geq 1}$ be as in Lemma 5.3. For every $l\geq 1$, there exists a continuous function $f^{(l)}:\mathbb{R}^{d}\times U\rightarrow\mathbb{R}^{d}$ such that,

• (a)

for every $x\in\mathbb{R}^{d}$, $f^{(l)}(x,U)=F^{(l)}(x)$ where $f^{(l)}(x,U):=\left\{f^{(l)}(x,u):\ u\in U\right\}$.

• (b)

for $K^{(l)}>0$ as in Lemma 5.3, for every $(x,u)\in\mathbb{R}^{d}\times U$, $\|f^{(l)}(x,u)\|\leq K^{(l)}(1+\|x\|)$.

• (c)

for every $x_{0}\in\mathbb{R}^{d}$, there exists $\delta^{(l)}>0$ and $L^{(l)}>0$ (depending on $x_{0}$), such that for every $x,x^{\prime}\in x_{0}+\delta^{(l)}U$, for every $u\in U$,

[TABLE]

• Proof

Fix $l\geq 1$. Let the map $f^{(l)}:\mathbb{R}^{d}\times U\rightarrow\mathbb{R}^{d}$ be defined such that, for every $(x,u)\in\mathbb{R}^{d}\times U$,

[TABLE]

where $\sigma$ and $\Pi$ are as in Lemma 5.4.

• (a)

By definition of $f^{(l)}$, $\sigma$ and $\Pi$, for every $(x,u)\in\mathbb{R}^{d}\times U$, we have that,

[TABLE]

Therefore, for every $x\in\mathbb{R}^{d}$, $f^{(l)}(x,U)\subseteq F^{(l)}(x)$. By Lemma 5.3$(c)$, we know that for every $x\in\mathbb{R}^{d}$, $\sup_{y\in F^{(l)}(x)}\|y\|\leq K^{(l)}(1+\|x\|)$. Thus for every $x\in\mathbb{R}^{d}$, for any $y\in F^{(l)}(x)$, there exists $u\in U$, such that $y=K^{(l)}(1+\|x\|)u$. For such a $u\in U$, by definition of $\Pi$, we have that $\Pi(F^{(l)}(x),K^{(l)}(1+\|x\|)u)=y$ and hence $f^{(l)}(x,u)=\sigma\left(\Pi\left(F^{(l)}(x),K^{(l)}(1+\|x\|)u\right)\right)=y$. Therefore for every $x\in\mathbb{R}^{d}$, $F^{(l)}(x)\subseteq f^{(l)}(x,U)$ from which it follows that $f^{(l)}(x,U)=F^{(l)}(x)$, for every $x\in\mathbb{R}^{d}$.

• (b)

Follows from part $(a)$ of this lemma and Lemma 5.3$(c)$.

• (c)

Fix $x_{0}\in\mathbb{R}^{d}$. Since $F^{(l)}$ is a locally Lipschitz continuous set-valued map (see Lemma 5.3$(d)$), we obtain $\delta_{F^{(l)}}>0$ and $L_{F^{(l)}}>0$ (depending on $x_{0}$) such that for every $x,x^{\prime}\in x_{0}+\delta_{F^{(l)}}U$, $\bm{\mathrm{H}}(F^{(l)}(x),F^{(l)}(x^{\prime}))\leq L_{F^{(l)}}\|x-x^{\prime}\|$. Set $\delta^{(l)}:=\delta_{F^{(l)}}$ and $L^{(l)}:=5d(L_{F^{(l)}}+K^{(l)})$. Then, for any $x,x^{\prime}\in x_{0}+\delta^{(l)}U$, for every $u\in U$,

[TABLE]

where, $\eqref{tmp1}$ follows from Lemma 5.4$(a)$, $\eqref{tmp2}$ follows from Lemma 5.4$(b)$ and $\eqref{tmp3}$ follows from our choice of $\delta^{(l)}$ and local Lipschitz continuity of $F^{(l)}$.

The set-valued map in recursion $\eqref{rec}$ can be replaced with the parametrization obtained in the lemma above as explained below.

• (1)

For every $l\geq 1$, by Lemma 5.3$(b)$, we know that for every $x\in\mathbb{R}^{d}$, $F(x)\subseteq F^{(l)}(x)$. Therefore for every $l\geq 1$, for every $n\geq 0$,

[TABLE]

• (2)

For every $l\geq 1$, by Lemma 5.5$(a)$, we know that for every $x\in\mathbb{R}^{d}$, $F^{(l)}(x)=f^{(l)}(x,U)$. It can now be shown that for every $n\geq 0$, there exists a $U$-valued random variable on $\Omega$, say $U^{(l)}_{n}$, such that for every $\omega\in\Omega$, for every $n\geq 0$,

[TABLE]

(for a proof see [14, Lemma 6.1]).

5.2 Solutions of the mean field and their approximation

In this section, we shall approximate the solutions of mean field (that is DI $\eqref{mfld}$) with the solutions of DI given by,

[TABLE]

for some $l\geq 1$. In order to accomplish this we need some notations which are introduced next.

For every $T>0$ and for every $x\in\mathbb{R}^{d}$, let $S(T,x)$ denote the set of solutions of DI $\eqref{mfld}$ on $[0,T]$. Formally,

[TABLE]

Since $F$ is a Marchaud map, we have that for every $T>0$ and for every $x\in\mathbb{R}^{d}$, $S(T,x)\neq\emptyset$. Similarly for every $l\geq 1$, for every $T>0$ and for every $x\in\mathbb{R}^{d}$, let $S^{(l)}(T,x)$ denote the set of solutions of DI $\eqref{amfld}$ on $[0,T]$. Formally,

[TABLE]

From Lemma 5.3, we know that for every $l\geq 1$, $F^{(l)}$ is a Marchaud map and hence for every $T>0$ and for every $x\in\mathbb{R}^{d}$, $S^{(l)}(T,x)\neq\emptyset$.

For any $Y\subseteq\mathbb{R}^{d}$, for any $T>0$, define $S(T,Y):=\cup_{y\in Y}S(T,y)$. Similarly, for every $l\geq 1$, $S^{(l)}(T,Y):=\cup_{y\in Y}S^{(l)}(T,y)$.

The next lemma summarizes some important relationships between the solutions of DI $\eqref{mfld}$ and those of DI $\eqref{amfld}$ needed later. It also states that for large enough $l\geq 1$, the solutions of DI $\eqref{amfld}$ are within an $\epsilon$-neighborhood of the solutions of DI $\eqref{mfld}$ for every initial condition lying in a compact subset of $\mathbb{R}^{d}$.

Lemma 5.6

For every $T>0$,

• (a)

for every $l\geq 1$, for every $x\in\mathbb{R}^{d}$, $S(T,x)\subseteq S^{(l+1)}(T,x)\subseteq S^{(l)}(T,x)$.

• (b)

for every $x\in\mathbb{R}^{d}$, $S(T,x)=\cap_{l\geq 1}S^{(l)}(T,x)$.

• (c)

for any $Y\subseteq\mathbb{R}^{d}$, $S(T,Y)=\cap_{l\geq 1}S^{(l)}(T,Y)$.

• (d)

for every $Y\subseteq\mathbb{R}^{d}$ compact, $S(T,Y)$ is a compact subset of $\mathcal{C}([0,T],\mathbb{R}^{d})$ (the vector space of $\mathbb{R}^{d}$-valued continuous functions on $[0,T]$).

• (e)

for every $Y\subseteq\mathbb{R}^{d}$ compact, for every $l\geq 1$, $S^{(l)}(T,Y)$ is a compact subset of $\mathcal{C}([0,T],\mathbb{R}^{d})$.

• (f)

for every $Y\subseteq\mathbb{R}^{d}$ compact, for every $\epsilon>0$, there exists $l^{\prime}\geq 1$, such that for every $l\geq l^{\prime}$, for every $\bm{\mathrm{x}}^{(l)}\in S^{(l)}(T,Y)$, there exists $\bm{\mathrm{x}}\in S(T,Y)$, such that $\sup_{t\in[0,T]}\|\bm{\mathrm{x}}(t)-\bm{\mathrm{x}}^{(l)}(t)\|<\epsilon$.

• Proof

Fix $T>0$.

• (a)

Fix $l\geq 1$ and $x\in\mathbb{R}^{d}$. Let $\bm{\mathrm{x}}\in S(T,x)$. Then we have that $\bm{\mathrm{x}}$ is absolutely continuous with $\bm{\mathrm{x}}(0)=x$ and for $a.e.\ t\in[0,T]$, $\frac{d\bm{\mathrm{x}}(t)}{dt}\in F(\bm{\mathrm{x}}(t))$. By Lemma 5.3$(b)$, we know that for every $t\in[0,T]$, $F(\bm{\mathrm{x}}(t))\subseteq F^{(l+1)}(\bm{\mathrm{x}}(t))$. Therefore for $a.e.\ t\in[0,T]$, $\frac{d\bm{\mathrm{x}}(t)}{dt}\in F^{(l+1)}(\bm{\mathrm{x}}(t))$, from which we get that $\bm{\mathrm{x}}\in S^{(l+1)}(T,x)$. Hence $S(T,x)\subseteq S^{(l+1)}(T,x)$. Using the fact that for every $x^{\prime}\in\mathbb{R}^{d}$, $F^{(l+1)}(x^{\prime})\subseteq F^{(l)}(x^{\prime})$ (see Lemma 5.3$(b)$), a similar argument gives us that $S^{(l+1)}(T,x)\subseteq S^{(l)}(T,x)$.

• (b)

Fix $x\in\mathbb{R}^{d}$. From part $(a)$ of this lemma we have that $S(T,x)\subseteq\cap_{l\geq 1}S^{(l)}(T,x)$. Let $\bm{\mathrm{x}}\in\cap_{l\geq 1}S^{(l)}(T,x)$. Then $\bm{\mathrm{x}}$ is absolutely continuous with $\bm{\mathrm{x}}(0)=x$ and for every $l\geq 1$, for $a.e.\ t\in[0,T]$, $\frac{d\bm{\mathrm{x}}(t)}{dt}\in F^{(l)}(\bm{\mathrm{x}}(t))$. Thus for $a.e.\ t\in[0,T]$, for every $l\geq 1$, $\frac{d\bm{\mathrm{x}}(t)}{dt}\in F^{(l)}(\bm{\mathrm{x}}(t))$. Hence for $a.e\ t\in[0,T]$, $\frac{d\bm{\mathrm{x}}(t)}{dt}\in\cap_{l\geq 1}F^{(l)}(\bm{\mathrm{x}}(t))=F(\bm{\mathrm{x}}(t))$, where the equality follows from Lemma 5.3$(e)$. Therefore $\bm{\mathrm{x}}\in S(T,x)$, from which we get that $\cap_{l\geq 1}S^{(l)}(T,x)\subseteq S(T,x)$.

• (c)

Follows from part $(a)$ and $(b)$ of this lemma.

• (d) & (e)

Follows from [2, Lemma 3.1].

• (f)

Suppose not. Then there exists $Y\subseteq\mathbb{R}^{d}$ compact and $\epsilon>0$, such that for every $l^{\prime}\geq 1$, there exists $l\geq l^{\prime}$ and $\bm{\mathrm{x}}^{(l)}\in S^{(l)}(T,Y)$, such that $\bm{d}(\bm{\mathrm{x}}^{(l)},S(T,Y))\geq\epsilon$, where $\bm{d}(\bm{\mathrm{x}}^{(l)},S(T,Y)):=\inf_{\bm{\mathrm{x}}\in S(T,Y)}\sup_{t\in[0,T]}\|\bm{\mathrm{x}}^{(l)}(t)-\bm{\mathrm{x}}(t)\|$. Thus we can obtain a sequence of solutions, say $\{\bm{\mathrm{x}}^{(l_{k})}\}_{k\geq 1}$, such that for every $k\geq 1$, $1\leq l_{k}<l_{k+1}$ and $\bm{\mathrm{x}}^{(l_{k})}\in S^{(l_{k})}(T,Y)$ with $\bm{d}(\bm{\mathrm{x}}^{(l_{k})},S(T,Y))\geq\epsilon$. From part $(a)$ of this lemma, we have that for every $k\geq 1$, $S^{(l_{k})}(T,Y)\subseteq S^{(1)}(T,Y)$ and hence $\{\bm{\mathrm{x}}^{(l_{k})}\}_{k\geq 1}\subseteq S^{(1)}(T,Y)$. Since $Y\subseteq\mathbb{R}^{d}$ is compact, by part $(e)$ of this lemma we know that $S^{(1)}(T,Y)$ is a compact subset of $\mathcal{C}([0,T],\mathbb{R}^{d})$. Thus there exists a subsequence of $\{\bm{\mathrm{x}}^{(l_{k})}\}_{k\geq 1}$, say $\{\bm{\mathrm{x}}^{(l_{k_{j}})}\}_{j\geq 1}$ such that $\bm{\mathrm{x}}^{(l_{k_{j}})}\to\bm{\mathrm{x}}^{*}$ as $j\to\infty$ in $\mathcal{C}([0,T],\mathbb{R}^{d})$ and $\bm{\mathrm{x}}^{*}\in S^{(1)}(T,Y)$. Since for every $j\geq 1$, $\bm{d}(\bm{\mathrm{x}}^{(l_{k_{j}})},S(T,Y))\geq\epsilon$, we get that $\bm{d}(\bm{\mathrm{x}}^{*},S(T,Y))\geq\epsilon$ and hence $\bm{\mathrm{x}}^{*}\notin S(T,Y)$. From part $(a)$ of this lemma, we get that for every $l\geq 1$, for $J:=\min\{j\geq 1:l_{k_{j}}\geq l\}$, $\{\bm{\mathrm{x}}^{(l_{k_{j}})}\}_{j\geq J}\subseteq S^{(l)}(T,Y)$. Further by part $(e)$ of this lemma we have that for every $l\geq 1$, $S^{(l)}(T,Y)$ is a compact subset of $\mathcal{C}([0,T],\mathbb{R}^{d})$. Thus for every $l\geq 1$, $\bm{\mathrm{x}}^{*}\in S^{(l)}(T,Y)$ and hence $\bm{\mathrm{x}}^{*}\in\cap_{l\geq 1}S^{(l)}(T,Y)=S(T,Y)$ (see part $(c)$ of this lemma). This leads to a contradiction.

The part $(f)$ of the above lemma provides the necessary approximation result. Further since the set-valued maps $F^{(l)}$ admit a single-valued parametrization ($f^{(l)}$ as in Lemma 5.5), a solution of DI $\eqref{amfld}$ can be viewed as a solution of the ordinary differential equation (o.d.e.) given by,

[TABLE]

for some $u:[0,\infty)\rightarrow U$ measurable and vice versa. The lemma below summarizes some useful results on the solutions of o.d.e. $\eqref{pmfld}$ and its vector field.

Lemma 5.7

For every $l\geq 1$,

• (a)

for every $T>0$, for any $u:[0,T]\rightarrow U$ measurable, for every initial condition, the set of solutions of o.d.e. $\eqref{pmfld}$ is non-empty. That is, for every $x_{0}\in\mathbb{R}^{d}$, there exists $\bm{\mathrm{x}}:[0,T]\rightarrow\mathbb{R}^{d}$ such that, $\bm{\mathrm{x}}$ is absolutely continuous, $\bm{\mathrm{x}}(0)=x_{0}$ and for $a.e.\ t\in[0,T]$, $\frac{d\bm{\mathrm{x}}(t)}{dt}=f^{(l)}(\bm{\mathrm{x}}(t),u(t))$.

• (b)

for every $T>0$, for every $Y\subseteq\mathbb{R}^{d}$ compact, there exists $C_{1}(Y,T,l)>0$, such that for every $u:[0,T]\rightarrow U$ measurable, every solution of o.d.e. $\eqref{pmfld}$ with initial condition in $Y$, say $\bm{\mathrm{x}}:[0,T]\rightarrow\mathbb{R}^{d}$, satisfies,

[TABLE]

• (c)

for any $Y\subseteq\mathbb{R}^{d}$ compact, there exists $L(Y,l)>0$, such that for every $T>0$, for every $u:[0,T]\rightarrow U$, the map $h:Y\times[0,T]\rightarrow\mathbb{R}^{d}$, given by $h(x,t):=f^{(l)}(x,u(t))$ for every $(x,t)\in Y\times[0,T]$, satisfies,

[TABLE]

for every $x,\ x^{\prime}\in Y$ and for every $t\in[0,T]$.

• (d)

for every $T>0$, for every $u:[0,T]\rightarrow U$, for every initial condition, o.d.e. $\eqref{pmfld}$ admits a unique solution.

• Proof

Fix $l\geq 1$.

• (a)

Fix $T>0$ and $u:[0,T]\rightarrow U$ measurable. The proof of this part is a direct application of [16, Thm. 3.8]. We show here that the sufficient conditions required to apply the said theorem are satisfied by the vector field of the o.d.e. $\eqref{pmfld}$. First, we show that $f^{(l)}(\cdot,u(\cdot))$ is a Caratheodary function (see [16, Defn. 3.2]). By Lemma 5.5, it is clear that for every $t\in[0,T]$, the map $x\rightarrow f^{(l)}(x,u(t))$ is continuous and for every $x\in\mathbb{R}^{d}$, the map $t\rightarrow f^{(l)}(x,u(t))$ is measurable. Further by Lemma 5.5$(b)$, we have that for any $c>0$, for every $x\in\mathbb{R}^{d}$ with $\|x\|\leq c$, for every $t\in[0,T]$, $\|f^{(l)}(x,t)\|\leq K^{(l)}(1+c)$. Thus $f^{(l)}(\cdot,u(\cdot))$ is a Caratheodary function. Final condition to verify is on the rate of growth of solutions. By Lemma 5.5$(b)$, $\|f^{(l)}(x,u(t))\|\leq\psi(\|x\|):=K^{(l)}(1+\|x\|)$. The function $\psi:[0,\infty)\rightarrow[0,\infty)$, is clearly positive everywhere and the function $\frac{1}{\psi}$ is locally integrable on $[0,\infty)$. A simple argument gives us that for every $r>0$, the integral $\int_{r}^{\infty}\frac{d\tilde{r}}{\psi(\tilde{r})}$ can be lower bounded by the tail of $\frac{1}{K^{(l)}}\sum_{n=1}^{\infty}\frac{1}{n}$. Hence for every $r>0$, $\int_{r}^{\infty}\frac{d\tilde{r}}{\psi(\tilde{r})}=\infty$. Now [16, Thm. 3.8] can be applied to obtain the required result.

• (b)

Fix $T>0$ and $Y\subseteq\mathbb{R}^{d}$ compact. Since $Y$ is compact, there exists $r>0$ such that $\sup_{y\in Y}\|y\|\leq r$. Set $C_{1}(Y,T,l):=(r+K^{(l)}T)e^{K^{(l)}T}$, where $K^{(l)}>0$ is as in Lemma 5.5$(b)$. For some $u:[0,T]\rightarrow U$ measurable and for some $x_{0}\in Y$, let $\bm{\mathrm{x}}:[0,T]\rightarrow\mathbb{R}^{d}$ be a solution of o.d.e. $\eqref{pmfld}$ with initial condition $x_{0}$. Then, for every $t\in[0,T]$, $\bm{\mathrm{x}}(t)=x_{0}+\int_{0}^{t}f^{(l)}(x(s),u(s))ds$ and hence for every $t\in[0,T]$

[TABLE]

where, $\eqref{tmp4}$ follows from the fact that $x_{0}\in Y$ and Lemma 5.5$(b)$. The required bound follows from $\eqref{tmp4}$ and Gronwall’s result (see [3, Sec. 11.2.1, Lemma 6]).

• (c)

Fix $Y\subseteq\mathbb{R}^{d}$ compact. It is enough to show that there exists $L(Y,l)>0$, such that for every $y_{1},\ y_{2}\in Y$, $\sup_{u\in U}\|f^{(l)}(y_{1},u)-f^{(l)}(y_{2},u)\|\leq L(Y,l)\|y_{1}-y_{2}\|$. From Lemma 5.5$(c)$, we know that for every $x_{0}\in Y$, there exists $\delta(x_{0},l)>0$ and $L(x_{0},l)>0$, such that for every $x,x^{\prime}\in x_{0}+\delta(x_{0},l)U$, for every $u\in U$, $\|f^{(l)}(y_{1},u)-f^{(l)}(y_{2},u)\|\leq L(Y,l)\|y_{1}-y_{2}\|$. Let $\mathcal{G}:=\{x_{0}+\frac{\delta(x_{0},l)}{2}\mathring{U}:x_{0}\in Y\}$, where, $\mathring{U}$ denotes the interior of $U$. Since $Y$ is compact and $\mathcal{G}$ is an open cover of $Y$, there exists $\{x_{1},x_{2},\dots,x_{k}\}\subseteq Y$, such that $Y\subseteq\cup_{i=1}^{k}(x_{i}+\frac{\delta(x_{i},l)}{2}\mathring{U})$. Set $\delta(Y,l):=\min_{1\leq i\leq k}\frac{\delta(x_{i},l)}{2}$ and $L_{0}(Y,l):=\max_{1\leq i\leq k}L(x_{i},l)$.

Let $y_{1},y_{2}\in(Y\times Y)\cap\{(y_{1},y_{2}):\|y_{1}-y_{2}\|<\delta(Y,l)\}$. Then we know that there exists $i\in\{1,\dots,k\}$, such that $y_{1}\in x_{i}+\frac{\delta(x_{i},l)}{2}\mathring{U}$. Further since $\|y_{1}-y_{2}\|<\delta(Y,l)\leq\frac{\delta(x_{i},l)}{2}$, we have that $y_{2}\in x_{i}+\delta(x_{i},l)\mathring{U}$. Therefore $y_{1},y_{2}\in x_{i}+\delta(x_{i},l)U$ and hence, for every $u\in U$, $\|f^{(l)}(y_{1},u)-f^{(l)}(y_{2},u)\|\leq L(x_{i},l)\|y_{1}-y_{2}\|\leq L_{0}(Y,l)\|y_{1}-y_{2}\|$. Thus for every $y_{1},y_{2}\in(Y\times Y)\cap\{(y_{1},y_{2}):\|y_{1}-y_{2}\|<\delta(Y,l)\}$, $\sup_{u\in U}\|f^{(l)}(y_{1},u)-f^{(l)}(y_{2},u)\|\leq L_{0}(Y,l)\|y_{1}-y_{2}\|$.

Let $E:=(Y\times Y)\cap\{(y_{1},y_{2}):\|y_{1}-y_{2}\|\geq\delta(Y,l)\}$. By Lemma 5.5$(b)$, the map $(y_{1},y_{2})\in Y\times Y\rightarrow\sup_{u\in U}\|f^{(l)}(y_{1},u)-f^{(l)}(y_{2},u)\|$ is well defined. Further using the fact that for every $(y_{1},y_{2}),(y_{1}^{\prime},y_{2}^{\prime})\in Y\times Y$, $|\sup_{u\in U}\|f^{(l)}(y_{1},u)-f^{(l)}(y_{2},u)\|-\sup_{u\in U}\|f^{(l)}(y_{1}^{\prime},u)-f^{(l)}(y_{2}^{\prime},u)\||\leq\sup_{u\in U}\|f^{(l)}(y_{1},u)-f^{(l)}(y_{1}^{\prime},u)\|+\sup_{u\in U}\|f^{(l)}(y_{2},u)-f^{(l)}(y_{2}^{\prime},u)\|$ and Lemma 5.5$(c)$, we have that the map $(y_{1},y_{2})\rightarrow\sup_{u\in U}\|f^{(l)}(y_{1},u)-f^{(l)}(y_{2},u)\|$ is continuous. Thus the map $(y_{1},y_{2})\in E\rightarrow\frac{\sup_{u\in U}\|f^{(l)}(y_{1},u)-f^{(l)}(y_{2},u)\|}{\|y_{1}-y_{2}\|}$ is a continuous function on a compact set $E$ and hence achieves a maximum, say $L_{1}(Y,l)\geq 0$. Therefore for every $(y_{1},y_{2})\in(Y\times Y)\cap\{(y_{1},y_{2}):\|y_{1}-y_{2}\|\geq\delta(Y,l)\}$, $\sup_{u\in U}\|f^{(l)}(y_{1},u)-f^{(l)}(y_{2},u)\|\leq L_{1}(y,l)\|y_{1}-y_{2}\|$.

Thus from the arguments in the two preceding paragraphs we have that there exists $L(Y,l):=\max\{L_{0}(Y,l),L_{1}(Y,l)\}$, such that, for every $y_{1},y_{2}\in Y$, $\sup_{u\in U}\|f^{(l)}(y_{1},u)-f^{(l)}(y_{2},u)\|\leq L(Y,l)\|y_{1}-y_{2}\|$.

• (d)

Using parts $(b)$ and $(c)$ of this lemma, the proof of uniqueness follows from arguments similar to that of [16, Thm. 3.4].

5.3 Bounding procedure

In this section we show that the lower bound on the probability of the event that the iterates converge to an attracting set given that after a large number of iterations the iterates lies in a neighborhood of it depends mainly on the additive noise terms.

In order to accomplish this we first define some terms which are a measure of the distance of the linearly interpolated trajectory of recursion $\eqref{rec}$, that is $\bar{X}$ (see $\eqref{lit})$ to the solutions of the DI $\eqref{mfld}$ over a $T>0$ length time interval, among others. Recall from section 3.2 that $\mathcal{O}^{\prime}\subseteq\mathbb{R}^{d}$, is an open neighborhood of the attracting set $A$ (as in assumption $(A4)$) with compact closure, such that $A\subseteq\mathcal{O}^{\prime}\subseteq\bar{\mathcal{O}^{\prime}}\subseteq\mathcal{O}$, where $\mathcal{O}$ denotes the fundamental neighborhood of $A$. Thus we can find an $\epsilon_{0}>0$, such that $N^{\epsilon_{0}}(\bar{\mathcal{O}^{\prime}})\subseteq\mathcal{O}$ and $N^{2\epsilon_{0}}(A)\subseteq\mathcal{O}^{\prime}$, where for any $\epsilon>0$, $N^{\epsilon}(\cdot)$ denotes the $\epsilon$-neighborhood of a set. Further, since $A$ is an attracting set for the flow of DI $\eqref{mfld}$, for $\epsilon_{0}>0$ as obtained above, there exists $T_{A}>0$, such that for every $x\in\mathcal{O}$, for every $t\geq T_{A}$, $\Phi(t,x)\in N^{\epsilon_{0}}(A$. Throughout the rest of this paper $\epsilon_{0}$ and $T_{A}$ will denote the constants as obtained above.

For every $T>0$, for every $n\geq 0$,

1. Definition 1

: let $\tau(n,T):=\min\{k\geq n:t(k)\geq t(n)+T\}$, where $t(n)$, for every $n\geq 0$ are as defined in section 3.1. That is $\tau(n,T)$ denotes the first iterate such that, at least time $T$ has elapsed since the $n^{th}$ iteration. Further the time elapsed from iteration $n$ to iteration $\tau(n,T)$, be denoted by $\Delta(n,T)$, that is $\Delta(n,T):=t(\tau(n,T))-t(n)$. Then by the choice of our step sizes we have that $T\leq\Delta(n,T)\leq T+1$. 2. Definition 2

: for every $\omega\in\Omega$, $\rho(\omega,n,T):=\inf_{\bm{\mathrm{x}}\in S(T,\bar{\mathcal{O}^{\prime}})}\sup_{t\in[0,T]}\|\bar{X}(\omega,t+t(n))-\bm{\mathrm{x}}(t)\|$, where $S(T,\bar{\mathcal{O}^{\prime}})$ denotes the set of solutions of DI $\eqref{mfld}$ as defined in equation $\eqref{ss0}$. 3. Definition 3

: for every $\omega\in\Omega$, for every $l\geq 1$, let $\bar{\bm{\mathrm{x}}}^{(l)}(\cdot;n,T,\omega):[0,T]\rightarrow\mathbb{R}^{d}$ denote the unique solution of the o.d.e.

[TABLE]

with initial condition $\bar{\bm{\mathrm{x}}}^{(l)}(0;n,T,\omega)=X_{n}(\omega)$, where $u(\cdot;n,T,\omega):[0,T]\rightarrow U$ is defined such that, for every $t\in[0,T]$, $u(t;n,T,\omega):=U_{k}^{(l)}(\omega)$, where $U_{k}^{(l)}$ is as in equation $\eqref{rec1}$ and $k$ is such that $t+t(n)\in[t(k),t(k+1))$ (for a proof of existence and uniqueness of solutions to o.d.e. $\eqref{pmfld1}$, see Lemma 5.7). It is easy to see that for every $l\geq 1$, $\bar{\bm{\mathrm{x}}}^{(l)}(\cdot;n,T,\omega)\in S^{(l)}(T,X_{n}(\omega))$, where $S^{(l)}(T,X_{n}(\omega))$ denotes the set of solutions of DI $\eqref{amfld}$, as defined in $\eqref{ss1}$. 4. Definition 4

: for every $\omega\in\Omega$, for every $l\geq 1$, $\rho^{(l)}_{1}(\omega,n,T):=\sup_{t\in[0,T]}\|\bar{X}(\omega,t+t(n))-\bar{\bm{\mathrm{x}}}^{(l)}(t;n,T,\omega)\|$ and $\rho^{(l)}_{2}(\omega,n,T):=\inf_{\bm{\mathrm{x}}\in S(T,\bar{\mathcal{O}^{\prime}})}\sup_{t\in[0,T]}\|\bar{\bm{\mathrm{x}}}^{(l)}(t;n,T,\omega)-\bm{\mathrm{x}}(t)\|$. 5. Definition 5

: for any $T_{u}\geq T_{A}$, for any $n_{0}\geq 0$, let $\{n_{m}\}_{m\geq 1}$ denote a subsequence of natural numbers defined such that for every $m\geq 0$, $T_{A}\leq T_{m}:=t(n_{m+1})-t(n_{m})\leq T_{u}$.

Now we collect sample paths of interest using the quantities $\rho$, $\rho_{1}^{(l)}$ and $\rho_{2}^{(l)}$. The next lemma summarizes results in this regard.

Lemma 5.8

For every $T_{u}\geq T_{A}$, for every $n_{0}\geq 0$, for every $l\geq 1$, for every event $E\in\mathscr{F}_{n_{0}}$, such that $E\subseteq\{\omega:X_{n_{0}}(\omega)\in\mathcal{O}^{\prime}\}$, for every $\{n_{m}\}_{m\geq 1}$ as in Definition 5,

• (a)

*for every $M\geq 0$,

---

[TABLE]

• (b)

[TABLE]

where, $\{T_{m}\}_{m\geq 0}$ is as in Definition 5.

• Proof

Fix $n_{0}\geq 0$, $l\geq 1$ and $E\in\mathscr{F}_{n_{0}}$, such that $E\subseteq\{\omega\in\Omega:X_{n_{0}}(\omega)\in\mathcal{O}^{\prime}\}$.

• (a)

For every $m\geq 0$, for every $\omega\in\Omega$, from Definition 2 and Definition 4, it is clear that,

[TABLE]

from which we get that for every $m\geq 0$,

[TABLE]

Therefore,

[TABLE]

The proof of the second inclusion follows from induction. Fix $M=0$ and $\omega\in E\cap\{\omega\in\Omega:\rho(\omega,n_{0},T_{0})<\epsilon_{0}\}$. Then $X_{n_{0}}(\omega)\in\mathcal{O}^{\prime}$. Since $T_{0}\geq T_{A}$, we have that for every $\bm{\mathrm{x}}\in S(T_{0},\bar{\mathcal{O}^{\prime}})$, $\bm{\mathrm{x}}(T_{0})\in N^{\epsilon_{0}}(A)$. Further, since $\rho(\omega,n,T_{0})<\epsilon_{0}$ and by Lemma 5.6$(d)$, we get that there exists $\bm{\mathrm{x}}\in S(T_{0},\bar{\mathcal{O}^{\prime}})$, such that $\|\bar{X}(\omega,t(n_{1}))-\bm{\mathrm{x}}(T_{0})\|=\|X_{n_{1}}(\omega)-\bm{\mathrm{x}}(T_{0})\|<\epsilon_{0}$ and hence $X_{n_{1}}(\omega)\in N^{2\epsilon_{0}}(A)\subseteq\mathcal{O}^{\prime}$. Therefore $\omega\in\{\omega\in\Omega:X_{n_{1}}(\omega)\in\mathcal{O}^{\prime}\}$. Thus the inclusion is true for $M=0$. Suppose the inclusion is true for some $M>0$. Let $\omega\in E\cap\left(\cap_{m=0}^{M+1}\left\{\omega\in\Omega:\rho(\omega,n_{m},T_{m})<\epsilon_{0}\right\}\right)$. Since the inclusion is true for $M$, we have that $X_{n_{M+1}}(\omega)\in\mathcal{O}^{\prime}$. Now by arguments exactly same as those for the base case (that is for $M=0$) we get that $X_{n_{M+2}}(\omega)\in\mathcal{O}^{\prime}$. Therefore the inclusion is true for $M+1$.

• (b)

The first inequality follows from part $(a)$ of this lemma. We shall provide a proof of the second inequality. Let $\omega\in E\cap\left(\cap_{m\geq 0}\left\{\omega\in\Omega:\rho(\omega,n_{m},T_{m})<\epsilon_{0}\right\}\right)$. Then by part $(a)$ of this lemma we have that for every $m\geq 0$, $X_{n_{m}}(\omega)\in\mathcal{O}^{\prime}$. Since $\bar{\mathcal{O}^{\prime}}$ is compact, by Lemma 5.6$(d)$, we have that, $S(T_{u},\bar{\mathcal{O}^{\prime}})$ is a compact subset of $\mathcal{C}([0,T_{u}],\mathbb{R}^{d})$, and hence there exists $C(\bar{\mathcal{O}^{\prime}},T_{m})>0$ such that, $\sup_{\bm{\mathrm{x}}\in S(T_{u},\bar{\mathcal{O}^{\prime}})}\sup_{t\in[0,T_{u}]}\|\bm{\mathrm{x}}(t)\|\leq C(\bar{\mathcal{O}^{\prime}},T_{m})$. Further since for every $m\geq 0$, $T_{m}\leq T_{u}$, we get that $\sup_{\bm{\mathrm{x}}\in S(T_{m},\bar{\mathcal{O}^{\prime}})}\sup_{t\in[0,T_{m}]}\|\bm{\mathrm{x}}(t)\|\leq\sup_{\bm{\mathrm{x}}\in S(T_{u},\bar{\mathcal{O}^{\prime}})}\sup_{t\in[0,T_{u}]}\|\bm{\mathrm{x}}(t)\|\leq C(\bar{\mathcal{O}^{\prime}},T_{m})$. By our choice of $\omega$, we have that for every $m\geq 0$, $\rho(\omega,n_{m},T_{m})<\epsilon_{0}$ and by Definition 2, we get that for every $m\geq 0$, $\sup_{t\in[0,T_{m}]}\|\bar{X}(\omega,t+t(n_{m}))\|\leq C(\bar{\mathcal{O}^{\prime}},T_{m})+\epsilon_{0}$. Therefore $\omega$, is such that $\sup_{n\geq 0}\|X_{n}(\omega)\|<\infty$ and for every $m\geq 0$, $X_{n_{m}}(\omega)\in\mathcal{O}^{\prime}$. Thus $\lambda(\omega)$ (see equation $\eqref{ls}$ for definition), is non-empty, compact and $\lambda(\omega)\cap\bar{\mathcal{O}^{\prime}}\subseteq\lambda(\omega)\cap B(A)\neq\emptyset$, where $B(A)$ denotes the basin of attraction of the attracting set $A$. By Theorem 3.1$(a)$, we have that for almost every $\omega$ in $E\cap\left(\cap_{m\geq 0}\left\{\omega\in\Omega:\rho(\omega,n_{m},T_{m})<\epsilon_{0}\right\}\right)$ the iterates converge to the attracting set $A$. Therefore we get that $\mathbb{P}\left(E\cap\left(\cap_{m\geq 0}\left\{\omega\in\Omega:\rho(\omega,n_{m},T_{m})<\epsilon_{0}\right\}\right)\right)\leq\mathbb{P}\left(E\cap\{\omega\in\Omega:X_{n}(\omega)\to A\ \text{as }n\to\infty)\}\right)$.

The quantity $\rho_{1}^{(l)}$ as in Definition 4, captures the difference between the linearly interpolated trajectory of recursion $\eqref{rec}$ and the solution of the o.d.e. $\eqref{pmfld1}$ over a $T>0$ length time interval. This difference can be shown to comprise of two components namely, the error due to discretization and the error due to additive noise terms. By the step size assumption, that is $(A2)$, we know that the step sizes are converging to zero. Hence intuition suggests that after a large number of iterations have elapsed the discretization error must be negligible and the contribution to the difference term $\rho_{1}^{(l)}$ is mainly due to the additive noise terms. The following is made precise in the lemma below. A brief outline of the proof of this lemma which follows from Lemma 5.7$(c)$ and [3, Ch. 2, Lemma 1], is presented in Appendix B.

Lemma 5.9

For every $l\geq 1$, for every $T_{u}\geq T_{A}$, there exists $N_{0}^{\prime}\geq 1$, such that for every $n_{0}\geq N_{0}^{\prime}$, for every $E\in\mathscr{F}_{n_{0}}$ such that, $E\subseteq\{\omega\in\Omega:X_{n_{0}}(\omega)\in\mathcal{O}^{\prime}\}$, for every sequence $\{n_{m}\}_{m\geq 0}$ as in Definition 5, for every $m\geq 0$, we have,

[TABLE]

where,

• •

$\mathcal{B}_{-1}^{(l_{0})}:=E$* and for every $M\geq 0$, $\mathcal{B}_{M}^{(l_{0})}\!:=\!E\cap\left(\cap_{m=0}^{M}\{\omega\!\in\!\Omega:\rho_{1}^{(l_{0})}(\omega,n_{m},T_{m})+\rho_{2}^{(l_{0})}(\omega,n_{m},T_{m})\!<\!\epsilon_{0}\}\!\right)$,*

• •

for every $j\geq 1$, $\zeta_{j}:=\sum_{n=0}^{j-1}a(n)M_{n+1}$, where $\{M_{n}\}_{n\geq 1}$ denote the additive noise terms as defined in assumption $(A3)$,

• •

$\{T_{m}\}_{m\geq 0}$* is as in Definition 5 and $K_{0}(T_{u})>0$ is a positive constant increasing in $T_{u}$*

Suppose event $E$ as in the lemma above occurs with some positive probability. Then the next lemma says that the lower bound of $\mathbb{P}\left(\{\omega\in\Omega:X_{n}(\omega)\to A\ \text{as }n\to\infty\}|E\right)$ depends mainly on the additive noise terms for $n_{0}$ large.

Lemma 5.10

For every $T_{u}\geq T_{A}$, there exists $l_{0}\geq 1$ and $N_{0}^{\prime}\geq 1$, such that for every $n_{0}\geq N_{0}^{\prime}$, for every $E\in\mathscr{F}_{n_{0}}$ such that, $E\subseteq\{\omega\in\Omega:X_{n_{0}}(\omega)\in\mathcal{O}^{\prime}\}$ and $\mathbb{P}(E)>0$, for every sequence $\{n_{m}\}_{m\geq 0}$ as in Definition 5, we have,

[TABLE]

where, the sequence of events $\{\mathcal{B}_{m}^{(l_{0})}\}_{m\geq-1}$, the sequence of random vectors $\{\zeta_{j}\}_{j\geq 1}$ and the constant $K_{0}(T_{u})$ are as defined in Lemma 5.9.

• Proof

By Lemma 5.6$(f)$, we get that there exists $l_{0}\geq 1$ (depending on $\bar{\mathcal{O}^{\prime}}$, $T_{u}$ and $\epsilon_{0}$) such that for every $\bm{\mathrm{x}}^{(l_{0})}\in S^{(l_{0})}(T_{u},\bar{\mathcal{O}^{\prime}})$, there exists $\bm{\mathrm{x}}\in S(T_{u},\bar{\mathcal{O}^{\prime}})$ such that $\sup_{t\in[0,T_{u}]}\|\bm{\mathrm{x}}^{(l_{0})}(t)-\bm{\mathrm{x}}(t)\|<\frac{\epsilon_{0}}{2}$. Further by Lemma 5.8$(a)$ and definition of $E$, we get that for every $m\geq 0$, $\mathcal{B}_{m-1}^{(l_{0})}\subseteq\{\omega\in\Omega:X_{n_{m}}(\omega)\in\mathcal{O}^{\prime}\}$. Therefore, for every $\omega\in\mathcal{B}_{m-1}^{(l_{0})}$, $\bar{\bm{\mathrm{x}}}^{(l_{0})}(\cdot;n_{m},T_{m},\omega)\in S^{(l_{0})}(T_{m},\bar{\mathcal{O}^{\prime}})$ and

[TABLE]

where $\eqref{tmp5}$ follows from the fact that $T_{m}\leq T_{u}$ and $\eqref{tmp6}$ follows from our choice of $l_{0}$ and Definition 3. Therefore for every $m\geq-1$,

[TABLE]

and hence,

[TABLE]

By Lemma 5.9, we know that there exists $N_{0}^{\prime}\geq 1$ such that, for every $n_{0}\geq N_{0}^{\prime}$, for every $m\geq 0$,

[TABLE]

from which it follows that,

[TABLE]

For $l_{0}\geq 1$ as obtained above and for $n_{0}\geq N_{0}^{\prime}$, we have that,

[TABLE]

where, $\eqref{tmp9}$ follows from Lemma 5.8$(b)$, $\eqref{tmp10}$ follows from the observation that,

[TABLE]

(where the union in R.H.S. is disjoint) and $\eqref{tmp11}$ follows from the fact that for every $m\geq 0$, $\mathbb{P}(\mathcal{B}_{m-1}^{(l_{0})}|\mathcal{B}_{-1}^{(l_{0})})\leq 1$. Using $\eqref{tmp7}$ and $\eqref{tmp8}$ in $\eqref{tmp11}$, we get that there exists $l_{0}\geq 1$ and $N_{0}^{\prime}\geq 1$, such that for every $n_{0}\geq N_{0}^{\prime}$, for every $E\in\mathscr{F}_{n_{0}}$ such that $E\subseteq\{\omega\in\Omega:X_{n_{0}}(\omega)\in\mathcal{O}^{\prime}\}$ and $\mathbb{P}(E)>0$,

[TABLE]

5.4 Review of the probability bounding procedure for the additive noise terms

In this section we fix $l_{0}$ and $n_{0}\geq N_{0}$, where $l_{0}$ and $N_{0}$ are as in Lemma 5.10 and provide an upper bound for $\mathbb{P}\left(\{\omega\in\Omega:\max_{n_{m}\leq j\leq n_{m+1}}\|\zeta_{j}(\omega)-\zeta_{n_{m}}(\omega)\|\geq\frac{\epsilon_{0}}{4K_{0}(T_{u})}\}|\mathcal{B}_{m-1}^{(l_{0})}\right)$, for every $m\geq 0$. The proof of the bounding procedure is similar to that of [3, Ch. 4, Lemma. 10] and we provide a brief outline here for the sake of completeness.

• (a)

From recursion $\eqref{rec}$, we have that for every $m\geq 0$, for every $n_{m}\leq j\leq n_{m+1}-1$, for every $\omega\in\Omega$, there exists $V_{j}(\omega)\in F(X_{j}(\omega))$, such that,

[TABLE]

By assumption $(A1)$(ii), we know that $\|V_{j}(\omega)\|\leq K(1+\|X_{j}(\omega)\|)$ and hence for $n_{m}\leq j\leq n_{m+1}-1$,

[TABLE]

Further by assumption $(A3)$, we get that, for every $m\geq 0$, for almost every $\omega\in\Omega$, for $n_{m}\leq j\leq n_{m+1}-1$,

[TABLE]

Now by arguments as in [3, Lemma 9], we get that, for every $m\geq 0$, for almost every $\omega\in\Omega$, for $n_{m}\leq j\leq n_{m+1}$,

[TABLE]

• (b)

Clearly $\{\zeta_{j}-\zeta_{n_{m}},\mathscr{F}_{j}\}_{n_{m}\leq j\leq n_{m+1}}$ is a martingale. By (34) and $(A3)$, we get that for $n_{m}\leq j<n_{m+1}$, $\|\zeta_{j+1}-\zeta_{j}\|=\|a(j)M_{j+1}\|\leq a(j)K(1+\|X_{j}\|)\leq a(j)K(1+e^{2KT_{u}}(1+2KT_{u}\|X_{n_{m}}\|))$. Since for every $\omega\in\mathcal{B}^{(l_{0})}_{m-1}$, $X_{n_{m}}(\omega)\in\mathcal{O}^{\prime}$ (whose closure is compact), there exists a $C>0$, such that $\|X_{n_{m}}(\omega)\|\leq C$. Therefore for every $m\geq 0$, for every $\omega\in\mathcal{B}^{(l_{0})}_{m-1}$, for every $n_{m}\leq j<n_{m+1}$, $\|\zeta_{j+1}-\zeta_{j}\|\leq a(j)K(1+e^{2KT_{u}}(1+2KT_{u}C))$. Thus applying the concentration inequality for martingales, by arguments exactly the same as in the proof of [3, Lemma 10], we get that for every $m\geq 0$,

[TABLE]

where, $\tilde{K}:=\epsilon_{0}^{2}/\left(32(K_{0}(T_{u}))^{2}dK(1+e^{2KT_{u}}(1+2KT_{u}C))\right)$.

• Proof

of Theorem 3.2: Let $l_{0}\geq 1$ and $N_{0}^{\prime}$ be as in Lemma 5.10. By definition of $b(\cdot)$, we get that there exists $N_{0}^{\prime\prime}\geq 1$, such that for every $n\geq N_{0}^{\prime\prime}$, $b(n)<\tilde{K}$. Define $N_{0}:=\max\{N_{0}^{\prime},N_{0}^{\prime\prime}\}$. Let $n_{0}\geq N_{0}$ and $\{n_{m}:=\tau(n_{m-1},T_{A})\}_{m\geq 1}$. $\{n_{m}\}_{m\geq 1}$ as defined satisfies the conditions mentioned in Definition 5. Then by Lemma 5.10 and $\eqref{tmp14}$, we get that for $n_{0}\geq N_{0}$,

[TABLE]

We know that $e^{-\tilde{K}/x}/x\to 0$ as $x\to 0$ and increases with $x$ for $0<x<\tilde{K}$. Therefore by our choice of $n_{0}$, we get that,

[TABLE]

from which it follows that for every $m\geq 0$, $e^{-\frac{\tilde{K}}{b(n_{m})-b(n_{m+1})}}\leq(b(n_{m})-b(n_{m+1}))\frac{e^{-\frac{\tilde{K}}{b(n_{0})}}}{b(n_{0})}.$ Substituting the above in $\eqref{tmp16}$, we get that for every $n_{0}\geq N_{0}$,

[TABLE]

6 Proof of finite resets theorem (Thm. 4.1)

From the definition of $\chi_{n}$ in equation $\eqref{indi}$, we know that the $\chi_{n}$ takes the value one if there is a reset of the $n^{th}$ iterate and is zero otherwise. Therefore $\sum_{n=1}^{\infty}\chi_{n}$ denotes the total number of resets.

Suppose the event $\{\sum_{n=1}^{\infty}\chi_{n}\geq k\}$ has zero probability for some $k\geq 1$. Then for $k\geq 1$, such that $\mathbb{P}(\sum_{n=1}^{\infty}\chi_{n}\geq k)=0$, we have $\mathbb{P}(\sum_{n=1}^{\infty}\chi_{n}<k)=1$, from which Theorem 4.1 trivially follows. Therefore without loss of generality assume $\mathbb{P}(\sum_{n=1}^{\infty}\chi_{n}\geq k)>0$, for every $k\geq 1$.

For every $k\geq 0$, let $G_{k}$ denote the event that there are at most $k$ resets and $G_{\infty}$ denote the event that there are finitely many resets. That is, for every $k\geq 0$, $G_{k}:=\{\sum_{n=1}^{\infty}\chi_{n}\leq k\}$ and $G_{\infty}:=\{\sum_{n=1}^{\infty}\chi_{n}<\infty\}$. Then it is clear that, for every $k\geq 1$, $G_{k}\subseteq G_{k+1}$ and $G_{\infty}=\cup_{k\geq 0}G_{k}$. Therefore $\lim_{k\to\infty}\mathbb{P}(G_{k})$ exists and $\mathbb{P}(G_{\infty})=\lim_{k\to\infty}\mathbb{P}(G_{k})$. For any $k\geq 1$,

[TABLE]

The event $\{\sum_{n=1}^{\infty}\chi_{n}=k\}$ can be written as a disjoint union of events as below. For every $k\geq 1$

[TABLE]

where, $\{\sum_{n=1}^{0}\chi_{n}=k-1\}:=\Omega$. Let $J(k):=\left\{n_{0}\geq 1:\mathbb{P}\left(\left\{\sum_{n=1}^{n_{0}-1}\chi_{n}=k-1\right\}\cap\left\{\chi_{n_{0}}=1\right\}\right)>0\right\}$. Then for every $k\geq 1$,

• (a)

By arguments in the second paragraph of this section we have that $\mathbb{P}(G_{k-1}^{c})=\mathbb{P}(\sum_{n=1}^{\infty}\chi_{n}\geq k)>0$. Further the event $\{\sum_{n=1}^{\infty}\chi_{n}\geq k\}$ can be written as a disjoint union of events as below.

[TABLE]

from which it follows that,

[TABLE]

Therefore $J(k)\neq\emptyset$.

• (b)

$\min\{n_{0}\in J(k)\}\geq k$, since there cannot be $k$ resets in less than $k$ iterations.

From $\eqref{tmp18}$ and definition of $J(k)$, we have that for every $k\geq 1$,

[TABLE]

*Step 1 (Obtaining $\mathcal{O}^{\prime},\ \epsilon_{0}$ and $T_{A}$) : * By $(A4)^{\prime}$, we have that $A$ is a globally attracting set of DI $\eqref{mfld}$. Let $\tilde{r}>0$ be such that $A\subseteq\tilde{r}\mathring{U}$. By definition of a globally attracting set and [2, Lemma 3.13], we get that for any $r\geq\tilde{r}$, $r\mathring{U}$ is a fundamental neighborhood of $A$. Let $k_{1}\geq 1$ be such that $r_{k_{1}}\geq\tilde{r}$. Set the fundamental neighborhood $\mathcal{O}:=r_{k_{1}+1}\mathring{U}$ and $\mathcal{O}^{\prime}:=r_{k_{1}}\mathring{U}$. Obtain $\epsilon_{0}>0$ and $T_{A}>0$ as in section 5.3. That is $\epsilon_{0}>0$ is such that $N^{2\epsilon_{0}}(A)\subseteq\mathcal{O^{\prime}}\subseteq N^{\epsilon_{0}}(\bar{\mathcal{O^{\prime}}})\subseteq\mathcal{O}$ and $T_{A}>0$, is such that for every $x\in\mathcal{O}$, for every $t\geq T_{A}$, $\Phi(t,x)\in N^{\epsilon_{0}}(A)$.

*Step 2 (Obtaining $\{n_{m}\}_{m\geq 1}$ as in Definition 5) : * Clearly there exists $k_{2}\geq 1$, such that for every $k\geq k_{2}$, $T_{A}\leq 2^{k}T_{W}$. For any $n_{0}\geq 1$, for every $m\geq 1$, define $n_{m}:=n_{2^{k_{2}},m-1}$, where for every $1\leq j\leq 2^{k_{2}}$, $n_{j,m-1}:=\tau(n_{j-1,m-1},T_{W})$ with $n_{0,m-1}:=n_{m-1}$ and $\tau(\cdot,\cdot)$ is as defined in Definition 1. Therefore for every $m\geq 1$, $T_{m-1}:=t(n_{m})-t(n_{m-1})=\sum_{j=0}^{2^{k_{2}}-1}\Delta(n_{j,m-1},T_{W})$, where $\Delta(\cdot,\cdot)$ is as defined in Definition 1. Thus for every $m\geq 0$, $T_{A}\leq 2^{k_{2}}T_{W}\leq T_{m}\leq 2^{k_{2}}T_{W}+2^{k_{2}}$ and hence $T_{u}=2^{k_{2}}T_{W}+2^{k_{2}}$.

*Step 3 (Redefining trajectories) : * Define $\bar{X}$, as defined in $\eqref{lit}$, with the iterates $\{X_{n}\}_{n\geq 0}$ (iterates before reset check) generated by Algorithm 1. For every $n\geq 1$, define $\tilde{X}(\cdot,\cdot;n):\Omega\times[t(n),\infty)\rightarrow\mathbb{R}^{d}$ such that for every $(\omega,t)\in\Omega\times[t(n),t(n+1))$,

[TABLE]

and for every $(\omega,t)\in[t(n+1),\infty)$, $\tilde{X}(\omega,t;n)=\bar{X}(\omega,t)$.

*Step 4 (Obtaining parameters) : * By arguments exactly same as the ones used to obtain $\eqref{rec1}$, we get that, for every $l\geq 1$, for every $n\geq 0$, there exists a $U$-valued random variable on $\Omega$, say $\tilde{U}^{(l)}_{n}$ such that, for every $\omega\in\Omega$,

[TABLE]

*Step 5 (Redefining distance measures) : * For every $\omega\in\Omega$, for every $n\geq n^{\prime}\geq 1$, for every $T>0$, for every $l\geq 1$,

• (a)

let $\tilde{\bm{\mathrm{x}}}^{(l)}(\cdot;n,n^{\prime},T,\omega):[0,T]\rightarrow\mathbb{R}^{d}$ denote the unique solution of the o.d.e.

[TABLE]

with initial condition $\tilde{\bm{\mathrm{x}}}^{(l)}(0;n,n^{\prime},T,\omega)=X_{n^{\prime}}^{\prime}(\omega)$, where $\tilde{u}(\cdot;n,n^{\prime},T,\omega):[0,T]\rightarrow U$ is defined such that, for every $t\in[0,T]$, $\tilde{u}(t;n,n^{\prime},T,\omega):=\tilde{U}_{k}^{(l)}(\omega)$, where $\tilde{U}_{k}^{(l)}$ is as in equation $\eqref{rec2}$ and $k$ is such that $t+t(n)\in[t(k),t(k+1))$ (for a proof of existence and uniqueness of solutions to o.d.e. $\eqref{pmfld2}$, see Lemma 5.7). It is easy to see that for every $l\geq 1$, $\tilde{\bm{\mathrm{x}}}^{(l)}(\cdot;n,n^{\prime},T,\omega)\in S^{(l)}(T,X_{n^{\prime}}^{\prime}(\omega))$, the set of solutions of DI $\eqref{amfld}$, as defined in $\eqref{ss1}$.

• (b)

define,

• (1)

$\tilde{\rho}(\omega,n,n^{\prime},T):=\inf_{\bm{\mathrm{x}}\in S(T,\bar{\mathcal{O}^{\prime}})}\sup_{t\in[0,T]}\|\tilde{X}(\omega,t+t(n);n^{\prime})-\bm{\mathrm{x}}(t)\|$,

• (2)

$\tilde{\rho}^{(l)}_{1}(\omega,n,n^{\prime},T):=\sup_{t\in[0,T]}\|\tilde{X}(\omega,t+t(n);n^{\prime})-\tilde{\bm{\mathrm{x}}}^{(l)}(t;n,n^{\prime},T,\omega)\|$,

• (3)

$\tilde{\rho}^{(l)}_{2}(\omega,n,n^{\prime},T):=\inf_{\bm{\mathrm{x}}\in S(T,\bar{\mathcal{O}^{\prime}})}\sup_{t\in[0,T]}\|\tilde{\bm{\mathrm{x}}}^{(l)}(t;n,n^{\prime},T,\omega)-\bm{\mathrm{x}}(t)\|$.

*Step 6 (Collecting sample paths) : * Fix $k>\max\{k_{1},k_{2}\}$ and $n_{0}\in J(k)$. By our definition of $\mathscr{F}_{n_{0}}$ (see section 4), we have that $E(k,n_{0}):=\{\sum_{n=1}^{n_{0}-1}\chi_{n}=k-1,\chi_{n_{0}}=1\}\in\mathscr{F}_{n_{0}}$ and is contained in $\{X_{n_{0}}^{\prime}(\omega)\in\mathcal{O}^{\prime}\}$. Given that there has been a reset at index $n_{0}$, the next reset check is performed by Algorithm 1 at the iteration index $n_{2^{k-k_{2}}}$. So for $n_{0}+1\leq j<n_{2^{k-k_{2}}}$, $X_{j}(\omega)=X_{j}^{\prime}(\omega)$. From arguments exactly the same as Lemma 5.8$(a)$, we get that,

[TABLE]

where, $\eqref{tmp22}$ follows from the fact that $k\geq k_{1}$ and hence $\mathcal{O}^{\prime}=r_{k_{1}}\mathring{U}\subseteq r_{k}\mathring{U}$. It is also worth mentioning here that the proof of Lemma 5.8$(a)$ holds irrespective of how the iterates are generated. Given that $\omega\in E(k,n_{0})\cap\left(\cap_{m=0}^{2^{k-k_{2}}-1}\{\tilde{\rho}^{(l)}_{1}(\omega,n_{m},n_{0},T_{m})+\tilde{\rho}^{(l)}_{2}(\omega,n_{m},n_{0},T_{m})<\epsilon_{0}\}\right)$, along this sample path there has been a reset at $n_{0}$ and at the next check performed at $n_{2^{k-k_{2}}}$ there has been no reset. Hence the next check for reset is performed by Algorithm 1 at $n_{2^{(k-k_{2})+1}}$. Again from arguments from Lemma 5.8$(a)$, we get that,

[TABLE]

Repeating the above for the third reset check after $n_{0}$ and so on, we obtain that,

[TABLE]

*Step 7 (Bounding) : * Define $\tilde{\mathcal{B}}^{(l)}_{-1}:=E(k,n_{0})$ and for every $M\geq 1$ define,

[TABLE]

Note that as in Lemma 5.10, we can obtain $l_{0}\geq 1$, such that for every $m\geq 0$, for every $\omega\in\tilde{B}^{(l_{0})}_{m-1}$ we have that $\tilde{\rho}^{(l_{0})}_{2}(\omega,n_{m},n_{0},T_{m})<\frac{\epsilon_{0}}{2}$, since for every $\omega\in\tilde{\mathcal{B}}^{(l_{0})}_{m-1}$, $X_{n_{m}}(\omega)\in\mathcal{O}^{\prime}$ and whether or not a reset check is performed at this index, we have that $X_{n_{m}}(\omega)=X_{n_{m}}^{\prime}(\omega)$. Thus for such an $l_{0}$, mimicking the proof of Lemma 5.10, we obtain that,

[TABLE]

From Lemma 5.9, we have that for $k>\max\{k_{1},k_{2},N_{0}^{\prime}\}$, for every $n_{0}\in J(k)$,

[TABLE]

Step 8 (Noise bound) Similar to item $(a)$ in section 5.4, from Algorithm (1), we have that for every $m\geq 0$, for every $n_{m}\leq n_{m+1}-1$, $\|X_{j+1}\|\leq\|X_{j}^{\prime}\|(1+a(j)K)+a(j)K+a(j)\|M_{j+1}\|$ and since $\|X_{j+1}^{\prime}\|\leq\|X_{j+1}\|$, we get that for every $n_{m}\leq j\leq n_{m+1}-1$,

[TABLE]

Now by arguments exactly same as those item $(a)$ of section 5.4, we get that for every $m\geq 0$, for every $n_{m}\leq j\leq n_{m+1}-1$, $\|X_{j+1}^{\prime}\|\leq e^{2KT_{u}}(\|X_{n_{m}}\|+2KT_{u})$. Now by using concentration inequality as in item $(b)$ of section 5.4, we get that for every $m\geq 0$,

[TABLE]

Using $\eqref{tmp23}$, $\eqref{tmp24}$ and $\eqref{tmp25}$ we get that, for every $k\geq\max\{k_{1},k_{2},N_{0}\}$ (where $N_{0}$ is as defined in the proof of Theorem 3.2), for every $n_{0}\in J(k)$,

[TABLE]

Substituting the above in $\eqref{tmp21}$ and using the fact that for $n\leq n^{\prime}$, $b(n^{\prime})\leq b(n)$, we get that, for every $k\geq\max\{k_{1},k_{2},N_{0}\}$,

[TABLE]

Substituting $\eqref{tmp26}$ in $\eqref{tmp17}$, we get that for every $k\geq\max\{k_{1},k_{2},N_{0}\}$,

[TABLE]

Letting $k\to\infty$ in the above equation and using the fact that $\mathbb{P}(G_{\infty})=\lim_{k\to\infty}\mathbb{P}(G_{k})$, we get that $\mathbb{P}(G_{\infty})=1$.

7 Conclusions and directions for future work

We have extended the lock-in probability result (Theorem 3.2) in [6] to stochastic approximation schemes with set-valued drift functions which serves as an important tool for analyzing recursions when their stability is not guaranteed. The extension to set-valued map allows one to obtain lock-in probability for stochastic approximation schemes with measurable drift functions and schemes where the drift function itself possess a non-additive unknown noise component (see [3, Ch. 5.3]). Further using Theorem 3.2, in the presence of a locally attracting set for the mean field, we have provided an alternate condition for verification of convergence in the absence of stability guarantee which involves verifying whether the iterates are entering infinitely often, an open neighborhood of the attractor with a compact closure. In the presence of a globally attracting set our modified recursion as in Algorithm 1, converges almost surely to the globally attracting set, the proof of which relies on the method used to obtain the lock-in probability result.

In future we wish to consider other applications of the lock-in probability result such as sample complexity (see [3, Ch. 4.2]) and almost sure convergence under tightness of the iterates (see [7]). Another interesting direction, is to explore various additive noise models where the above result can be extended for the case of set-valued drift functions.

Appendix A Definitions of some topological concepts

Let $(\mathcal{M},\Gamma)$ be a topological space. $\{O_{i}\}_{i\in I}$ is an *covering *of $\mathcal{M}$ if, for every $i\in I$, $O_{i}\subseteq\mathcal{M}$ and $\cup_{i\in I}O_{i}=\mathcal{M}$. Further a covering $\{O_{i}\}_{i\in I}$ is said to be *locally finite *if for every $p\in\mathcal{M}$, there exists an $O\in\Gamma$, such $O_{i}\cap O\neq\emptyset$ for only finitely many $i\in I$. Given any two coverings $\mathcal{C}:=\{O_{i}\}_{i\in I}$ and $\mathcal{C}^{\prime}:=\{O_{j}^{\prime}\}_{j\in J}$, $\mathcal{C}^{\prime}$ is said to be a *refinement *of $\mathcal{C}$ if, for every $i\in I$, there exists a $j\in J$ such that, $O_{i}\subseteq O_{j}^{\prime}$. $\mathcal{C}^{\prime}$ is said to be a locally finite refinement of $\mathcal{C}$, if $\mathcal{C}^{\prime}$ is a refinement of $\mathcal{C}$ and is locally finite. The topological space $(\mathcal{M},\Gamma)$ is *paracompact *if it is a Hausdorff space and if every open covering has a locally finite open refinement.

A family of functions $\{\psi_{i}\}_{i\in I}$ is called a *locally Lipschitz partition of unity *if for all $i\in I$,

• •

$\psi_{i}$ is locally Lipschitz continuous and non negative,

• •

the supports of $\psi_{i}$, defined as $\overline{\{p\in\mathcal{M}:\psi_{i}(p)\neq 0\}}$, are a closed locally finite covering of $\mathcal{M}$,

• •

for each $p\in\mathcal{M}$, $\sum_{i\in I}\psi_{i}(p)=1$.

A partition of unity $\{\psi_{i}\}_{i\in I}$ is said to be subordinated to the covering $\{O_{i}\}_{i\in I}$, if for every $i\in I$, $\overline{\{p\in\mathcal{M}:\psi_{i}(p)\neq 0\}}\subseteq O_{i}$.

Appendix B Proof of Lemma 5.9

Fix $l\geq 1$ and $T_{u}\geq T_{A}$. Fix $n_{0}\geq 1$ and $\{n_{m}\}_{m\geq 0}$ as in Definition 5. Fix $m\geq 0$. Let $\omega\in\{\omega\in\Omega:\rho_{1}^{(l)}(\omega,n_{m},T_{m})\geq\frac{\epsilon_{0}}{2}\}\cap\mathcal{B}_{m-1}^{(l)}$. Then, by Lemma 5.8$(a)$ we have that, $X_{n_{m}}(\omega)\in\mathcal{O}^{\prime}$. By $\eqref{lit}$ and Definition 3, we have that for every $0\leq t\leq T_{m}$, there exists $n_{m}\leq k\leq n_{m+1}-1$ such that $t\in[t(k),t(k+1)]$, and

[TABLE]

for some $\alpha\in[0,1]$ and

[TABLE]

Therefore for any $t\in[0,T_{m}]$,

[TABLE]

Now the aim is to provide an upper bound for each terms in the $R.H.S.$ of the above inequality which is independent of $m$ and $t$. In order to apply [3, Ch. 2, Lemma 1] the only additional condition needed is the Lipschitz continuity of $f^{(l)}(\cdot,\cdot)$ uniformly over $u$ and this is obtained using local Lipschitz continuity as follows. Let $M_{1}>0$ be such that, for every $x\in\mathcal{O}^{\prime}$, $\|x\|\leq M_{1}$. By Lemma 5.7$(b)$ and item (a) in section 5.4, we have that for $r:=\max\{C_{1}(\bar{\mathcal{O}^{\prime}},T_{u},l),e^{2KT_{u}}(M_{1}+2KT_{u})\}$,

[TABLE]

Further by Lemma 5.7$(c)$, we know that there exists $L(r,l)>0$, such that for every $x,x^{\prime}\in rU$, for every $t\in[0,T_{m}]$

[TABLE]

The rest of the bounding procedure is exactly the same as [3, Ch. 2, Lemma 1]. We obtain that, for every $t\in[0,T_{m}]$,

[TABLE]

Now set $N_{0}^{\prime}$ such that $(M_{1}+KT_{u})e^{2L(r,l)T_{u}}L(r,l)\sum_{j\geq 0}a(n_{0}+j)^{2}+(M_{1}+KT_{u})e^{L(r,l)T_{u}}a(n_{0})<\frac{\epsilon_{0}}{2}$ and define $K_{0}(T_{u}):=e^{L(r,l)T_{u}}$. Then, for every $n_{0}\geq N_{0}^{\prime}$, we have that $\omega\in\{\omega\in\Omega:\max_{n_{m}\leq j\leq n_{m+1}}\|\zeta_{j}(\omega)-\zeta_{n_{m}}(\omega)\|\geq\frac{\epsilon_{0}}{4K_{0}(T_{u})}\}\cap\mathcal{B}^{(l)}_{m-1}$, from which Lemma 5.9 follows.