A Bayesian Theory of Change Detection in Statistically Periodic Random Processes

TL;DR

This paper introduces a Bayesian framework for change detection in a new class of stochastic processes with periodic statistical behavior, using optimal control of Markov decision processes to derive effective stopping rules.

Contribution

It develops a Bayesian change detection theory for i.p.i.d. processes and formulates it as an optimal control problem for periodic MDPs, providing algorithms and asymptotic optimality results.

Findings

Optimal change detection policy is nonstationary and periodic.

Single-threshold policy is asymptotically optimal as false alarm probability approaches zero.

Numerical results show the asymptotic policy is not strictly optimal.

Abstract

A new class of stochastic processes called independent and periodically identically distributed (i.p.i.d.) processes is defined to capture periodically varying statistical behavior. A novel Bayesian theory is developed for detecting a change in the distribution of an i.p.i.d. process. It is shown that the Bayesian change point problem can be expressed as a problem of optimal control of a Markov decision process (MDP) with periodic transition and cost structures. Optimal control theory is developed for periodic MDPs for discounted and undiscounted total cost criteria. A fixed-point equation is obtained that is satisfied by the optimal cost function. It is shown that the optimal policy for the MDP is nonstationary but periodic in nature. A value iteration algorithm is obtained to compute the optimal cost function. The results from the MDP theory are then applied to detect changes in i.p.i.d. processes. It is shown that while the optimal change point algorithm is a stopping rule based on a periodic sequence of thresholds, a single-threshold policy is asymptotically optimal, as the probability of false alarm goes to zero. Numerical results are provided to demonstrate that the asymptotically optimal policy is not strictly optimal.

Figures

Tables

Table 1. TABLE I: Performance comparison for i.i.d. data.

	Cost of	Cost of
$\theta$	Periodic-Shiryaev	Optimal Policy
$0.5$	$11.1$	$5.0$
$1.0$	$12.0$	$5.0$
$2.0$	$9.4$	$5.0$

Table 2. TABLE II: Performance comparison for different mean parameter choices.

		Cost of	Cost of
${\theta }_1$	${\theta }_2$	Single-Threshold Policy	Optimal Policy
$2.0$	$0.0$	$7.2$	$5.0$
$2.0$	$0.5$	$8.8$	$5.0$
$3.0$	$0.5$	$6.6$	$5.0$
$3.0$	$1.0$	$7.2$	$5.0$
$1.0$	$0.1$	$9.5$	$5.0$
$0.5$	$0.0$	$8.1$	$5.0$

Table 3. TABLE III: Performance comparison for different choices of delay and false alarm penalties.

						Cost of	Cost of
						Single-Threshold	Optimal
${\lambda }_0$	${\lambda }_1$	$d_0$	$d_1$	${\theta }_1$	${\theta }_2$	Policy	Policy
$20$	$5$	$10$	$1$	$2$	$1$	$10.2$	$5.0$
$20$	$5$	$1$	$1$	$2$	$1$	$4.6$	$3.7$
$5$	$5$	$1$	$1$	$2$	$1$	$3.2$	$3.2$

Equations

$$f_{n+T}=f_n,\forall n\ge 1.$$

$$g_{n+T}=g_n,\forall n\ge 1.$$

$$Y_n\sim \{\begin{array}{ll}{f}_n,& \forall n<\nu ,\\ g_n& \forall n\ge \nu .\end{array}$$

$$g_i\ne f_i,\text{for some }i=1,2,\cdots ,T.$$

$${\pi }_n=\mathsf{P}(\nu =n),\text{ for }n=1,2,\cdots .$$

$${\mathsf{P}}^{\pi }=\sum _{n=1}^{\infty }{\pi }_n{\mathsf{P}}_n.$$

$${\mathsf{P}}^{\pi }(\tau <\nu ).$$

$${\mathsf{E}}^{\pi }\left[(\tau -\nu {)}^{+}\right]$$

$${\mathsf{E}}^{\pi }\left[\tau -\nu |\tau \ge \nu \right],$$

$$\underset{\tau \in {\mathbf{C}}_{\alpha }}{\min }{\mathsf{E}}^{\pi }\left[(\tau -\nu {)}^{+}\right],$$

$${\mathbf{C}}_{\alpha }=\{\tau :{\mathsf{P}}^{\pi }(\tau <\nu )\le \alpha \},$$

$$\mathsf{P}(\nu =n)=(1-\rho {)}^{n-1}\rho ,\text{ for }n=1,2,\cdots .$$

$${\mathsf{P}}^{\pi }=\sum _{n=1}^{\infty }(1-\rho {)}^{n-1}\rho {\mathsf{P}}_n.$$

$$\underset{\tau }{\min }{\mathsf{E}}^{\pi }\left[(\tau -\nu {)}^{+}\right]+{\lambda }_f{\mathsf{P}}^{\pi }(\tau <\nu ),$$

$$p_n={\mathsf{P}}^{\pi }(\nu \le n|Y_1,\cdots ,Y_n),\text{ for }n\ge 1.$$

$$\underset{\tau }{\min }{\mathsf{E}}^{\pi }\left[\sum _{n=0}^{\tau -1}{p}_n+{\lambda }_f(1-p_{\tau })\right].$$

$$p_n=\frac{{\tilde{p}}_{n-1}{g}_n(Y_n)}{{\tilde{p}}_{n-1}{g}_n(Y_n)+(1-{\tilde{p}}_{n-1})f_n(Y_n)},$$

$${\tilde{p}}_{n-1}=p_{n-1}+(1-p_{n-1})\rho .$$

$${\Theta }_k\in \{\mathcal{A},0,1\},\forall k.$$

$$U_k\in \{1(\text{stop}),2(\text{continue})\}.$$

$$\begin{array}{rl}(Y_k|{\Theta }_k=0,& U_{k-1}=2)\sim f_k\\ (Y_k|{\Theta }_k=1,& U_{k-1}=2)\sim g_k,\end{array}$$

$$P(u_k)=\{\begin{array}{l}{P}_1,\text{if }{u}_k=1\\ P_2,\text{if }{u}_k=2,\end{array}$$

$$P_1=\left[\begin{array}{ccc}{p}_{\mathcal{AA}}& p_{\mathcal{A}0}& p_{\mathcal{A}1}\\ p_{0\mathcal{A}}& p_{00}& p_{01}\\ p_{1\mathcal{A}}& p_{10}& p_{11}\end{array}\right]=\left[\begin{array}{ccc}1& 0& 0\\ 1& 0& 0\\ 1& 0& 0\end{array}\right]$$

$$P_2=\left[\begin{array}{ccc}{p}_{\mathcal{AA}}& p_{\mathcal{A}0}& p_{\mathcal{A}1}\\ p_{0\mathcal{A}}& p_{00}& p_{01}\\ p_{1\mathcal{A}}& p_{10}& p_{11}\end{array}\right]=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1-\rho & \rho \\ 0& 0& 1\end{array}\right].$$

$${\tilde{\pi }}_0=({\tilde{\pi }}_0(\mathcal{A}),{\tilde{\pi }}_0(0),{\tilde{\pi }}_0(1){)}^T=(0,1,0{)}^T.$$

$$\begin{array}{rl}{C}_k(0,1)& ={\lambda }_k\\ C_k(1,2)& =d_k\\ C_k(\theta ,u)& =0,\text{otherwise}.\end{array}$$

$$\begin{array}{rl}{\lambda }_{k+T}& ={\lambda }_k\\ d_{k+T}& =d_k.\end{array}$$

$$I_k=(Y_1,\cdots ,Y_k,U_1,\cdots ,U_{k-1})$$

$$\tilde{\Phi }=({\tilde{\varphi }}_1,{\tilde{\varphi }}_2,\cdots )$$

$$\begin{array}{r}V({\tilde{\pi }}_0)=\underset{\tilde{\Phi }}{\min }\mathsf{E}\left[\sum _{k=0}^{\infty }{C}_k({\Theta }_k,U_k)\right].\end{array}$$

Full text

000The work of Taposh Banerjee was supported by a grant from the U.S. Army Research Laboratory, W911NF1820295. Taposh Banerjee is with the Department of Electrical and Computer Engineering, The University of Texas at San Antonio, San Antonio, TX, 78249. Prudhvi Gurram is with Booz Allen Hamilton, McLean, VA 22102, and the U.S. Army Research Laboratory, Adelphi, MD 20783. Gene Whipps is with the U.S. Army Research Laboratory, Adelphi, MD 20783 (e-mail: taposh.banerjee@utsa.edu; gurram_prudhvi@bah.com; gene.t.whipps.civ@mail.mil). Copyright (c) 2019 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE by sending a request to pubs-permissions@ieee.org.

I Introduction

In the problem of quickest change detection, the objective is to detect a change in the distribution of a sequence of random variables with the minimum possible delay, subject to a constraint on the rate of false alarms [1, 2, 3]. Optimal or asymptotically optimal algorithms for quickest change detection are available in the literature. The results can be divided broadly into two categories: results for independent and identically distributed (i.i.d.) processes with algorithms that can be computed recursively and using finite memory and enjoy strong optimality properties [4, 5], and results for non-i.i.d. data with algorithms that cannot be necessarily computed recursively or using finite memory but are asymptotically optimal [6, 7, 8, 9].

In this paper, we develop theory and algorithms for detecting changes in stochastic processes that have periodically varying statistical characteristics. In this non-i.i.d. setting, we will show that the optimal algorithms can be computed recursively and using finite memory. The motivation for this problem comes from the following anomaly detection problems in cyber-physical systems and biology where such periodic behavior is observed.

Traffic monitoring: In [11] and [12], we reported results on multimodal traffic data we collected from NYC around a 5K run; during, before and after the run. We collected CCTV images, Twitter and Instagram data. We extracted counts of persons and vehicles appearing in CCTV images over time using a deep neural network-based object detector. We observed that in the absence of the event (in the normal regime), the counts have a periodic statistical behavior (over a day or a week) with increased intensity every day during morning and evening rush hours. 2. 2.

Social networks: We also observed in [11] and [12] that the aggregate social network behavior shows periodic characteristics under the normal regime. Also, the total number of Instagram messages posted in a rectangular area around the CCTV cameras showed periodic behavior. 3. 3.

Power grid monitoring: The power usage by end users have a periodic pattern with low usage at nighttime and high usage at daytime [10]. 4. 4.

Neural spike patterns: In brain-computer interface studies where single neural spike data is collected, the spike firing pattern can exhibit statistically periodic behavior in the absence of any external stimuli; see, for example, [13]. 5. 5.

ECG: Several biological signals including the ECG have a periodic behavior [14].

The problem of anomaly detection in the above-mentioned applications can be seen as a problem of detecting changes in deviation from periodic statistical behavior.

In this paper, we develop a Bayesian theory for anomaly detection in problems where the statistical characteristics are periodic. We introduce a class of stochastic processes called independent and periodically identically distributed (i.p.i.d) processes that can be used to model such periodic statistical behavior. We then develop algorithms for quickest detection of changes in i.p.i.d. processes and prove their optimality with respect to the Bayesian criterion of Shiryaev [15]. In the Shiryaev formulation, the objective is to detect a change in the distribution of a stochastic process to minimize the average detection delay, subject to a constraint on the probability of false alarm. In the Shiryaev problem, each time we take an observation, we pay a penalty for delay if the change has already occurred. If an alarm is raised and the change has not yet occurred, we pay a penalty for a false alarm. In this paper, we also study a more general or modified Shiryaev formulation where the penalties on the delay and the false alarm are dependent on time. The latter problem is relevant for detecting changes in non-i.i.d. processes. The definition of i.p.i.d. processes and precise problem formulations are given in Section II.

When the observations are i.i.d. before and after the change, optimal algorithms are obtained in the Bayesian setting of Shiryaev using optimal stopping theory or dynamic programming [5], [17], [22]. However, since the processes under investigation here are not i.i.d., but i.p.i.d., the traditional optimal control theory cannot be applied. We show in this paper that the change detection problems for i.p.i.d. processes we study in this paper can be mapped to a problem of optimal control of a Markov decision process (MDP) with a nonstationary but periodic transition and cost structure. As a result, in Section III, we first develop an optimal control theory for periodic MDPs. With a view towards optimal stopping, we develop the optimal control theory for the total cost problem with finite control spaces [17]. For stationary problems, the optimal policy can be obtained using the framework of dynamic programming and can be shown to be Markovian and stationary [17]. A general recipe for solving nonstationary (including periodic) problems can also be found in [17]; e.g., see pg. 256. In fact, it is suggested in [17] that the optimal policy for periodic problems is periodic in nature; see also [18]. In this paper, we explicitly derive optimal policies for periodic problems using a more direct approach and prove that the optimal policies are indeed periodic. We obtain a fixed point equation satisfied by the optimal cost function. We also obtain a value iteration algorithm for computing the optimal cost using which the periodic optimal policy can be computed. The optimal control theory developed here for periodic MDPs should be of independent interest for other control applications as well.

In Section IV, we apply the optimal control theory developed in Section III to the change detection problems. We show that the optimal change detection algorithm is periodic. The change detection statistic, as in the classical i.i.d. setting, is the a posteriori probability that the change has already occurred. But, unlike the i.i.d. setting where a single-threshold policy is strictly optimal, the stopping threshold for the i.p.i.d. problem varies with time. In fact, we show that the sequence of thresholds is periodic. We provide examples where a periodic and nonstationary policy is strictly better than a single threshold stationary policy.

In Section V, however, we show that if the constraint on the probability of false alarm is small, then we can, in fact, use a fixed time-invariant threshold. Specifically, we show that a single-threshold algorithm is asymptotically optimal for the classical Shiryaev formulation by showing that the proposed algorithm achieves a universal lower bound on the delay of any change detection procedure [7]. We will show that while the exact optimality result and periodic MDP theory is valid only for geometric priors, the asymptotic optimality result is valid for a large class of distributions on the change point.

II Mathematical Model

We begin by first introducing a class of stochastic processes that can be used to model data showing periodic statistical behavior.

Definition 1

A stochastic process $\{Y_{n}\}$ is called independent and periodically identically distributed (i.p.i.d) if

The random variables $\{Y_{n}\}$ are independent. 2. 2.

If $Y_{n}$ has density $f_{n}$, for $n\geq 1$, then there is a positive integer $T$ such that the sequence of densities $\{f_{n}\}$ is periodic with period $T$:

[TABLE]

We say that the process is i.p.i.d. with the law $(f_{1},\cdots,f_{T})$. Note that the law of an i.p.i.d. process is completely characterized by the finite-dimensional product distribution involving $(f_{1},\cdots,f_{T})$. We assume that in the normal regime, the data can be modeled as an i.p.i.d. process. At some point in time, due to an event, the distribution of the i.p.i.d. process deviates from $(f_{1},\cdots,f_{T})$. Our objective in this paper is to develop algorithms to observe $\{Y_{n}\}$ in real time and detect changes in the distribution as quickly as possible, subject to a constraint on the rate of false alarms. If the period $T=1$, an i.p.i.d. process reduces to an i.i.d. process. Optimal algorithms for change detection in i.i.d. processes has been extensively developed in the literature [1, 2, 3, 4, 5, 6, 7, 8, 9].

We now define a change point model. Consider another periodic sequence of densities $\{g_{n}\}$ such that

[TABLE]

We assume that at some point in time $\nu$, called the change point in the following, the law of the i.p.i.d. process is governed not by the densities $(f_{1},\cdots,f_{T})$, but by the new set of densities $(g_{1},\cdots,g_{T})$:

[TABLE]

The densities $(g_{1},\cdots,g_{T})$ need not be all different from the set of densities $(f_{1},\cdots,f_{T})$, but we assume that there exists at least an $i$ such that they are different:

[TABLE]

II-A Classical Shiryaev Formulation

Let $\tau$ be a stopping time for the process $\{Y_{n}\}$, i.e., a positive integer-valued random variable such that the event $\{\tau\leq n\}$ belongs to the $\sigma$-algebra generated by $Y_{1},\cdots,Y_{n}$. In other words, whether or not $\tau\leq n$ is completely determined by the first $n$ observations. We declare that a change has occurred at the stopping time $\tau$. To find the best stopping rule to detect the change in distribution, we need a performance criterion. Towards this end, we model the change point $\nu$ as a random variable with a prior $\pi$:

[TABLE]

For each $n\in\mathbb{N}$, we use $\mathsf{P}_{n}$ to denote the law of the observation process $\{Y_{n}\}$ when the change occurs at $\nu=n$, and use $\mathsf{E}_{n}$ to denote the corresponding expectation. Using this notation, we define the average probability measure

[TABLE]

To capture a penalty on the false alarms, we use the probability of false alarm defined as

[TABLE]

To penalize the detection delay, we use the average detection delay given by

[TABLE]

or its conditional version

[TABLE]

where $x^{+}=\max\{x,0\}$.

The optimization problem we are interested in solving is

[TABLE]

where

[TABLE]

and $\alpha$ is a given constraint on the probability of false alarm. In the above problem, we can also use the conditional version of the delay $\mathsf{E}^{\pi}\left[\tau-\nu|\tau\geq\nu\right]$.

When the change point $\nu$ is a geometric random variable, the classical approach to solving problem (3) is to solve a relaxed version using dynamic programming. Specifically, let

[TABLE]

Then

[TABLE]

The relaxed Bayesian optimization problem is

[TABLE]

where $\lambda_{f}>0$ is a penalty on the cost of false alarms. The above optimization problem can be stated as a problem in partially observable MDPs (POMDPs); see Section II-B, and also [1, 22], and [11]. Specifically, define $p_{0}=0$ and

[TABLE]

Then, it can be shown that problem (4) is equivalent to solving

[TABLE]

If the period $T=1$ and the processes are i.i.d., then the problem in (6) can be solved using classical belief state MDPs [1, 22, 11, 17]. However, in our case, the belief updates are not stationary.

Lemma II.1

The belief $p_{n}$ in (5) can be recursively computed using the following equations: $p_{0}=0$ and for $n\geq 1$,

[TABLE]

where

[TABLE]

Proof:

The proof is provided in the appendix. ∎

Note that the likelihood ratios are a function of the time index $n$. Thus, the belief updates are nonstationary. However, because the processes are i.p.i.d. in nature and there are only finitely many densities $(f_{1},\cdots,f_{T})$ and $(g_{1},\cdots,g_{T})$, the belief updates have a periodic structure that repeats after $T$ time slots.

The optimal stopping problem (6) cannot be solved using classical optimal stopping theory or dynamic programming [5], [17], [22]. This is because in these theories it is assumed that the Markov process to be controlled is homogeneous. The Markov process $\{p_{n}\}$ to be controlled in (6) is not homogeneous. However, as shown in Lemma V.1, the transition structure is periodic. Motivated by this observation, in Section III, we develop the optimal stopping theory or optimal control theory for periodic MDPs. In Section IV, we will apply the results obtained for periodic MDPs to the periodic optimal stopping problem (6).

II-B Modified Shiryaev Formulation

In this section, we formulate a more general optimal stopping problem than stated in (6). In the problem in (6), the delay penalty at all times is $1$ unit and the false alarm penalty is $\lambda_{f}$ units. Since the processes under study here are not i.i.d., we formulate a POMDP where the delay and false alarm penalties are a function of time. Since we are investigating i.p.i.d. processes, we assume that the delay and false alarm penalties are periodic as well. The precise problem formulation is given below.

• •

States: Let $\{\Theta_{k}\}_{k\geq 0}$ be a sequence of states with values $\{\theta_{k}\}$. The state process is a finite-state Markov chain taking values

[TABLE]

The state $\mathcal{A}$ is a special absorbing state introduced for mathematical convenience in a stopping time POMDP [22].

• •

Control: The control sequence for the POMDP is the process $\{U_{k}\}_{k\geq 0}$ taking values $\{u_{k}\}$ and is binary valued:

[TABLE]

The control $U_{k}=2$ is used to continue the observation process and $U_{k}=1$ is used to stop it. At the time of stopping, an alarm is raised indicating that a change in the distribution of the observations has occurred.

• •

Observations: The distribution of the observations $\{Y_{k}\}$ depends on the state $\Theta_{k}$ and if the control is to continue: for $k\geq 1$,

[TABLE]

with the understanding that the observation process is i.p.i.d. with law $(f_{1},\cdots,f_{T})$ before the change, and i.p.i.d. with law $(g_{1},\cdots,g_{T})$ after the change. No observation is collected at time $k=0$ and the distribution of the observations when the state equals $\mathcal{A}$ is irrelevant to the problem.

• •

Transition Structure: The Markov chain $\{\Theta_{k}\}$ evolves according to a transition structure that depends on the control process $\{U_{k}\}$. Let $P(u_{k})$ be the transition matrix for the Markov chain, given the control is $U_{k}=u_{k}$. Then, we have

[TABLE]

where

[TABLE]

and

[TABLE]

The initial distribution $\tilde{\pi}_{0}$ for the Markov chain $\{\Theta_{k}\}$ is

[TABLE]

Thus, the Markov chain $\{\Theta_{k}\}$ starts at [math]. As long as the control $U_{k}=2$, which means to continue, the states evolve according to the transition probability matrix $P_{2}$. The values selected for elements of matrix $P_{2}$ ensure that the absorption time to the state $1$ is inevitable and is a geometrically distributed random variable with parameter $\rho$. Before the absorption, the distributions of the observations are i.p.i.d. with law $(f_{1},\cdots,f_{T})$. After the state is absorbed in $1$ the distributions of the observations changes to that of an i.p.i.d. process with law $(g_{1},\cdots,g_{T})$.

• •

Cost: The cost $C_{k}(\theta,u)$ associated with state $\Theta_{k}=\theta$ and control $U_{k}=u$ is defined for $k\geq 0$ as

[TABLE]

Thus, $\lambda_{k}$ is the penalty on the false alarm at time $k$ and $d_{k}$ is the penalty on the delay at time $k$. We assume that the penalty sequences are periodic with period $T$: for any $k\geq 0$,

[TABLE]

• •

Policy: Let

[TABLE]

be the information at time $k$. Also define a policy

[TABLE]

to be a sequence of mappings such that $U_{k}=\tilde{\phi}_{k}(I_{k}),\;\forall k$.

We want to find a control policy to optimize the long term cost

[TABLE]

Using arguments similar to those used to obtain (6), it can be shown that the probability sequence

[TABLE]

is a sufficient statistics also for the problem in (17). Consequently, solving (17) is equivalent to solving the following MDP problem:

[TABLE]

If $\lambda_{k}=\lambda_{f}$ and $d_{k}=1$, $\forall k$, then the problem in (18) reduces to the problem in (6). If the period $T=1$ and the processes are i.i.d., then the problem in (18) can also be solved using classical MDP theory [1, 22, 11, 17]. However, for $T>1$, the observation process is i.p.i.d. and the classical theory cannot be applied. The optimal control theory developed in Section III below for periodic MDP can and will be used to solve (18) and hence its special case (6).

III Optimal Control of Periodic MDPs

In this section, we develop an optimal control theory for MDPs with periodic cost and transition structure. We have stochastic processes $\{X_{k}\}$, $\{U_{k}\}$, and $\{W_{k}\}$ taking values is spaces as follows:

[TABLE]

The process $\{X_{k}\}$ is an MDP generated according to the transition structure

[TABLE]

Here $\{U_{k}\}$ is the control process and $\{W_{k}\}$ is the disturbance process. We assume that given $X_{k}=x_{k}$ and $U_{k}=u_{k}$, the distribution of the disturbance $W_{k}$ is independent of the past disturbances $W_{0},\cdots,W_{k-1}$. We use $t_{k}(w_{k}|x_{k},u_{k})$ to denote this conditional distribution. Thus, the state and disturbance spaces are real-valued and the control spaces are finite. The results in the paper are, in fact, valid for more general spaces. To accommodate more general spaces, the proof techniques may have to be slightly modified [17, 20].

The main assumption in our model is that the transition functions $\{\phi_{k}\}$ and the conditional distributions $\{t_{k}(w_{k}|x_{k},u_{k})\}$ are periodic: there is a positive integer $T$ such that for $k\geq 0$,

[TABLE]

The objective is to choose the control process so as to minimize the cost

[TABLE]

where $\alpha\in[0,1]$ is the discount factor which is allowed to be equal to $1$ with a view towards problems in optimal stopping. The cost functions $\{c_{k}\}$ are assumed to be non-negative and periodic with the same period $T$: for $k\geq 0$,

[TABLE]

The assumption of non-negativity of the cost functions (which is the same as the assumption $P$ in [17]) ensures that all infinite summations are well-defined by monotone convergence theorem [21].

In order to minimize this long-term additive cost (22), we search over Markov control policies of type

[TABLE]

such that

[TABLE]

As done in [17], it can be argued that restricting the search over Markov policies is sufficient. For a policy $\Pi$, we define the cost-to-go function starting with the state $X_{0}=x_{0}$ as

[TABLE]

where the expectation is with respect to the disturbances. We are interested in solving the following problem: for $x_{0}\in\mathbb{R}$,

[TABLE]

In this section, we show that the optimal policy for the problem in (25) is periodic with period $T$, i.e., it is of the type

[TABLE]

We also provide an explicit way to compute this optimal periodic policy.

For $\ell=0,1,\cdots,T-1$, and $x\in\mathbb{R}$, define the operator

[TABLE]

where the expectation $\mathsf{E}^{(\ell)}$ is defined with respect to the conditional distribution $t_{\ell}(w|x,u)$. We also define the operator for a Markov map $\mu$ and $x\in\mathbb{R}$:

[TABLE]

Finally, define the $T-$fold operator

[TABLE]

which is the successive application of the $T$ operators defined in (26). Our first result is the following.

Theorem III.1

The optimal cost function $V^{*}$ in (25) satisfies the following fixed-point equation: for any $x\in\mathbb{R}$,

[TABLE]

Proof:

The proof is provided in the appendix. ∎

Next, we show that if the optimal cost function is known, then the optimal policy can be obtained and shown to be periodic.

Theorem III.2

The optimal policy is periodic. Specifically, let $V^{*}$ be the optimal cost function and let $[\mu^{*}_{0},\cdots\mu^{*}_{T-1}]$ be such that for $\ell=0,1,\cdots,T-1$, and $x\in\mathbb{R}$,

[TABLE]

Then, the optimal policy is given by

[TABLE]

Proof:

The proof is provided in the appendix. ∎

An optimal policy always exists in our case because we assume the control spaces to be finite.

The previous result is useful only when we have an algorithm to compute the optimal cost function $V^{*}$. This is facilitated by the theorem below. Define

[TABLE]

where we have used $\left[\Psi\right]^{k}(V_{0})$ to denote the operator $\Psi$ in (28) applied $k$ times to the all-zero function $V_{0}$.

Theorem III.3

The limit of sequence $\{V_{k}\}$ in (32) exists:

[TABLE]

Furthermore,

[TABLE]

Proof:

The proof is provided in the appendix. ∎

Thus, the recipe for finding the periodic optimal policy is to start with the all-zero function and apply the value iteration (32) to obtain the optimal cost function $V^{*}=V_{\infty}$. Finally, solve (30) to obtain the $T$ Markov maps $[\mu^{*}_{0},\cdots\mu^{*}_{T-1}]$ to construct the optimal policy $\Pi^{*}$ in (31).

IV Detecting Changes in I.P.I.D. Processes

In this section, we solve the problem in (18), and hence the problem in (6), using Theorems III.1, III.2, and III.3. Specifically, we can state the following result. Let

[TABLE]

where $p_{0}=p$.

Theorem IV.1

The optimal cost function in (34) satisfies

[TABLE]

where for $\ell=0,\cdots,T-1$,

[TABLE]

In the above equation,

[TABLE]

where $\tilde{p}=p+(1-p)\rho$, and

[TABLE]

In the above theorem, the densities are assumed to be with respect to the Lebesgue measure. The expressions can be modified to allow for counting measures (summations) or more general measures.

If $T=1$ and $d_{0}=1$, then we are reduced to the classical formulation of Shiryaev. It is well known that the optimal policy for the classical case is a single-threshold policy in which the change is declared the first time the statistic or probability $p_{n}$ is above a pre-defined threshold. The threshold depends on the choice of the false alarm penalty $\lambda_{0}$.

For $T>1$, it is interesting to ask if the Shiryaev single-threshold stopping rule is still the optimal policy. In the subsections below, we will show that the optimal policy, in fact, utilizes multiple thresholds, where the number of distinct thresholds can be up to $T$. For reference below, we define the Shiryaev stopping rule $\tau_{ps}$ here: for $A\geq 0$,

[TABLE]

In the rest of the paper, we call this stopping rule or policy the periodic-Shiryaev stopping rule or algorithm to emphasize that the recursion for $p_{n}$ is periodic.

IV-A Example: Change Detection in I.P.I.D. Processes For Different Values of $T$

In this section, we show two examples in the i.p.i.d. setting where the optimal policy is not stationary. In fact, it is periodic with periodic thresholds.

In Fig. 1, we report results for $T=2$. Specifically, consider a change detection problem where the period of the i.p.i.d. processes is $T=2$ and the pre- and post-change i.p.i.d. densities are Gaussian:

[TABLE]

The parameters for change point, false alarm and delay are as follows:

[TABLE]

The optimal cost function was obtained using value iteration (33) with the operators $\{\Psi^{(\ell)}\}$ as defined in (36). The cost functions at each iteration

[TABLE]

with $J_{0}$ being the all-zero function on $[0,1]$, are plotted in Fig. 1(a). We used a $100$ point resolution or discretization of the interval $[0,1]$ in the value iteration. In Fig 1(b), we have plotted the norm distance $\|J_{k}-J_{k+1}\|_{2}$ as a function of the iteration index $k$. In Fig. 1(c), we have plotted the stopping cost and the continue cost appearing in (36) for $\ell=0,1$, where we use stages to refer to the $T$ distinct time slots. It can be inferred from Fig. 1(c) that the optimal policy has alternating thresholds: $A_{1}=0.6$ and $A_{2}=0$. For stopping, $p_{n}$ is compared with the threshold $A_{1}$ during the odd time slots and compared with the threshold $A_{2}$ during the even time slots. The optimal cost achieved by this alternating threshold policy is $J^{*}(0)=5.0$, as can be seen in Fig. 1(a). In Fig. 1(d), we have plotted the total cost achieved by the periodic-Shiryaev algorithm (39) for different values of constant threshold $A$. These costs were obtained through Monte-Carlo simulations using $10,000$ sample paths. The best achievable cost of the single-threshold periodic-Shiryaev algorithm is $10.2$ establishing that the optimal cost of $5.0$ cannot be achieved using a single-threshold policy.

Next, we consider a change detection problem where the period of the i.p.i.d. processes is $T=4$ and the pre- and post-change i.p.i.d. densities are given by

[TABLE]

The parameters for change point, false alarm and delay are as follows:

[TABLE]

Results for this case are reported in Fig. 2. Similar to results for $T=2$ in Fig. 1, we see here also that the optimal cost is again $5.0$ (Fig. 2(a)). The Fig. 2(c) also shows that there are four thresholds in this case, one for each of the four stages in a cycle or period of length $T=4$. Again, the best cost achievable by the periodic-Shiryaev algorithm equals $11.3$ and is strictly larger than the cost of the optimal policy.

IV-B Change Detection in I.I.D. Data

In this section, we show examples where the periodic-Shiryaev algorithm is not strictly optimal even for i.i.d. data, as long as $T>1$ and we use the modified Shiryaev formulation discussed in Section II-B. Specifically, we consider the change point problem with parameters

[TABLE]

and

[TABLE]

In Table I, we have reported the comparison between the optimal policy and the periodic-Shiryaev algorithm for the above parameters for different choices of the post-change parameter $\theta$. All points are obtained using Monte-Carlo simulations with $10,000$ sample paths.

IV-C Performance Comparison For Different Mean Choices

In this section, we report comparison between the performance of the optimal policy and the periodic-Shiryaev algorithm for different choices of mean parameters for $T=2$:

[TABLE]

The parameters for change point, false alarm and delay are as follows.

[TABLE]

The results are collected in Table II. The values in the table suggests that the superiority of the optimal policy over the periodic-Shiryaev is maintained for different values of the mean parameters $\theta_{1}$ and $\theta_{2}$.

IV-D Performance Comparison For Different Choices of Delay and False Alarm Penalties

In the previous sections, we have shown examples where the periodic-Shiryaev is strictly sub-optimal. In this section, we show that the performance gap depends on the choice of the delay and false alarm penalties $\lambda_{0},\lambda_{1},d_{0},d_{1}$. In Table III, we show the performance comparison for

[TABLE]

The performance gap reduces if the false alarm penalties are kept different but the delay penalties are set to the same value. The performance gap, in fact, vanishes as the false alarm and delay penalties are equal and the problem reduced to the classical Shiryaev case. In the next section, we provide a theoretical basis for this observation by showing that the periodic-Shiryaev algorithm is, in fact, asymptotically optimal for the classical Shiryaev formulation, as the false alarm rate goes to zero. While we could not find an example where the two algorithms have different performance for the classical Shiryaev formulation, we conjecture that such an example exists and necessarily involve high values of probability of false alarm.

V Asymptotic Optimality of Single-Threshold Policies

In this section, we show that the periodic-Shiryaev algorithm is asymptotically optimal for the classical Shiryaev formulation (3) as the probability of false alarm goes to zero.

For easy reference, we recall the definition of the periodic-Shiryaev algorithm here. Define

[TABLE]

and stop the first time this probability is above a threshold, i.e., use the stopping rule

[TABLE]

While the statistic $p_{n}$ is always well-defined in a Bayesian setting, recall that in general for a non-i.i.d. model, the Shiryaev statistic cannot be computed recursively using a finite amount of memory [7, 3]. Another convenient way to compute the statistic $p_{n}$ is to compute its transformation $R_{n}$ defined as

[TABLE]

The statistic $R_{n}$ can also be computed recursively.

Lemma V.1

In the i.p.i.d. setting, the statistic $R_{n}$ (51) can be computed recursively as

[TABLE]

with $R_{0}=0$. Further, if the prior $\pi$ is $\text{Geom}(\rho)$ then the above recursion simplifies to

[TABLE]

Proof:

The proof is provided in the appendix. ∎

V-A Universal Performance Bounds For Change Detection in I.P.I.D. Processes

In this section, we obtain a universal lower bound on the performance of any stopping rule for detecting changes in an i.p.i.d. process.

Let there exist $d\geq 0$ such that

[TABLE]

If $\pi=\text{Geom}(\rho)$, then

[TABLE]

Thus, $d=|\log(1-\rho)|$.

Further, let

[TABLE]

where $D(g_{i}\;\|\;f_{i})$ is the Kullback-Leibler divergence between the densities $g_{i}$ and $f_{i}$. We assume that

[TABLE]

and

[TABLE]

Theorem V.1

Let the information number $I$ be as defined in (55) and satisfy $0<I<\infty$. Also, let $d$ be as in (54). Then, for any stopping time $\tau\in\mathbf{C}_{\alpha}$, i.e., for any $\tau$ satisfying the false alarm constraint $\mathsf{P}^{\pi}(\tau<\nu)\leq\alpha$, we have

[TABLE]

Here $o(1)\to 0$ as $\alpha\to 0$.

Proof:

The proof is provided in the appendix. ∎

V-B Optimality of Periodic-Shiryaev Algorithm

We now show that the periodic-Shiryaev algorithm (50) is asymptotically optimal for problem (3) as the false alarm constraint $\alpha\to 0$. We will establish the optimality by showing that the periodic-Shiryaev algorithm achieves the lower bound specified in Theorem V.1.

Let

[TABLE]

and define for $\epsilon>0$

[TABLE]

We assume for simplicity that $\{f_{i}\}$ and $\{g_{i}\}$ are densities with respect to Lebesgue measure on the real line. The results below can be easily extended to densities with respect to more general measures (including the counting measure).

Theorem V.2

Let

[TABLE]

Then, we have

[TABLE]

The implication of (59) is that with $A=1-\alpha$ in (50), we have

[TABLE]

Proof:

The proof is provided in the appendix. ∎

Thus, the periodic-Shiryaev algorithm achieves the lower bound and is asymptotically optimal. The arguments provided in the proofs of the theorems above can be extended to also establish asymptotic optimality with respect to higher order moments of the detection delay.

V-C Numerical Results

In Fig. 3, we have plotted the average detection delay (ADD) $\mathsf{E}[(\tau_{ps}-\nu)^{+}]$ as a function of the magnitude of the logarithm of the probability of false alarm (PFA) for the following set of parameters:

[TABLE]

The values for simulations were obtained using $5000$ sample paths. The values for the analysis curve in the figure were obtained by setting the probability of false alarm using the threshold $A=1-\alpha$ and using the delay expression $\frac{|\log\alpha|}{I+|\log(1-\rho)|}$. As can be observed from the figure, the analytical expression provides an accurate estimate of the delay.

In Fig. 4, we have plotted a typical sample path of the periodic-Shiryaev algorithm for the same set of parameters used in Fig. 3.

VI Conclusion

We established the optimality of periodic policies for optimal control of MDPs in which the cost structure and the transition probabilities are periodic, all with the same period. We then applied this result to solve an optimal stopping problem using the framework of partially observable MDPs. The optimal stopping problem we studied is the problem of detecting changes in i.p.i.d. processes. The exact optimality theory suggests that the optimal policy has multiple thresholds (alternating, in case the period $T=2$). This structural behavior, or its effect, is absent in the low false alarm regime, where we showed that using a single, fixed, threshold is asymptotically optimal, as the probability of false alarm goes to zero. A Bayesian analysis often provides important insights into a problem. The insight we obtain from this paper is that when analyzing a non-Bayesian or minimax version of the problem studied in this paper, one can conjecture that a single-threshold policy like cumulative sum is not strictly optimal for all values of the false alarm rate [4].

Proof:

For $k\leq n$ we have

[TABLE]

where $\Gamma_{n}=\mathsf{P}(\nu>n)$. This can be formally proved using the rigorous definition of conditional expectations using sub-sigma-algebras [21]. This implies

[TABLE]

Using this we can obtain an expression for $1-p_{n}$:

[TABLE]

The numerator of $p_{n}$ can be written as

[TABLE]

Using this expansion and dividing the numerator and the denominator of $p_{n}$ by $\sum_{k=1}^{n-1}\pi_{k}\prod_{i=1}^{k-1}f_{i}(Y_{i})\prod_{i=k}^{n-1}g_{i}(Y_{i})+\Gamma_{n-1}\prod_{i=1}^{n-1}f_{i}(Y_{i})$ we get

[TABLE]

The lemma is proved if we now used the geometric prior.

∎

Proof:

Recall from (24) that for any policy $\Pi=[\mu_{0},\mu_{1},\cdots]$, the cost-to-go function is given by

[TABLE]

Rearranging terms we get

[TABLE]

where for any $x\in\mathbb{R}$

[TABLE]

Using the fact that the transition structures, conditional distribution of the disturbances, and the cost structures are periodic with period $T$, we have

[TABLE]

From this we get

[TABLE]

Further,

[TABLE]

where the operator $\Psi^{(T-1)}$ is defined in (26). Continuing in this manner, we get

[TABLE]

Thus, for any $x\in\mathbb{R}$, we have

[TABLE]

Since the right hand side is not a function of the policy $\Pi$, we can take the minimum over all policies on the left to get

[TABLE]

We now prove that the inequality above (66) is also true in the other direction. Let $[\mu^{*}_{0},\cdots\mu^{*}_{T-1}]$ be such that $\forall x\in\mathbb{R}$ and for $\ell=0,1,\cdots,T-1$,

[TABLE]

Thus, $\mu^{*}_{0},\cdots\mu^{*}_{T-1}$ are the optimal solutions to the minimization problem encountered in the definition of operators $\Psi^{(0)},\cdots,\Psi^{(T-1)}$, respectively. These optimal solutions always exist because the control space $\mathbb{U}$ is finite. Now consider the policy $\Pi^{*}$ given by

[TABLE]

By definition we have

[TABLE]

We now show that $V_{\Pi^{*}}\leq\Psi^{(0)}\Psi^{(1)}\Psi^{(2)}\cdots\Psi^{(T-1)}\left(V^{*}\right)$ thus proving the reverse inequality in (66).

Now, note that by the monotone convergence theorem

[TABLE]

Since the limit always exists, we can take the limit through a subsequence:

[TABLE]

where $\mathbb{N}$ is the set of positive integers. Now note that for any $m$

[TABLE]

where we have used the definition of $\Pi^{*}$ and (66). Also, we have used the notation $\left[\Psi_{\mu_{0}^{*}}^{(0)}\Psi_{\mu_{1}^{*}}^{(1)}\Psi_{\mu_{2}^{*}}^{(2)}\cdots\Psi_{\mu_{T-1}^{*}}^{(T-1)}\right]^{m}$ to denote that the composition of $T$ operators is applied $m$ times. Iterating in the above equation we get for any $m$

[TABLE]

Since the upper bound is not a function of the limit index $m$, we get

[TABLE]

Thus, all inequalities in the above equation are equality giving us

[TABLE]

This proves the theorem.

∎

Proof:

The proof follows from (74) in the proof of Theorem III.1. ∎

Proof:

Recall that the operator $\Psi$ is defined as

[TABLE]

Also, we use the notation $\left[\Psi\right]^{k}$ to denote that the operator $\Psi$ is applied $k$ times. We are interested in the sequence of functions obtained by repeatedly applying the operator $\Psi$ to the all-zero function $V_{0}\equiv 0$:

[TABLE]

First note that for any $\ell=0,1,\cdots,T-1$, and functions $J_{1}\leq J_{2}$

[TABLE]

Since $V_{0}\equiv 0$ and the cost functions are non-negative, we have $V_{1}\geq V_{0}$. Using this and the monotonicity property established in the above display we obtain the following series of inequalities:

[TABLE]

These equations establish that the sequence $V_{k}=\left[\Psi\right]^{k}(V_{0})$ is a monotonically increasing sequence. Hence, the limit

[TABLE]

always exists.

We next want to show that the limit $V_{\infty}$ satisfies the fixed-point equation $V_{\infty}=\Psi(V_{\infty})$. Towards this end, note that since $V_{k}\leq V_{\infty}$, we have

[TABLE]

Taking the limit $k\to\infty$ we get

[TABLE]

Let there exists $\tilde{x}\in\mathbb{R}$ such that

[TABLE]

We now show that such an $\tilde{x}$ cannot exist. To show this, we evaluate $\left[\Psi\right]^{k+1}(V_{0})$ at the value $\tilde{x}$:

[TABLE]

Let $u_{k}(\tilde{x})$ be the control achieving the minimum in (79). Note that this optimal value depends on $\tilde{x}$ as well as the index $k$. Consider the sequence of these values $\{u_{k}(\tilde{x})\}$ as we vary $k$. Since the control space $\mathbb{U}$ is finite, there exists $\tilde{u}\in\mathbb{U}$ such that $u_{k}(\tilde{x})=\tilde{u}$ infinite often, i.e., there exists a $\tilde{u}\in\mathbb{U}$ and an infinite set $\mathbb{N}_{1}\subset\mathbb{N}$ such that

[TABLE]

Thus, for $k\in\mathbb{N}_{1}$ we have

[TABLE]

We wish to take $k\to\infty$ in the above equation by keeping $k$ in $\mathbb{N}_{1}$. To evaluate the limits in the above equation, we note that if ${J_{k}}$ is a sequence of non-negative increasing functions with monotone limit $J_{k}\uparrow J_{\infty}$, then for $\ell=0,\cdots,T-1$,

[TABLE]

where we have used monotone convergence theorem and the fact that the control space $\mathbb{U}$ is finite. From this, we get the following set of limits:

[TABLE]

Using the above limits and taking $k\to\infty$ in (80) while $k\in\mathbb{N}_{1}$ we have from the monotone convergence theorem again

[TABLE]

This contradicts (78) and hence it must be true that

[TABLE]

Thus, $V_{\infty}$ also satisfies the fixed-point equation that $V^{*}$ satisfies. If the solution to the fixed point equation is unique, then the theorem is proved. But, just as in the classical theory, which is obtained by setting $T=1$, the solution to the fixed point equation for $\alpha=1$ need not be unique. So, we now need an additional argument to show that $V_{\infty}=V^{*}$.

We first show that $V^{*}\geq V_{\infty}$. This is true because $V^{*}\geq V_{0}$. Now, applying the $T$-fold operator $\Psi$ we get

[TABLE]

Applying the operator $\Psi$ multiple times gives us

[TABLE]

Taking the limit $k\to\infty$ we get

[TABLE]

To prove $V^{*}\leq V_{\infty}$, let $\mu^{\infty}_{0},\cdots\mu^{\infty}_{T-1}$ be the optimal solutions to the minimization problem encountered in (84). Again, these optimal solutions always exist because the control space $\mathbb{U}$ is finite. Now consider the policy $\Pi^{\infty}$ given by

[TABLE]

By definition we have

[TABLE]

We now show that $V_{\Pi^{\infty}}\leq V_{\infty}$ proving what we need: $V^{*}\leq V_{\infty}$. Towards this end, note that

[TABLE]

Now note that for any $m$

[TABLE]

where we have used definition of $\Pi^{\infty}$ and the fact that $V_{\infty}$ satisfies $\Psi^{(0)}\Psi^{(1)}\Psi^{(2)}\cdots\Psi^{(T-1)}\left(V_{\infty}\right)=V_{\infty}$. Iterating in the above equation we get for any $m$

[TABLE]

Since the upper bound is not a function of the limit index $m$, we get

[TABLE]

Thus, all the inequalities in the above equation are equalities and the theorem is proved.

∎

Proof:

Using the definition of $R_{n}=p_{n}/(1-p_{n})$ we have

[TABLE]

To get the recursion for $R_{n}$ we write

[TABLE]

proving (52). Substituting for a geometric prior gives (53). ∎

Proof:

Let $Z_{i}=\log\frac{g_{i}(Y_{i})}{f_{i}(Y_{i})}$ be the log likelihood ratio at time $i$. We show that the sequence $\{Z_{i}\}$ satisfies the following statement:

[TABLE]

where $I$ is as defined in (55). The lower bound then follows from Theorem 1 in [7]. Towards proving (91), note that as $n\to\infty$

[TABLE]

The above display is true because of the i.p.i.d. nature of the observation processes. This implies that as $n\to\infty$

[TABLE]

To show this, note that

[TABLE]

For a fixed $\epsilon>0$, because of (92) and the second term on the right, the LHS in (93) is greater than $I(1-\epsilon)$ for $n$ large enough. Now, let the maximum on the LHS be achieved at a point $k_{n}$, then

[TABLE]

Now $k_{n}$ cannot be bounded because of the presence of $n$ in the denominator and the lower bound $I(1-\epsilon)$. This implies $k_{n}>i$, for any fixed $i$, and $k_{n}\to\infty$. Thus, $\frac{1}{k_{n}}\sum_{i=\nu}^{\nu+k_{n}}Z_{i}\to I$. Since $k_{n}/n\leq 1$, we have that the LHS in (93) is less than $I(1+\epsilon)$, for $n$ large enough. This proves (93). To prove (91), note that due to the i.p.i.d. nature of the process

[TABLE]

The right hand side goes to zero because of (93) (a.s. convergence implies convergence in probability) and because the maximum on the right hand side in (95) is over only finitely many terms. This proves (91) and the theorem. ∎

Proof:

First consider the sequence

[TABLE]

Due to the periodic nature of the process, the above sequence is also periodic with period $T$, and as a result, there are at most $T$ distinct values in the above sequence:

[TABLE]

Thus, if we show that these $T$ numbers are finite for any $\epsilon>0$, then we automatically have

[TABLE]

Furthermore, since we do not make any explicit assumptions on the actual values taken by the densities $(f_{1},\cdots,f_{T})$ and $(g_{1},\cdots,g_{T})$, it is enough to show that

[TABLE]

This is because if we cyclically shift both the set of densities $(f_{1},\cdots,f_{T})$ and $(g_{1},\cdots,g_{T})$ by one, then $L_{\epsilon}^{(1)}$ with the new configuration of densities will be identically distributed as $L_{\epsilon}^{(2)}$ in the old configuration.

Recall the definition of $L_{\epsilon}^{(1)}$:

[TABLE]

To show that $L_{\epsilon}^{(1)}$ is integrable, note that for any $n$

[TABLE]

Taking complements we get

[TABLE]

This implies

[TABLE]

Summing over $n$ we get

[TABLE]

To complete the theorem, we need to show that the rightmost term in (96) is finite.

For $\ell=1,2,\cdots,T$, define

[TABLE]

and

[TABLE]

Note from (55) that $I=\frac{I_{1}+\cdots+I_{T}}{T}$. Using these definitions, we can write $\frac{1}{k}\sum_{i=1}^{k}Z_{i}-I$ as

[TABLE]

From the above equation, we get for any $n$

[TABLE]

Thus, to show the summability of the LHS in the above equation, we need to show the summability of each of the $T$ terms on the RHS. Again, due to the i.p.i.d. nature of the process, and because we have made no explicit assumptions about the densities $(f_{1},\cdots,f_{T})$ and $(g_{1},\cdots,g_{T})$, it is enough to establish the summability of the first term on the right.

Define for $\ell=1,2,\cdots,T$,

[TABLE]

Using this definition we write

[TABLE]

Thus, with $\tilde{Z}_{i}^{(1)}=Z_{i}^{(1)}-I_{i}^{(1)}$, we have

[TABLE]

Now, $\frac{1}{k}\sum_{i=1}^{k}I_{i}^{(1)}=\frac{1}{k}I_{1}\lfloor\frac{k}{T}\rfloor\to\frac{I_{1}}{T},\quad\text{ as }k\to\infty$. Thus, for $n$ large enough, the second term on the right in (97) is identically zero. Thus, we only need to show the summability of the first term on the right in (97). Towards this end, note that the probability \mathsf{P}_{1}\left(\sup_{k\geq n}\big{|}\frac{1}{k}\sum_{i=1}^{k}\tilde{Z}_{i}^{(1)}\big{|}>\frac{\epsilon}{2T}\right) is decreasing with increasing $n$. Using this fact, we can write

[TABLE]

Now, note that \big{|}\sum_{i=1}^{k}\tilde{Z}_{i}^{(1)}\big{|} changes only when $k$ changes by multiples of $T$: $k=1,T+1,2T+1,\cdots$. As a result, in between these multiples of $T$, \big{|}\frac{1}{k}\sum_{i=1}^{k}\tilde{Z}_{i}^{(1)}\big{|} monotonically decreases with increase in $k$. This implies that

[TABLE]

The rightmost summation is finite if (58) is satisfied because the sum inside the supremum $\sum_{i=1}^{mT+1}\tilde{Z}_{i}^{(1)}$ is a sum of $m$ i.i.d. random variables with distribution of $Z_{1}$ under $\mathsf{P}_{1}$ [3], [7]. The rest of the arguments follow from Theorem 3 in [7]. ∎