On strong stationary times and approximation of Markov chain hitting times by geometric sums

TL;DR

This paper extends the understanding of Markov chain hitting times by providing bounds on their approximation via geometric sums of IID variables, using strong stationary times for improved accuracy.

Contribution

It introduces explicit bounds for approximating hitting times with geometric sums, utilizing strong stationary times, and discusses their construction and properties.

Findings

Derived bounds on the distance between hitting times and geometric sum approximations.

Showed how strong stationary times can be used to improve hitting time approximations.

Provided methods for constructing the approximating distributions.

Abstract

Consider a discrete time, ergodic Markov chain with finite state space which is started from stationarity. Fill and Lyzinski (2014) showed that, in some cases, the hitting time for a given state may be represented as a sum of a geometric number of IID random variables. We extend this result by giving explicit bounds on the distance between any such hitting time and an appropriately chosen geometric sum, along with other related approximations. The compounding random variable in our approximating geometric sum is a strong stationary time for the underlying Markov chain; we also discuss the approximation and construction of this distribution.

Equations

$$s(t)=1-\underset{s\in S}{\inf }\left\{\frac{\mathbb{P}(X_t=s)}{{\pi }_s}\right\}.$$

$$\mathbb{P}(T^{(j)}>t)=\frac{P^t(j,j)-{\pi }_j}{1-{\pi }_j},t=0,1,\dots ,$$

$$s(t)=1-\frac{\mathbb{P}(X_t=j)}{{\pi }_j}=1-\frac{{\sum }_{s\ne j}{P}^t(s,j){\pi }_s}{{\pi }_j(1-{\pi }_j)}=1-\frac{{\pi }_j-P^t(j,j){\pi }_j}{{\pi }_j(1-{\pi }_j)}=\frac{P^t(j,j)-{\pi }_j}{1-{\pi }_j}.$$

$$\begin{gathered} \mathbb{P}(T^{(j)}>t,X_t=j)=\mathbb{P}(X_t=j)-\mathbb{P}(T^{(j)}\le t,X_t=j)=\mathbb{P}(X_t=j)-\mathbb{P}(T^{(j)}\le t){\pi }_j \\ ={\pi }_j\left(1-s(t)-\mathbb{P}(T^{(j)}\le t)\right)=0, \end{gathered}$$

$$d_{TV}(\mathcal{L}(W_j),\mathcal{L}(U))=\underset{A\subseteq {\mathbb{Z}}^{+}}{\sup }|\mathbb{P}(W_j\in A)-\mathbb{P}(U\in A)|=\underset{(W_j,U)}{\inf }\mathbb{P}(W_j\ne U),$$

$$Y+T^{(j)}\stackrel{d}{=}(W_j|W_j>0).$$

$$d_{TV}(\mathcal{L}(W_j),\mathcal{L}(U))\le \frac{1-{\pi }_j}{{\pi }_j}{d}_{TV}(\mathcal{L}(W_j),\mathcal{L}(Y)).$$

$$d_{TV}(\mathcal{L}(W_j),\mathcal{L}(Y))\le \mathbb{P}(\widetilde{W_j}<T^{(j)})\le \sum _{t=1}^{\infty }\mathbb{P}(X_t=j,T^{(j)}>t).$$

$$d_{TV}(\mathcal{L}(W_j),\mathcal{L}(Y))$$

$$={\pi }_j\sum _{t=1}^{\infty }\left[\frac{1-P^t(j,j)}{1-{\pi }_j}-\mathbb{P}(T^{(j)}\le t)\right]$$

$$={\pi }_j\sum _{t=1}^{\infty }\left[\mathbb{P}(T^{(j)}>t)-\frac{P^t(j,j)-{\pi }_j}{1-{\pi }_j}\right]$$

$$={\pi }_j\mathbb{E}{T}^{(j)}-\frac{{\pi }_j^2}{1-{\pi }_j}\mathbb{E}{W}_j,$$

$$d_{TV}(\mathcal{L}(W_j),\mathcal{L}(U))\le (1-{\pi }_j)\mathbb{E}{T}^{(j)}-{\pi }_j\mathbb{E}{W}_j.$$

$$\mathbb{P}(T^{(j)}>t)=\frac{P^t(j,j)-{\pi }_j}{1-{\pi }_j},t=0,1,\dots ,$$

$$P=\left(\begin{array}{cc}1/2& 1/2\\ 1/2-\delta & 1/2+\delta \end{array}\right),$$

$$P^t=\frac{1}{2(1-\delta )}\left(\begin{array}{cc}1-2\delta +{\delta }^t& 1-{\delta }^t\\ 1-2\delta -{\delta }^t(1-2\delta )& 1+{\delta }^t(1-2\delta )\end{array}\right).$$

$$d_{TV}(\mathcal{L}(W_1),\mathcal{L}(N))\le \frac{\delta (1-2\delta )}{2(1-\delta {)}^2},$$

$$\mathbb{E}h(W_j)-\mathbb{E}h(U)=(1-{\pi }_j)\mathbb{E}\left[f_h(W_j+T^{(j)})-f_h(Y+T^{(j)})\right],$$

$$f_{h}(x)=-\mathbb{E}\left[\sum_{r=0}^{\infty}(1-\pi_{j})^{r}\{h(V_{r})-\mathbb{E}h(U)\}\Bigg{|}V_{0}=x\right]\,,$$

$$\mathbb{E}{W}_j\le \frac{1-{\pi }_j}{{\pi }_j}\mathbb{E}{T}^{(j)}\le \frac{1}{{\pi }_j}\sum _{l\ne j}{\pi }_l\mathbb{E}{T}_{(l)},$$

$$\mathbb{E}{e}^{\theta W_j}\le \frac{{\pi }_j}{1-(1-{\pi }_j)\mathbb{E}{e}^{\theta T^{(j)}}}.$$

$$\begin{gathered} \mathbb{P}(T^{(j)}>t)=\underset{i\in S}{\sup }\left(1-\frac{{\sum }_{l\ne j}{\pi }_l{P}^t(l,i)}{{\pi }_i(1-{\pi }_j)}\right)=\frac{1}{1-{\pi }_j}\underset{i\in S}{\sup }\left(\sum _{l\ne j}\left[{\pi }_l-\frac{{\pi }_l{P}^t(l,i)}{{\pi }_i}\right]\right) \\ \le \frac{1}{1-{\pi }_j}\sum _{l\ne j}{\pi }_l\underset{i\in S}{\sup }\left(1-\frac{P^t(l,i)}{{\pi }_i}\right)\le \frac{1}{1-{\pi }_j}\sum _{l\ne j}{\pi }_l\mathbb{P}(T_{(l)}>t). \end{gathered}$$

$$\mathbb{E}{T}^{(j)}\ge {\pi }_j+\frac{{\pi }_j}{1-{\pi }_j}\sum _{s\ne j}\frac{{\pi }_{s}P(s,s)}{1-P(s,s)}.$$

$$\overline{w}\le \sum _{j\in S}\sum _{l\ne j}{\pi }_l\mathbb{E}{T}_{(l)}\le (|S|-1)t^{\ast },$$

$$\frac{P^t(m,k)}{{\pi }_k}=\frac{P^t(k,m)}{{\pi }_m}\le \frac{P^t(k,m)}{{\pi }_l},$$

$$\begin{gathered} \mathbb{P}(T_{(m)}>t)=1-\underset{k}{\inf }\left(\frac{P^t(m,k)}{{\pi }_k}\right)\ge 1-\underset{k}{\inf }\left(\frac{P^t(k,m)}{{\pi }_l}\right)\ge 1-\underset{k}{\inf }\left(\frac{P^t(k,l)}{{\pi }_l}\right) \\ =1-\underset{k}{\inf }\left(\frac{P^t(l,k)}{{\pi }_k}\right)=\mathbb{P}(T_{(l)}>t), \end{gathered}$$

$${\pi }^{(j)}={\nu }^{(j)}\Lambda ,\text{mbox}and\Lambda P=P^{\ast }\Lambda .$$

$$(\Lambda P)(S_j,l)=\frac{1}{1-{\pi }_j}\left[{\pi }_l-{\pi }_{j}P(j,l)\right].$$

$$c_r=\min \left\{\frac{Q_{r-1}(S_j,l)}{{\pi }_l}:l\in A_{r-1},Q_{r-1}(S_j,l)>0\right\}.$$

$$A_r=\left\{l\in A_{r-1}:\frac{Q_{r-1}(S_j,l)}{{\pi }_l}\ge c_r\right\}\ne \varnothing$$

$$\mathbb{E}{T}^{(j)}=1+\mathbb{P}(X_1^{\ast }\notin \{S,S_j\})+\mathbb{P}(X_1^{\ast }=S_j)\mathbb{E}{T}^{(j)}.$$

$$\mathbb{E}{T}^{(j)}=1+\frac{1-P^{\ast }(S_j,S)}{1-P^{\ast }(S_j,S_j)}.$$

Full text

Key words and phrases: Markov chains; passage time; geometric sum; strong stationary time

AMS 2010 subject classification: 60J10 (Primary); 60E15, 60F05, 62E17 (Secondaries)

1 Preliminaries

Throughout, we let $X=\{X_{t}:t\geq 0\}$ be an irreducible, ergodic, discrete time Markov chain with finite state space $S$, transition matrix $P$, and stationary distribution $\pi$. Our primary interest in this note is in the time it takes $X$, when initialised in the stationary regime, to hit a given state $j\in S$. Many structural results are known about such hitting times. In particular, it is known that, under certain conditions, this hitting time may be expressed as a geometric sum of independent and identically distributed (IID) random variables; see Theorem 4.2 of Fill and Lyzinski [6] for a precise statement. Our main goal in this note is to consider the approximation of such a hitting time by a geometric sum. Motivated by the results of [6], an appropriate choice for the compounding distribution in the approximating geometric sum is that of a strong stationary time for the underlying Markov chain $X$ with an appropriate initial distribution. Recall that a strong stationary time, $T$, for $X$ is a randomized stopping time such that $X_{T}\sim\pi$, and $X_{T}$ is independent of $T$. We refer the reader to [1] and [4] for background on strong stationary times, and to [5], [8] and references therein for more recent developments.

We use the remainder of this section to introduce some necessary background material on strong stationary times, and to provide an explicit link between geometric sums of strong stationary times and Markov chain hitting times. Section 2 then contains our main approximation results, and a comparison of the present results with those of Daly [3], who has previously considered the approximation of Markov chain hitting times by geometric sums. Note, however, that the approximating geometric sum chosen in [3] is very different to that used here, and that the results of [3] have some deficiencies which the present results remedy (for example, the bounds in [3] offer no guarantee of sharpness). We also note that the approximating geometric sum we choose here stochastically dominates our hitting time, and explore implications of this in Section 2. Finally, in Section 3 we conclude with some remarks about the construction of the strong stationary times needed for our approximating geometric sum.

Throughout this note we let the $t$-step transition probabilities of our Markov chain $X$ be denoted by $P^{t}(k,l)=\mathbb{P}(X_{t}=l|X_{0}=k)$ for $k,l\in S$. For a state $j\in S$, we define the distribution $\pi^{(j)}$ as the stationary distribution $\pi$ restricted to the states other than $j$: $\pi^{(j)}=\frac{\pi-\pi_{j}\delta_{j}}{1-\pi_{j}}$, where $\delta_{j}$ is the Dirac delta. We will write $N\sim\mbox{Geom}(p)$ if $\mathbb{P}(N=k)=p(1-p)^{k}$ for $k=0,1,2,\ldots$, and let ‘$\stackrel{{\scriptstyle d}}{{=}}$’ denote equality in distribution

A strong stationary time $T$ will be called ‘fastest’ if it is stochastically smaller than any other strong stationary time. Note that Proposition 3.2 of Aldous and Diaconis [1] guarantees the existence of such a fastest strong stationary time for any Markov chain $X$ of the type we consider here.

The tails of strong stationary times are closely related to separation, which may be used to estimate how far $X_{t}$ is from stationarity. The separation at time $t$ is defined by

[TABLE]

As with strong stationary times, for convenience we suppress dependence on the initial distribution of $X$ in the notation.

The following lemma (see Proposition 3.2 of [1]) says that tails of a fastest strong stationary time achieve separation.

Lemma 1.1** ([1]).**

There exists a strong stationary time $T$ for $X$ such that $\mathbb{P}(T>t)=s(t)$ for all $t\geq 0$.

We now give a lemma which collects facts about strong stationary times from the literature which we will need in what follows. Although these facts are known, we give a short proof for the benefit of the reader, and to motivate the work that follows (in particular, the proof of Theorem 1.3).

Lemma 1.2**.**

Let $j\in S$ be such that $\pi_{y}\mathbb{P}(X_{t}=j)\leq\pi_{j}\mathbb{P}(X_{t}=y)$, for all $t\geq 0$ and $y\in S$, when $X_{0}\sim\pi^{(j)}$. Then $P^{t}(j,j)$ is decreasing in $t$ and, letting the random variable $T^{(j)}$ be defined by

[TABLE]

$T^{(j)}$* has the distribution of a fastest strong stationary time for $X$ with initial distribution $X_{0}\sim\pi^{(j)}$. Furthermore, $\mathbb{P}(T^{(j)}>t,X_{t}=j)=0$ for all $t\geq 0$.*

Proof.

Under the given condition on state $j$, when $X_{0}\sim\pi^{(j)}$,

[TABLE]

Since $s(t)$ is known to be decreasing in $t$ (see, for example, Chapter 9 of [2]), we have that $P^{t}(j,j)$ is decreasing. By Lemma 1.1, $T^{(j)}$ as defined has the distribution of a fastest strong stationary time. Finally,

[TABLE]

where the second equality follows from Lemma 6.9 of [7], the third by the assumption made on state $j$, and the final equality follows from Lemma 1.1. ∎

Using Lemma 1.2, Theorem 1.3 below gives an explicit link between strong stationary times and geometric sums. Its conclusion is the same as that of Theorem 4.2 of Fill and Lyzinski [6], though the stated conditions are somewhat stronger (see Remark 1.4 below), and the proof uses different techniques. We give it here to motivate the main results of this paper that will follow in Section 2.

Theorem 1.3**.**

Let $W_{j}=\inf\{t\geq 0:X_{t}=j\}$, the first time the Markov chain $X$ reaches the state $j$, where we assume the initial distribution $X_{0}\sim\pi$. Assume that the state $j$ satisfies $\pi_{y}\mathbb{P}(X_{t}=j)\leq\pi_{j}\mathbb{P}(X_{t}=y)$, for all $t\geq 0$ and $y\in S$, when $X_{0}\sim\pi^{(j)}$. Let the random variable $T^{(j)}$ be defined as in Lemma 1.2. Then $W_{j}\stackrel{{\scriptstyle d}}{{=}}T^{(j)}_{1}+\cdots+T^{(j)}_{N}$, where $N\sim\mbox{Geom}(\pi_{j})$ and $T^{(j)},T^{(j)}_{1},T^{(j)}_{2},\ldots$ are IID.

Proof.

Using the characterization of geometric sums presented in Section 2 of Daly [3], and the fact that $\mathbb{P}(W_{j}=0)=\pi_{j}$, the conclusion follows on showing that $W_{j}+T^{(j)}\stackrel{{\scriptstyle d}}{{=}}(W_{j}|W_{j}>0)$.

From Lemma 1.2, $T^{(j)}$ has the distribution of a fastest strong stationary time for $X$ with initial distribution $X_{0}\sim\pi^{(j)}$. Note that we may construct $(W_{j}|W_{j}>0)$ as the first hitting time of the state $j$ in a copy of the Markov chain $X$ started according to this initial distribution. Also, by Lemma 1.2, the probability that this Markov chain visits state $j$ before time $T^{(j)}$ is zero, at which point we have achieved stationarity, and the subsequent time needed before it visits state $j$ has the same distribution as the original hitting time $W_{j}$ from stationarity. ∎

Remark 1.4**.**

Theorem 4.2 of Fill and Lyzinski [6] also gives conditions under which the hitting time $W_{j}$ is distributed as a geometric sum. Their result is stronger, as they work under the assumption that $P^{t}(j,j)$ is decreasing in $t$ in the place of our assumption that $\pi_{y}\mathbb{P}(X_{t}=j)\leq\pi_{j}\mathbb{P}(X_{t}=y)$ for all $t$ and $y$. Lemma 1.2 shows that our assumption is more restrictive than that of Fill and Lyzinski.**

One of our aims in this note is to find an approximate version of Theorem 1.3, in the sense that we wish to quantify explicitly the departure of the distribution of $W_{j}$ from that of a geometric sum, as well as deriving other bounds related to this approximation. We will do this in Section 2 below.

2 Approximation of Markov chain hitting times

Let $W_{j}=\inf\{t\geq 0:X_{t}=j\}$ be the Markov chain hitting time defined in Theorem 1.3, and let $T^{(j)}$ be a strong stationary time for the Markov chain $X$ with initial distribution $X_{0}\sim\pi^{(j)}$. Our main gaol in this section is to consider the approximation of $W_{j}$ by the geometric sum $U=T^{(j)}_{1}+\cdots+T^{(j)}_{N}$, where $N\sim\mbox{Geom}(\pi_{j})$ and $T^{(j)},T^{(j)}_{1},T^{(j)}_{2},\ldots$ are IID. We firstly do this by bounding the total variation distance $d_{TV}(\mathcal{L}(W_{j}),\mathcal{L}(U))$, defined by

[TABLE]

where the infimum is taken over all couplings of $W_{j}$ and $U$. Following the work of Daly [3], to derive such a bound we construct an integer-valued random variable $Y$ (which may depend on $T^{(j)}$) such that $Y+T^{(j)}\geq 0$ and

[TABLE]

Proposition 3.2 of [3] then gives us that

[TABLE]

Remark 2.1**.**

The results of [3] are stated under the assumption that the support of $T^{(j)}$ is bounded. This assumption may be removed using by making suitable adjustments to the proof of Theorem 2.1 of that paper. Specifically, we need analogues of Lemmas 4.1 and 4.2 of [3] which may be applied when the functions involved are not polynomials of bounded degree. We note that a suitable analogue of Lemma 4.1 follows from Rouché’s theorem, and that Lemma 4.2 may be generalised by applying the residue theorem and a suitable change of variable. We note that this allows us to remove the assumption that the compounding random variable has bounded support from all the results of [3], including the approximation results for Markov chain hitting times.**

Write $\widetilde{W_{j}}=(W_{j}|W_{j}>0)$. As in Section 1, we may construct $\widetilde{W_{j}}$ as the time of the first visit of the Markov chain $X$ to the state $j$ when initialized with distribution $X_{0}\sim\pi^{(j)}$. This hitting time may come either before the strong stationary time $T^{(j)}$ for this chain, or not. On the event that $\widetilde{W_{j}}<T^{(j)}$, we set $Y=\widetilde{W_{j}}-T^{(j)}$; otherwise we have achieved stationarity at time $T^{(j)}$, and the remaining time until we reach state $j$ is distributed as $W_{j}$, and we may set $Y=W_{j}$. Here we construct $W_{j}$ using a Bernoulli random variable $\xi\sim\mbox{Be}(\pi_{j})$ (independent of all else) with $\mathbb{P}(\xi=1)=1-\mathbb{P}(\xi=0)=\pi_{j}$, and set $W_{j}=0$ if $\xi=1$; otherwise we set $W_{j}=\widetilde{W_{j}}$.

We therefore have that

[TABLE]

Evaluating this bound, (using the fact that $T^{(j)}$ is a strong stationary time) we have

[TABLE]

where the final equality follows from the identity $\pi_{j}\mathbb{E}W_{j}=\sum_{t=0}^{\infty}\left[P^{t}(j,j)-\pi_{j}\right]$ given in Proposition 10.19 of [7]. We then arrive at the following.

Theorem 2.2**.**

Let $W_{j}=\inf\{t\geq 0:X_{t}=j\}$, where we assume that $X_{0}\sim\pi$. Let $T^{(j)}$ be a strong stationary time (independent of $W_{j}$) for the Markov chain $X$ with initial distribution $X_{0}\sim\pi^{(j)}$. Let $U=T^{(j)}_{1}+\cdots+T^{(j)}_{N}$, where $N\sim\mbox{Geom}(\pi_{j})$ and $T^{(j)},T^{(j)}_{1},T^{(j)}_{2},\ldots$ are IID. Then

[TABLE]

Note that Theorem 1.3 above follows as an immediate corollary of this result: if $j$ is such that $\pi_{y}\mathbb{P}(X_{t}=j)\leq\pi_{j}\mathbb{P}(X_{t}=y)$ for all $t$ and $y$, then Lemma 1.2 gives us that we may take the strong stationary time $T^{(j)}$ to have distribution

[TABLE]

and so the right-hand side of (1) is zero (again, using Proposition 10.19 of [7]).

Daly [3] also treats the problem of approximation of the random variable $W_{j}$ by a geometric sum (with the same choice of geometric random variable $N$, but a different choice of compounding random variable). Daly made his choice of compounding distribution to reflect (in a certain sense) the minimum time needed for the Markov chain to make a jump to the state $j$, while we have chosen $T^{(j)}$ to reflect the time needed to achieve stationarity. This gives us a fundamentally different approximating geometric sum, and a result which allows us to derive bounds which reflect cases in which the hitting time $W_{j}$ is exactly distributed as a geometric sum (as in Theorem 1.3). There are no such known results corresponding to the approximation for Markov chain hitting times given in [3]. Some illustration of this is provided by Example 2.3 below.

Example 2.3**.**

Let $X$ have state space $\{0,1\}$ and transition matrix

[TABLE]

for some $0\leq\delta<1/2$, so that $\pi_{0}=\frac{1-2\delta}{2(1-\delta)}$, $\pi_{1}=\frac{1}{2(1-\delta)}$, and

[TABLE]

We let $W_{1}=\inf\{t\geq 0:X_{t}=1\}$. Note that $P^{t}(1,1)$ is decreasing in $t$, so Theorem 4.2 of Fill and Lyzinski [6] gives that $W_{1}\stackrel{{\scriptstyle d}}{{=}}T^{(1)}_{1}+\cdots+T^{(1)}_{N}$, where $N\sim\mbox{Geom}(\pi_{1})$, and $T^{(1)},T^{(1)}_{1},T^{(1)}_{2},\ldots$ are IID with $\mathbb{P}(T^{(1)}>t)=\delta^{t}$. As one would expect, it is easily checked that the upper bound of Theorem 2.2 is zero, also reflecting that fact that $W_{1}$ is distributed as a geometric sum. However, Theorem 3.1 of Daly [3], which also considers approximation of Markov chain hitting times by geometric sums, gives only the bound

[TABLE]

i.e., the approximating geometric sum chosen by that result is the geometric random variable $N$, and that theorem does not reflect the fact that $W_{1}$ is itself distributed as a geometric sum.**

In Theorem 2.4 below, we note that the approximating geometric sum $U$ is stochastically larger than the hitting time $W_{j}$. We use ‘$\geq_{st}$’ to denote the usual stochastic ordering, defined for random variables $Y$ and $Z$ by $Y\geq_{st}Z$ if $\mathbb{E}h(Y)\geq\mathbb{E}h(Z)$ for all increasing functions $h$. Some consequences of this for bounding hitting times and strong stationary times will be considered following the proof of the theorem.

Theorem 2.4**.**

Let $W_{j}=\inf\{t\geq 0:X_{t}=j\}$, where we assume that $X_{0}\sim\pi$. Let $T^{(j)}$ be a strong stationary time (independent of $W_{j}$) for the Markov chain $X$ with initial distribution $X_{0}\sim\pi^{(j)}$. Let $U=T^{(j)}_{1}+\cdots+T^{(j)}_{N}$, where $N\sim\mbox{Geom}(\pi_{j})$ and $T^{(j)},T^{(j)}_{1},T^{(j)}_{2},\ldots$ are IID. Then $U\geq_{st}W_{j}$.

Proof.

Following equation (3.3) of Daly [3], we write

[TABLE]

where $Y$ is the random variable constructed in the proof of Theorem 2.2, $f_{h}(0)=0$, and

[TABLE]

for $x>0$,where $V_{r}=V_{0}+T^{(j)}_{1}+\cdots+T^{(j)}_{r}$. From (3), we can check that if $h$ is increasing, then $f_{h}$ is decreasing. Since we have constructed $Y$ in the proof of Theorem 2.2 in such a way that $Y\leq W_{j}$ almost surely, using (2) then gives us the desired inequality. ∎

Corollary 2.5**.**

Let $W_{j}$ and $T^{(j)}$ be as in Theorem 2.4. Then

1. (i)

[TABLE]

where $T_{(l)}$ is a strong stationary time for $X$ with initial distribution $X_{0}\sim\delta_{l}$. 2. (ii)

If $\theta>0$ is such that $(1-\pi_{j})\mathbb{E}e^{\theta T^{(j)}}<1$, then

[TABLE]

Proof.

The first part of (i) and (ii) follow immediately from the standard choices $h(x)=x$ and $h(x)=e^{\theta x}$ (for $\theta>0$), respectively, in Theorem 2.4. Note that these inequalities are sharp, given our previous comments about cases where $W_{j}$ is equal in distribution to a geometric sum.

For the second inequality in (i), we apply the first inequality and choose $T^{(j)}$ to be the fastest strong stationary time. With this choice, using Lemma 1.1,

[TABLE]

The desired bound follows by summing each side of the above inequality. ∎

Remark 2.6**.**

We could combine Corollary 2.5 with a stochastic lower bound on $W_{j}$ to derive inequalities for the strong stationary time $T^{(j)}$ defined in Theorem 2.4. For example, noting that (when constructing $W_{j}$ with $X_{0}\sim\pi$), from state $s\not=j$ it takes at least a geometrically distributed number of steps (with parameter $1-P(s,s)$) before the chain reaches state $j$, we have that $W_{j}$ is stochastically larger than $I(\xi=0)(1+\eta)$, where $\xi\sim\mbox{Be}(\pi_{j})$ and $\eta|X_{0}\sim\mbox{Geom}(1-P(X_{0},X_{0}))$ (with $X_{0}\sim\pi^{(j)}$) are independent of all else. Since the expectation of this random variable is a lower bound for the expectation of $W_{j}$, we may combine this observation with Corollary 2.5(i) to obtain a lower bound for the expectation of the strong stationary time $T^{(j)}$ in that result:

[TABLE]

An analogous lower bound could also be derived for the moment generating function of $T^{(j)}$ based on Corollary 2.5(ii).**

Defining the average hitting time $\overline{w}=\sum_{j\in S}\pi_{j}\mathbb{E}W_{j}$, we note that Corollary 2.5(i) gives the bound

[TABLE]

where $t^{*}=\sup_{l\in S}\mathbb{E}T_{(l)}$ is the worst-case expected strong stationary time. As a consequence of this, we note that $(|S|-1)t^{*}/n$ serves as an upper bound on the total variation distance between the ergodic average of the Markov chain $X$ up to time $n$ and the stationary distribution $\pi$; see Corollary 3 of Roberts and Rosenthal [9].

We conclude this section by noting that in some cases it is possible to identify the state which achieves the worst case expected strong stationary time, as in the following.

Proposition 2.7**.**

Let $X$ be a reversible Markov chain with transition matrix $P$, and let the state $m\in S$ be such that $\pi_{m}\geq\pi_{l}$ for all $l\in S$. Let $T_{(l)}$ be a fastest strong stationary time for $X$ with $X_{0}\sim\delta_{l}$. If the minimum element of $P^{t}$ occurs in the column corresponding to transitions into state $m$ for each $t\geq 1$, then $T_{(m)}\geq_{st}T_{(l)}$ for each $l\in S$.

Proof.

Since $X$ is reversible, we have

[TABLE]

for each $k,l\in S$ and $t\geq 1$. Also, by assumption, $\inf_{k}P^{t}(k,m)\leq\inf_{k}P^{t}(k,l)$ for any $l\in S$. Using Lemma 1.1, applying these two inequalities, and using reversibility of $X$, we have

[TABLE]

for any $l\in S$. ∎

3 Greedy construction of $T^{(j)}$

In Section 2, we have considered the approximation of the hitting time $W_{j}$ by a geometric sum $U$ whose compounding distribution is that of $T^{(j)}$, a strong stationary time for $X$ with $X_{0}\sim\pi^{(j)}$. We have already given some bounds which may be useful in the approximation of $T^{(j)}$; we now comment on the explicit construction of this random variable using the greedy approach of Section 3.4 of [4]. Although this approach offers no guarantee of constructing the fastest strong stationary time, which is the one which minimizes the upper bound of Theorem 2.2, Diaconis and Fill [4] note that strong stationary times constructed in this way do often achieve this in the examples they consider. We note this construction here since the choice $X_{0}\sim\pi^{(j)}$ gives the constructed $T^{(j)}$ a particularly simple structure, which may lead to explicit evaluation of the distribution of this choice of $T^{(j)}$.

Following the work of [4], we construct $T^{(j)}$ as the first passage time of a dual Markov chain $X^{*}$ with state space $S^{*}=\{A\subseteq S:A\not=\emptyset\}$ to the state $S\in S^{*}$. We specify the Markov chain $X^{*}$ through its initial distribution $\nu^{(j)}$ and transition matrix $P^{*}$. These are related to $\pi^{(j)}$ and $P$ through an intertwining matrix $\Lambda$:

[TABLE]

Letting $S_{j}=S\setminus\{j\}\in S^{*}$, the greedy construction of [4] gives that $X^{*}$ is initialised in state $S_{j}$ with probability 1, that is, $\nu^{(j)}=\delta_{S_{j}}$. Hence, the row of $\Lambda$ corresponding to $S_{j}$ contains the vector $\pi^{(j)}$, and we may place the vector $\pi$ along every other row of $\Lambda$. With this choice, it is straightforward to check that for any $A\in S^{*}$ with $A\not=S_{j}$, the row of $\Lambda P$ corresponding to $A$ also consists of the vector $\pi$, and hence the row of the transition matrix $P^{*}$ corresponding to $A$ consists of a 1 in the column corresponding to $S$, and 0s elsewhere. It remains only to calculate the entries $P^{*}(S_{j},A)$ for $A\in S^{*}$, i.e., the row of $P^{*}$ corresponding to $S_{j}$.

Using the above construction of $\Lambda$, we calculate that, for each $l\in S$,

[TABLE]

This gives us the following construction of $P^{*}(S_{j},A)$, for $A\in S^{*}$: Begin by defining $A_{0}=S$ and $Q_{0}(S_{j},l)=\frac{1}{1-\pi_{j}}[\pi_{l}-\pi_{j}P(j,l)]$. Then, having defined $A_{r-1}$ and $Q_{r-1}$ (for $r\geq 1$), let

[TABLE]

If the set over which this minimum is taken is empty, stop and set $z=r-1$. Otherwise, define

[TABLE]

and $Q_{r}(S_{j},\cdot)=Q_{r-1}(S_{j},\cdot)-c_{r}\pi_{\cdot}$ on $A_{r}$, and continue this recursive procedure. On termination of this procedure, we have a sequence of elements of $S^{*}$, $A_{z}\subseteq A_{z-1}\subseteq\cdots\subseteq A_{1}$. We define $P^{*}(S_{j},A_{r})=c_{r}\sum_{l\in A_{r}}\pi_{l}$ for each $A_{r}$, and $P^{*}(S_{j},A)=0$ for any $A\in S^{*}$ which is not equal to one of the $A_{r}$.

This construction gives a particularly simple Markov chain $X^{*}$: We start deterministically in the state $S_{j}$, and from every state apart from $S_{j}$ we may jump only to the absorbing state $S$. Considering the first transition made by $X^{*}$ allows us to calculate $\mathbb{E}T^{(j)}$ explicitly, since it is clear that

[TABLE]

We hence obtain the following.

Proposition 3.1**.**

For $T^{(j)}$ constructed as above,

[TABLE]

Example 3.2**.**

Suppose, for simplicity, that

[TABLE]

for each $l\in S$, and that the minimum $\min_{l\in S}\left(\frac{\pi_{l}-\pi_{j}P(j,l)}{\pi_{l}}\right)$ is uniquely achieved at $l=j$ (i.e., the choice $l=j$ achieves this minimum, and there is no other state $k\in S$ to do so). This final assumption is along the lines of that made in Theorem 1.3, though is considerably weaker (since we look only at the one-step transition probabilities here).**

Following the greedy construction outlined above, under these assumptions we have that $c_{1}=\frac{1-P(j,j)}{1-\pi_{j}}$ and $A_{1}=S$. Then

[TABLE]

and $A_{2}=S_{j}$. The procedure continues, but note that we have now determined the entries $P^{*}(S_{j},S)$ and $P^{*}(S_{j},S_{j})$ of the transition matrix $P^{*}$ which (thanks to the simple structure of the Markov chain $X^{*}$) are all that are required for calculations. Letting $\alpha=\frac{1-P(j,j)}{1-\pi_{j}}$ and $\beta=(1-\pi_{j})c_{2}$, we may then use the underlying structure of $X^{*}$ to calculate that $\mathbb{P}(T^{(j)}=1)=\alpha$ and $\mathbb{P}(T^{(j)}=t)=(1-\alpha)(1-\beta)\beta^{t-2}$ for $t\geq 2$.**

Example 3.3**.**

Suppose, as in the previous example, that (4) holds for each $l\in S$, but now assume that there is some $l^{*}\not=j$ such that $\min_{l\in S}\left(\frac{\pi_{l}-\pi_{j}P(j,l)}{\pi_{l}}\right)=\frac{\pi_{l^{*}}-\pi_{j}P(j,l^{*})}{\pi_{l^{*}}}$. As above we have $A_{1}=S$, but now we know that $l^{*}\not\in A_{2}$ and so $S_{j}\not\subseteq A_{2}$. Since the sets $A_{r}$ are decreasing in $r$, we thus have that $S_{j}\not=A_{r}$ for any $r$, and so $P^{*}(S_{j},S_{j})=0$. Letting $\gamma=P^{*}(S_{j},S)$, we therefore have that $\mathbb{P}(T^{(j)}=1)=\gamma=1-\mathbb{P}(T^{(j)}=2)$.**

Acknowledgements: The authors thanks Jim Fill for insightful discussions which began this work, and the organisers of Stochastic Models VI in June 2018, where those discussions took place. Thanks are also due to Robert Gaunt and Xiong Jin for the observations in Remark 2.1.