A new matrix-infinite-product-form solution for upper block-Hessenberg Markov chains and its quasi-algorithmic constructibility

TL;DR

This paper introduces a novel matrix-infinite-product-form solution for upper block-Hessenberg Markov chains that does not require parameter sets or conditions, and features a quasi-algorithmic construction method.

Contribution

It proposes a new MIP-form solution that simplifies computation and removes the need for certain parameter conditions, with a quasi-algorithmic constructibility feature.

Findings

The new MIP-form solution requires no parameter set or convergence conditions.

It is constructed via an iterative recursive procedure with finite complexity per iteration.

The solution is applicable to stationary distribution computation in UBH-MCs.

Abstract

This paper presents a new matrix-infinite-product-form (MIP-form) solution for the stationary distribution in upper block-Hessenberg Markov chains (UBH-MCs). The existing MIP-form solution (Masuyama, Queueing Syst., Vol. 92, 2019, pp. 173--200) requires a certain parameter set that satisfies both a Foster-Lyapunov drift condition and a convergence condition. In contrast, the new MIP-form solution requires no such parameter sets and no other conditions. The new MIP-form solution also has "quasi-algorithmic constructibility", which is a newly introduced feature of being constructed by iterating infinitely many times a recursive procedure of finite complexity per iteration. This feature is not found in the other solutions for the stationary distribution in UBH-MCs.

Equations

$$\mathbb{S}=\bigcup _{k\in {\mathbb{Z}}_{+}}\{k\}\times {\mathbb{M}}_k,$$

$$-\infty <q(k,i;k,i)$$

$$0\le q(k,i;\ell ,j)$$

$$\sum _{(\ell ,j)\in \mathbb{S}}q(k,i;\ell ,j)$$

$$\bm{Q}=\hbox{}\;\vbox{\kern 58.5733pt\hbox{$\kern 123.57433pt\kern-8.75pt\left(\kern-123.57433pt\vbox{\kern-58.5733pt\vbox{\halign{$#$\hfil\kern 2\p@\kern\@tempdima& \thinspace\hfil$#$\hfil&& \quad\hfil$#$\hfil\cr\hfil\crcr\kern-12.0pt\cr$\hfil\kern 2.0pt\kern 8.75pt&\mathbb{L}_{0}&\mathbb{L}_{1}&\mathbb{L}_{2}&\mathbb{L}_{3}&\cdots\crcr\kern 2.0pt\cr\mathbb{L}_{0}$\hfil\kern 2.0pt\kern 8.75pt&\bm{Q}_{0,0}&\bm{Q}_{0,1}&\bm{Q}_{0,2}&\bm{Q}_{0,3}&\cdots\cr\mathbb{L}_{1}$\hfil\kern 2.0pt\kern 8.75pt&\bm{Q}_{1,0}&\bm{Q}_{1,1}&\bm{Q}_{1,2}&\bm{Q}_{1,3}&\cdots\cr\mathbb{L}_{2}$\hfil\kern 2.0pt\kern 8.75pt&\bm{O}&\bm{Q}_{2,1}&\bm{Q}_{2,2}&\bm{Q}_{2,3}&\cdots\cr\mathbb{L}_{3}$\hfil\kern 2.0pt\kern 8.75pt&\bm{O}&\bm{O}&\bm{Q}_{3,2}&\bm{Q}_{3,3}&\cdots\cr~{}\vdots$\hfil\kern 2.0pt\kern 8.75pt&\vdots&\vdots&\vdots&\vdots&\ddots\crcr\cr}}\kern-12.0pt}\,\right)$}},$$

$$\mathbf{\pi }\mathbf{Q}=0,\mathbf{\pi }\mathbf{e}=1,\mathbf{\pi }>0,$$

$$\mathbf{\pi }$$

$${\mathbf{\pi }}^{(n)}=\frac{({\mathbf{\pi }}_0,{\mathbf{\pi }}_1,\dots ,{\mathbf{\pi }}_n)}{{\sum }_{\ell =0}^n{\mathbf{\pi }}_{\ell }\mathbf{e}},$$

$$\displaystyle\left\{\begin{array}[]{l}\bm{\pi}_{k}=\bm{\pi}_{0}\bm{R}_{1}\bm{R}_{2}\cdots\bm{R}_{k},\qquad k\in\mathbb{N},\\ \bm{\pi}_{0}\left(\bm{Q}_{0,0}+\bm{R}_{1}\bm{Q}_{1,0}\right)=\bm{0},\\ \bm{\pi}_{0}\bm{e}+\bm{\pi}_{0}\displaystyle\sum_{k=1}^{\infty}\bm{R}_{1}\bm{R}_{2}\cdots\bm{R}_{k}\bm{e}=1.\end{array}\right.$$

$${\mathbf{Q}}_{k-1,k}+{\mathbf{R}}_k{\mathbf{Q}}_{k,k}+{\mathbf{R}}_k{\mathbf{R}}_{k+1}{\mathbf{Q}}_{k+1,k}=\mathbf{O},k\in \mathbb{N}.$$

$${\mathbf{R}}_k={\mathbf{Q}}_{k-1,k}(-{\mathbf{Q}}_{k,k}-{\mathbf{R}}_{k+1}{\mathbf{Q}}_{k+1,k}{)}^{-1},k\in \mathbb{N}.$$

$$\prod_{m=k}^{n}\downarrow\bm{A}_{m}=\left\{\begin{array}[]{ll}\bm{A}_{n}\bm{A}_{n-1}\cdots\bm{A}_{k},&k\leq n,\\ \bm{I},&k>n.\end{array}\right.$$

$$\mathbf{\pi }=\underset{n\to \infty }{\lim }{\left(\frac{{\mathbf{\alpha }}_n^{†}{\mathbf{U}}_n^{\ast }{\prod }_{m=k}^{n-1}\downarrow {\mathbf{U}}_m}{{\mathbf{\alpha }}_n^{†}{\mathbf{U}}_n^{\ast }{\sum }_{\ell =0}^n({\prod }_{m=\ell }^{n-1}\downarrow {\mathbf{U}}_m)\mathbf{e}}\right)}_{k\in {\mathbb{Z}}_{[0,n]}},$$

$$\mathbf{Q}\mathbf{v}\le -\mathbf{e}+b{1}_{\mathbb{C}},$$

$$1_{\mathbb{A}}(k,i)=\left\{\begin{array}[]{l@{~~~}l}1,\hfil~{}~{}~{}&(k,i)\in\mathbb{A},\\ 0,\hfil~{}~{}~{}&(k,i)\in\mathbb{S}\setminus\mathbb{A}.\end{array}\right.$$

$$\mathbf{\pi }|\mathbf{Q}|\mathbf{v}=\sum _{(n,i)\in \mathbb{S}}\pi (n,i)|q(n,i;n,i)|v(n,i)<\infty ,$$

$$\sum _{\ell =n+1}^{\infty }{\mathbf{Q}}_{k,\ell }{\mathbf{v}}_{\ell },n\in {\mathbb{Z}}_{+},k\in {\mathbb{Z}}_{[0,n]},$$

$$\mathbf{\pi }:=\underset{\genfrac{}{}{0ex}{}{n\to \infty }{n\ge K}}{\lim }({}_{(n)}{\hat{\mathbf{\pi }}}^{\ast }{)}_{k\in {\mathbb{Z}}_{[0,n]}}:=\underset{\genfrac{}{}{0ex}{}{n\to \infty }{n\ge K}}{\lim }{\left(\frac{{\mathbf{\alpha }}_n^{\ast }{\mathbf{U}}_n^{\ast }{\prod }_{m=k}^{n-1}\downarrow {\mathbf{U}}_m}{{\mathbf{\alpha }}_n^{\ast }{\mathbf{U}}_n^{\ast }{\sum }_{\ell =0}^n({\prod }_{m=\ell }^{n-1}\downarrow {\mathbf{U}}_m)\mathbf{e}}\right)}_{k\in {\mathbb{Z}}_{[0,n]}},$$

$$\Vert {\mathbf{h}}_1-{\mathbf{h}}_2{\Vert }_1:=\sum _{j\in {\mathbb{G}}_1\cap {\mathbb{G}}_2}|h_1(j)-h_2(j)|+\sum _{j\in {\mathbb{G}}_1\setminus {\mathbb{G}}_2}|h_1(j)|+\sum _{j\in {\mathbb{G}}_2\setminus {\mathbb{G}}_1}|h_2(j)|,$$

$$\displaystyle\hskip 0.50003pt{}_{(n)}\bm{Q}=\hbox{}\;\vbox{\kern 68.46663pt\hbox{$\kern 203.84586pt\kern-8.75pt\left(\kern-203.84586pt\vbox{\kern-68.46663pt\vbox{\halign{$#$\hfil\kern 2\p@\kern\@tempdima& \thinspace\hfil$#$\hfil&& \quad\hfil$#$\hfil\cr\hfil\crcr\kern-12.0pt\cr$\hfil\kern 2.0pt\kern 8.75pt&\mathbb{L}_{0}&\mathbb{L}_{1}&\mathbb{L}_{2}&\cdots&\mathbb{L}_{n-2}&\mathbb{L}_{n-1}&\mathbb{L}_{n}\crcr\kern 2.0pt\cr\mathbb{L}_{0}$\hfil\kern 2.0pt\kern 8.75pt&\bm{Q}_{0,0}&\bm{Q}_{0,1}&\bm{Q}_{0,2}&\cdots&\bm{Q}_{0,n-2}&\bm{Q}_{0,n-1}&\bm{Q}_{0,n}\cr\mathbb{L}_{1}$\hfil\kern 2.0pt\kern 8.75pt&\bm{Q}_{1,0}&\bm{Q}_{1,1}&\bm{Q}_{1,2}&\cdots&\bm{Q}_{1,n-2}&\bm{Q}_{1,n-1}&\bm{Q}_{1,n}\cr\mathbb{L}_{2}$\hfil\kern 2.0pt\kern 8.75pt&\bm{O}&\bm{Q}_{2,1}&\bm{Q}_{2,2}&\cdots&\bm{Q}_{2,n-2}&\bm{Q}_{2,n-1}&\bm{Q}_{2,n}\cr~{}\vdots$\hfil\kern 2.0pt\kern 8.75pt&\vdots&\vdots&\vdots&\ddots&\vdots&\vdots&\vdots\cr\mathbb{L}_{n-1}$\hfil\kern 2.0pt\kern 8.75pt&\bm{O}&\bm{O}&\bm{O}&\cdots&\bm{Q}_{n-1,n-2}&\bm{Q}_{n-1,n-1}&\bm{Q}_{n-1,n}\cr\mathbb{L}_{n}$\hfil\kern 2.0pt\kern 8.75pt&\bm{O}&\bm{O}&\bm{O}&\cdots&\bm{O}&\bm{Q}_{n,n-1}&\bm{Q}_{n,n}\crcr\cr}}\kern-12.0pt}\,\right)$}}.$$

$$\hskip 0.50003pt{}_{(n)}\widehat{\bm{\alpha}}=\hbox{}\;\vbox{\kern 17.28775pt\hbox{$\kern 38.34186pt\kern-8.75pt\left(\kern-38.34186pt\vbox{\kern-17.28775pt\vbox{\halign{$#$\hfil\kern 2\p@\kern\@tempdima& \thinspace\hfil$#$\hfil&& \quad\hfil$#$\hfil\cr\hfil\crcr\kern-12.0pt\cr$\hfil\kern 2.0pt\kern 8.75pt&\mathbb{S}_{n-1}&\mathbb{L}_{n}\crcr\kern 2.0pt\cr$\hfil\kern 2.0pt\kern 8.75pt&\bm{0}&\bm{\alpha}_{n}\crcr\cr}}\kern-12.0pt}\,\right)$}},$$

$${}_{(n)}\hat{\mathbf{Q}}={}_{(n)}\mathbf{Q}-{}_{(n)}\mathbf{Q}\mathbf{e}{}_{(n)}\hat{\mathbf{\alpha }},n\in {\mathbb{Z}}_{+}.$$

$${}_{(n)}\hat{\mathbf{\pi }}=\frac{{}_{(n)}\hat{\mathbf{\alpha }}(-{}_{(n)}\mathbf{Q}{)}^{-1}}{{}_{(n)}\hat{\mathbf{\alpha }}(-{}_{(n)}\mathbf{Q}{)}^{-1}\mathbf{e}},n\in {\mathbb{Z}}_{+}.$$

$$\displaystyle\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}=\hbox{}\;\vbox{\kern 20.72223pt\hbox{$\kern 92.7788pt\kern-8.75pt\left(\kern-92.7788pt\vbox{\kern-20.72223pt\vbox{\halign{$#$\hfil\kern 2\p@\kern\@tempdima& \thinspace\hfil$#$\hfil&& \quad\hfil$#$\hfil\cr\hfil\crcr\kern-12.0pt\cr$\hfil\kern 2.0pt\kern 8.75pt&\mathbb{L}_{0}&\mathbb{L}_{1}&\cdots&\mathbb{L}_{n}\crcr\kern 2.0pt\cr$\hfil\kern 2.0pt\kern 8.75pt&\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{0}&\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{1}&\cdots&\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{n}\crcr\cr}}\kern-12.0pt}\,\right)$}}.$$

$${\mathbf{U}}_0^{\ast }$$

$${\mathbf{U}}_n^{\ast }$$

$${\mathbf{U}}_{n,k}^{\ast }$$

$${\mathbf{U}}_n^{\ast }$$

$${\mathbf{U}}_m={\mathbf{Q}}_{m+1,m}{\mathbf{U}}_m^{\ast }.$$

$${\mathbf{U}}_{n,k}^{\ast }$$

$${\mathbf{u}}_n^{\ast }$$

Full text

A new matrix-infinite-product-form solution for upper block-Hessenberg Markov chains and its quasi-algorithmic constructibility111To appear in Advances in Applied Probability, vol. 55, no. 1, March 2023

---

Hiroyuki Masuyama222E-mail: masuyama@sys.i.kyoto-u.ac.jp

Graduate School of Management, Tokyo Metropolitan University, Tokyo 192–0364, Japan.

Abstract

[TABLE]

[TABLE]

1 Introduction

This paper studies the stationary distribution in continuous-time upper block-Hessenberg Markov chains (UBH-MCs). This study focuses on establishing a theoretical procedure for constructing the exact stationary distribution in an algorithmic way, rather than establishing a practical procedure for computing an approximate stationary distribution (of course, such practicality is a significant issue, too). Additionally, this study would be a partial answer to the question: “What is a class of Markov chains whose exact stationary distributions are theoretically constructible by an algorithmic procedure?”

The class of UBH-MCs is characterized by the upper block-Hessenberg (UBH) structure of the infinitesimal generator matrix (called generator for short), and this class includes M/G/1-type Markov chains and level-dependent quasi-birth-and-death processes (LD-QBDs). Remarkably, even a multi-dimensional random walk on the nonnegative lattice is expressed as a UBH-MC such that its level variable is the sum of coordinate components and its phase variable is the set of all the coordinate components but one. Thus, the class of UBH-MCs is a powerful tool for modeling information and communication systems, inventory systems, transportation systems, etc. Moreover, the time-average performance of these systems is evaluated through the stationary distribution. That is why this study focuses on the stationary distribution in UBH-MCs.

We now introduce the definition of UBH-MCs in order to describe the background and purpose of this study. Let $\mathbb{S}$ denote a countable set such that

[TABLE]

where $\mathbb{Z}_{+}=\{0,1,2,\dots\}$ and $\mathbb{M}_{k}=\{1,2,\dots,M_{k}\}\subset\mathbb{N}:=\{1,2,3,\dots\}$. Let $\bm{Q}:=(q(k,i;\ell,j))_{(k,i;\ell,j)\in\mathbb{S}^{2}}$ denote an essential $Q$-matrix (see Definition B.1), and thus

[TABLE]

Assume that $\bm{Q}$ is of UBH form:

[TABLE]

where $\mathbb{L}_{k}:=\{k\}\times\mathbb{M}_{k}$ is called level $k$ and an element $i$ of $\mathbb{M}_{k}$ is called phase $i$ (of level $k$). A Markov chain having this $Q$-matrix $\bm{Q}$ as its generator is said to be an upper block-Hessenberg Markov chain (UBH-MC). Note that a UBH-MC may be called a level-dependent M/G/1-type Markov chain. Note also that if $\bm{Q}_{k,k+m}=\bm{O}$ for all $m\geq 2$ and $k\in\mathbb{Z}_{+}$ then the generator is of block-tridiagonal form and thus the UBH-MC is reduced to an LD-QBD.

We provide a basic assumption and some definitions associated with the stationary distribution vector of the UBH generator $\bm{Q}$ given in (1.1). Assume that $\bm{Q}$ is ergodic (i.e., irreducible and positive recurrent) throughout the paper, unless otherwise stated. Let $\bm{\pi}:=(\pi(k,i))_{(k,i)\in\mathbb{S}}$ denote the unique and positive stationary distribution vector of the ergodic generator $\bm{Q}$ (see, e.g., [1, Chapter 5, Theorems 4.4 and 4.5]). By definition,

[TABLE]

where $\bm{e}$ denotes a column vector of ones with an appropriate dimension. For later reference, let $\bm{\pi}_{k}=(\pi(k,i))_{i\in\mathbb{M}_{k}}$ for $k\in\mathbb{Z}_{+}$, which leads to the level-wise partition of $\bm{\pi}$:

[TABLE]

Furthermore, let $\bm{\pi}^{(n)}$, $n\in\mathbb{Z}_{+}$, denote

[TABLE]

which is referred to as the (finite-level) conditional stationary distribution vector. For any sufficiently large $n$, $\bm{\pi}^{(n)}$ can be considered an approximation to $\bm{\pi}$.

The main purpose of this paper is to present a quasi-algorithmically constructible solution for the stationary distribution vector $\bm{\pi}$ of the ergodic UBH generator $\bm{Q}$ given in (1.1). We here provide our notion (not strict definition) of quasi-algorithmic constructions for the stationary distribution (the stationary distribution vector) in ergodic countable-state Markov chains, which are not, of course, restricted to UBH-MCs.

Notion 1.1

(Quasi-algorithmic constructions for the stationary distribution in countable-state Markov chains.) Consider an ergodic countable-state Markov chain with generator (essential $Q$-matrix) $\cal{Q}$. Let $\pi$ denote the stationary distribution of the ergodic Markov chain. Let $\mathscr{P}$ denote a procedure for sequentially generating tentative solutions $\pi(0),\pi(1),\pi(2),\dots$ for the stationary distribution $\pi$. The procedure $\mathscr{P}$ is said to be a quasi-algorithmic construction of $\pi$ and furthermore $\pi$ is said to be quasi-algorithmic constructible (or have quasi-algorithmic constructibility), if $\mathscr{P}$ has the following features:

1. (i)

The procedure $\mathscr{P}$ generates each tentative solution $\pi(n)$ by using at most a finite (not necessarily bounded) number of elements of $\cal{Q}$ together with (if required) some or all the components of the previous tentative solutions $\{\pi(m);m\in\mathbb{Z}_{[0,n-1]}\}$, where $\mathbb{Z}_{[0,k]}=\{0,1,\dots,k\}$ for $k\in\mathbb{Z}_{+}$. 2. (ii)

It takes at most finite complexity for $\mathscr{P}$ to generate each tentative solution $\pi(n)$. 3. (iii)

The sequence $\{\pi(n);n\in\mathbb{Z}_{+}\}$ converges to $\pi$ in some mathematical sense (e.g., in the $\ell_{1}$-distance).

Remark 1.2

To the best of our knowledge, there is no universal formal definition of “algorithm”. Indeed, the term of “algorithm” is widely used with some ambiguity in various contexts. Nevertheless, it would be generally accepted, as an informal definition, that an “algorithm” is a finite set of operations (of finite computational complexity) to accomplish a particular task. Considering this situation, we introduce the notion of quasi-algorithmic constructions in order to distinguish our new solution presented in this paper from the existing ones for the stationary distribution in UBH-MCs.

Remark 1.3

Any quasi-algorithmic construction can be implemented as an “algorithm” if the construction is equipped with an appropriate stopping criterion so that it stops after finitely many iterations, and if it is possible to store the elements required to compute each tentative solution.

As far as we know, there have been no studies that achieve a quasi-algorithmic construction of the stationary distribution vector in UBH-MCs including LD-QBDs. There are, however, many related studies on the computation of the stationary distribution vector in these Markov chains.

Bright and Taylor [5] presented a nice matrix-product form of the stationary distribution vector $\bm{\pi}=(\bm{\pi}_{0},\bm{\pi}_{1},\dots)$ of the LD-QBD generator:

[TABLE]

The set of matrices $\{\bm{R}_{k};k\in\mathbb{N}\}$ is the minimal nonnegative solution for the system of matrix equations:

[TABLE]

The matrix-product form (1.7) appears easy to compute. Thus, based on it, Bright and Taylor [5] proposed a procedure for approximately computing the stationary distribution vector of the LD-QBD generator. Besides, some researchers use the matrix-product form (1.7) as the foundation of their respective algorithms (see, e.g., [3, 17]).

However, the matrix-product form (1.7) does not lead to any quasi-algorithmic construction of the stationary distribution vector in LD-QBDs. Equation (1.8) yields

[TABLE]

This recursive formula shows that each $\bm{R}_{k}$ can be computed provided that $\bm{R}_{k+1}$ is given. Therefore, to implement the algorithms based on (1.7), we have to truncate the infinite sequence $\{\bm{R}_{k};k\in\mathbb{Z}_{+}\}$ at a sufficiently large $K^{*}\in\mathbb{Z}_{+}$ by letting $\bm{R}_{K^{*}+1}=\bm{O}$. Moreover, owing to this implementation, computing better approximations (to the stationary distribution) requires setting $R$-matrices of higher levels to be zero matrices, which implies that all the components of such an approximation are computed anew from scratch.

Several researchers have studied the approximate computation of the stationary distribution in UBH-MCs. Takine [20] presented an algorithm for the conditional stationary distribution vector $\bm{\pi}^{(n)}$ under some additional conditions, which are removed by Kimura and Takine [6]. As mentioned in Section 1 of [20], the conditional stationary distribution vector $\bm{\pi}^{(n)}$ can be a good approximation to the stationary distribution vector $\bm{\pi}$ for all sufficiently large $n\in\mathbb{N}$. Klimenok and Dudin [10], Li et al. [13], and Shin and Pearce [19] proposed respective algorithms for approximately computing the stationary distribution vector $\bm{\pi}$ in UBH-MCs by making transition rates (or transition probabilities) eventually level independent.

In summary, all the existing algorithms mentioned above are originally designed for approximately computing the stationary distribution vector $\bm{\pi}$ in UBH-MCs (including LD-QBDs). Thus, although those algorithms fulfill practical purposes, they do not make any theoretical contribution to the quasi-algorithmic construction of the stationary distribution vector $\bm{\pi}$.

Unlike those previous studies, Masuyama [16] has taken the first step to the quasi-algorithmic construction of $\bm{\pi}$ in UBH-MCs. To describe the numerical procedure proposed in [16] (and to facilitate later discussion), we introduce the notation: A finite dimensional matrix (resp. vector) is extended (if necessary) to an infinite dimensional matrix (resp. vector) by appending zeros to it with its original elements fixed in their original positions. This notation enables us to define addition, subtraction, and product over vectors and matrices with different sizes. Furthermore, for $k,n\in\{0,\pm 1,\pm 2,\dots\}$, the notation “$\prod_{m=k}^{n}\downarrow$” denotes a product operator of matrices (including scalars) such that

[TABLE]

Masuyama [16] proposed a sequential update algorithm (Algorithm 1 therein) that generates a matrix-infinite-product form (MIP-form) solution for $\bm{\pi}=(\bm{\pi}_{0},\bm{\pi}_{1},\dots)$:

[TABLE]

where (i) $\bm{U}_{n}=\bm{Q}_{n+1,n}\bm{U}_{n}^{*}$; (ii) each $\bm{U}_{n}^{*}$, $n\in\mathbb{Z}_{+}$ is computed in a finite number of operations starting from $\bm{U}_{0}^{*}=(-\bm{Q}_{0,0})^{-1}$; and (iii) $\bm{\alpha}^{{\dagger}}_{n}$ is a $1\times M_{n}$ probability vector that is an optimal solution for a certain linear fractional programing (LFP) problem, Problem 2.3 in Section 2.3. For the reader’s convenience, we summarize the sequential update algorithm [16, Algorithm 1] in Algorithm 1 in Section 4.

The existing MIP-form solution (1.9) (and thus the sequential update algorithm in [16]) requires Conditions 1 and 2 below (see [16, Conditions 1 and 2]).

Condition 1** **(Foster-Lyapunov drift condition)

There exist a constant $b\in(0,\infty)$, a finite set $\mathbb{C}\subset\mathbb{S}$, and a positive column vector $\bm{v}:=(v(k,i))_{(k,i)\in\mathbb{S}}$ such that $\inf_{(k,i)\in\mathbb{S}}v(k,i)>0$ and

[TABLE]

where $\bm{1}_{\mathbb{A}}:=(1_{\mathbb{A}}(k,i))_{(k,i)\in\mathbb{S}}$, $\mathbb{A}\subseteq\mathbb{S}$, denotes a column vector given by

[TABLE]

The space of parameter sets $(\bm{v},b,\mathbb{C})$ satisfying (1.10) is denoted by $\cal{V}$, for convenience.

Condition 2** **(Convergence condition for the sequential update algorithm in [16])

[TABLE]

where, for any matrix $\bm{X}$ (resp. vector $\bm{x}$), the symbol $|\bm{X}|$ (resp. $|\bm{x}|$) denotes the matrix (resp. vector) obtained by taking the absolute value of each element of $\bm{X}$ (resp. $\bm{x}$).

Conditions 1 and 2 diminish the usefulness of the MIP-form solution (1.9). A parameter set $(\bm{v},b,\mathbb{C})\in\cal{V}$ of Condition 1 is required to describe the objective function of Problem 2.3 with optimal solution $\bm{\alpha}^{{\dagger}}_{n}$. In fact, Condition 1 holds if and only if the irreducible generator $\bm{Q}$ is ergodic (see [11, Theorem 1.1]). However, even though we know that $\bm{Q}$ is ergodic, it may not be easy to find $(\bm{v},b,\mathbb{C})\in\cal{V}$ in some cases. In addition, Condition 2 is required to prove the convergence of the MIP-form solution (1.9) (see [16, Theorem 3.2]), though this condition does not necessarily hold for all UBH-MCs (such an example is presented in Appendix A).

We note that the existing MIP-form solution (1.9) is not quasi-algorithmically constructible. For each $n\in\mathbb{Z}_{+}$, the MIP-form solution (1.9) requires an optimal solution $\bm{\alpha}_{n}^{{\dagger}}$ for Problem 2.3, and its objective function includes the infinite sums:

[TABLE]

where $\bm{v}_{\ell}=(v(\ell,i))_{i\in\mathbb{M}_{\ell}}$ is a subvector of $\bm{v}$. Clearly, the infinite sum has infinite computational complexity, in general. Therefore, the existing MIP-form solution (1.9) is not quasi-algorithmically constructible, and computing it requires the truncation of the infinite sums (1.11), which causes truncation error.

This paper presents a new MIP-form solution for $\bm{\pi}=(\bm{\pi}_{0},\bm{\pi}_{1},\dots)$ quasi-algorithmically constructible:

[TABLE]

where $K\in\mathbb{Z}_{+}$ is a free parameter, and where $\bm{\alpha}_{n}^{*}$, $n\geq K$, is a $1\times M_{n}$ probability vector that is an optimal solution for the LFP problem (Problem 3.1 in Section 3) different from Problem 2.3 of the existing MIP-form solution (1.9). The objective function of Problem 3.1 does not include such a parameter set as $(\bm{v},b,\mathbb{C})\in\cal{V}$ and is computed in a finite number of operations without any infinite sum like (1.11). Thus, the new MIP-form solution (1.12) is quasi-algorithmically constructible. In addition, the new solution holds without any additional conditions, such as Condition 2 for convergence, except for the ergodicity of $\bm{Q}$.

The rest of this paper is divided into five sections. Section 2 provides preliminary results including the existing MIP-form solution presented in [16]. Section 3 presents our new MIP-form solution together with its theoretical foundations, which is the main result of this paper. Section 4 considers the advantages of our MIP-form solution over the existing one. Section 5 discusses special cases where our solution can be established more effectively. Finally, Section 6 contains concluding remarks.

2 The existing MIP-form solution

This section reviews the existing MIP-form solution (1.9) to facilitate describing our new MIP-form solution (established in Section 3) and understanding its advantages over the existing one. First, we introduce the foundation of the existing MIP-form solution, the last-block-column linearly augmented (LBC-LA) truncation approximation to the stationary distribution vector $\bm{\pi}$. We then provide a matrix-product form of the LBC-LA truncation approximation. Finally, as a certain limit of the matrix-product form, we express the existing MIP-form solution (1.9) for $\bm{\pi}$.

For later reference, we define the $\ell_{1}$-distance for vectors of different dimensions. Let $\mathbb{G}$ be a countable (possibly infinite) set. Let $\bm{h}_{1}=(h_{1}(j))_{j\in\mathbb{G}_{1}}$ and $\bm{h}_{2}:=(h_{2}(j))_{j\in\mathbb{G}_{2}}$ be real-valued vectors, where $\mathbb{G}_{1}$ and $\mathbb{G}_{2}$ are subsets of $\mathbb{G}$. For vectors $\bm{h}_{1}$ and $\bm{h}_{2}$, let

[TABLE]

which is referred to as the $\ell_{1}$-distance of the difference between $\bm{h}_{1}$ and $\bm{h}_{2}$.

2.1 Definition of the LBC-LA truncation approximation

We begin with some definitions to describe the LBC-LA truncation approximation. Let $\mathbb{S}_{n}=\bigcup_{k=0}^{n}\mathbb{L}_{k}$ for $n\in\mathbb{Z}_{+}$ and $\mathbb{S}_{-1}=\varnothing$ for convenience. For any set $\mathbb{X}$, let $|\mathbb{X}|$ denote the cardinality of $\mathbb{X}$. For any $n\in\mathbb{Z}_{+}$, let $\hskip 0.50003pt{}_{(n)}\bm{Q}$ denote a submatrix of $\bm{Q}$ consisting of the first $|\mathbb{S}_{n}|$ rows and columns, that is,

[TABLE]

Furthermore, let $\hskip 0.50003pt{}_{(n)}\widehat{\bm{\alpha}}$, $n\in\mathbb{Z}_{+}$, denote a $1\times|\mathbb{S}_{n}|$ probability vector that has its probability masses on the last block (corresponding to $\mathbb{L}_{n}$), that is,

[TABLE]

where $\bm{\alpha}_{n}:=(\alpha_{n}(j))_{j\in\mathbb{M}_{n}}$ is a probability vector.

We define the LBC-LA truncation approximation to $\bm{\pi}$ as a stationary distribution vector of the finite essential $Q$-matrix $\hskip 0.50003pt{}_{(n)}\widehat{\bm{Q}}$ given by

[TABLE]

The $Q$-matrix $\hskip 0.50003pt{}_{(n)}\widehat{\bm{Q}}$ is referred to as the last-block-column linearly augmented (LBC-LA) truncation of generator $\bm{Q}$. Clearly, the LBC-LA truncation $\hskip 0.50003pt{}_{(n)}\widehat{\bm{Q}}$ has at least one stationary distribution vector. One of them is given by

[TABLE]

This stationary distribution vector $\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}$ is referred to as the LBC-LA truncation approximation to $\bm{\pi}$, and is partitioned level-wise:

[TABLE]

2.2 A matrix-product form of the LBC-LA truncation approximation

This subsection presents a matrix-product form of the LBC-LA truncation approximation $\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}$. We first introduce some vectors and matrices needed to express the matrix-product form of $\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}$. We then present the matrix-product form together with recursive formulas for the components of it. Finally, we mention a connection of the matrix-product form to the existing MIP-form solution (1.9) for $\bm{\pi}$.

We define the component matrices of the matrix-product form of $\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}$. Let $\bm{U}_{n}^{*}$, $n\in\mathbb{Z}_{+}$, denote an $M_{n}\times M_{n}$ matrix such that its $(i,j)$-th element represents the expected total sojourn time in state $(n,j)$ before the first visit to $\overline{\mathbb{S}}_{n}:=\mathbb{S}\setminus\mathbb{S}_{n}=\cup_{k=n+1}^{\infty}\mathbb{L}_{k}$ (i.e., to any state in levels $n+1,n+2,\dots$) starting from state $(n,i)$. The matrices $\bm{U}_{n}^{*}$, $n\in\mathbb{Z}_{+}$ are determined recursively:

[TABLE]

and, for $n\in\mathbb{N}$,

[TABLE]

Furthermore, let $\bm{U}^{*}_{n,k}$, $n\in\mathbb{Z}_{+}$, $k\in\mathbb{Z}_{[0,n]}$ denote

[TABLE]

It then follows from (2.3) that

[TABLE]

For simplicity of notation, we define $\bm{U}_{m}$, $m\in\mathbb{Z}_{+}$ as

[TABLE]

Using (2.6), we rewrite (2.4) as

[TABLE]

Finally, $\bm{u}^{*}_{n}:=(u_{n}^{*}(i))_{i\in\mathbb{M}_{n}}$, $n\in\mathbb{Z}_{+}$, is defined as the row sum vector of the $|\mathbb{L}_{n}|\times|\mathbb{S}_{n}|$ matrix $(\bm{U}_{n,0}^{*}~{}\bm{U}_{n,1}^{*}~{}\cdots~{}\bm{U}_{n,n}^{*})$:

[TABLE]

where $\bm{u}_{n}^{*}>\bm{0}$ follows from [16, Remark 2.3].

Remark 2.1

The probabilistic interpretation of $\bm{U}_{n}^{*}$ leads to that of $\bm{U}_{n,\ell}^{*}$: Its $(i,j)$-th element represents the expected total sojourn time in state $(\ell,j)$ before the first visit to $\overline{\mathbb{S}}_{n}$ starting from state $(n,i)$. This interpretation implies that the matrix $(\bm{U}_{n,0}^{*}~{}\bm{U}_{n,1}^{*}~{}\cdots~{}\bm{U}_{n,n}^{*})$ is identified with the rows in level $n$ of $(-\hskip 0.50003pt{}_{(n)}\bm{Q})^{-1}$ because this inverse matrix consists of the expected total sojourn times in the respective states in $\mathbb{S}_{n}$ before the first visit to $\overline{\mathbb{S}}_{n}$ starting from the states in $\mathbb{S}_{n}$. We thus have (see [12, Theorem 5.5.1] and [16, Remark 2.2])

[TABLE]

where the second equality is due to (2.6) and (2.7).

We are now ready to present the matrix-product form of $\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}=(\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{0},\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{1},\dots,\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{n})$.

Proposition 2.2** **([16, Equation (2.21) and Lemma 2.2])

For $n\in\mathbb{Z}_{+}$ and $k\in\mathbb{Z}_{[0,n]}$,

[TABLE]

The components $\bm{U}_{n,k}^{*}$ and $\bm{u}_{n}^{*}$ of the matrix-product form (2.10) are the following recursive formulas (see [16, Equations (3.23) and (3.24)]):

[TABLE]

and

[TABLE]

where $\bm{U}_{n}^{*}$ is given in (2.5).

Proposition 2.2 implies that an MIP-form solution for $\bm{\pi}$ can be obtained as a limit of the matrix-product form (2.10), that is,

[TABLE]

However, as mentioned in [16, Section 2.3], achieving such a limit convergent to $\bm{\pi}$ requires choosing the probability vectors $\bm{\alpha}_{n}$, $n\in\mathbb{Z}_{+}$ appropriately. To do this, Masuyama [16] formulated a series of certain linear fractional programing (LFP) problems. The details are given in the next subsection.

2.3 The existing MIP-form solution for the stationary distribution vector

This subsection provides a brief summary of the existing MIP-form solution (1.9) for $\bm{\pi}$. We first show the LFP problem whose optimal solution is the key component of the existing MIP-form solution. We then present the proposition that summarizes the theoretical results on the existing MIP-form solution.

To obtain an MIP-form solution for $\bm{\pi}$, Masuyama [16] formulates the following LFP problem indexed with $n\in\mathbb{Z}_{+}$:

Problem 2.3

[TABLE]

where $\bm{y}_{n}:=(y_{n}(i))_{i\in\mathbb{M}_{n}}$ is given by

[TABLE]

Remark 2.4

Equation (2.8) implies that $\bm{\alpha}_{n}\bm{u}_{n}^{*}>0$ for any feasible solution $\bm{\alpha}_{n}$. Therefore, the objective function of Problem 2.3 is well-defined.

Remark 2.5

If the UBH generator $\bm{Q}$ satisfies $\bm{Q}_{k,k+m}=\bm{O}$ for $m\geq 2$ ($\bm{Q}$ is an LD-QBD generator), then (2.14) is reduced to

[TABLE]

Proposition 2.6 below shows how to find an optimal solution for Problem 2.3 and how to construct, with such a solution, the existing MIP-form solution (1.9), originally presented in [16].

Proposition 2.6** **([16, Theorems 3.1 and 3.2, and Corollary 3.1])

If Condition 1 is satisfied, then the following hold.

1. (i)

For $n\in\mathbb{Z}_{+}$, let $\bm{\alpha}_{n}^{{\dagger}}:=(\alpha_{n}^{{\dagger}}(j))_{j\in\mathbb{M}_{n}}$ denote a probability vector such that

[TABLE]

where

[TABLE]

The probability vector $\bm{\alpha}_{n}^{{\dagger}}$ is an optimal solution for Problem 2.3. 2. (ii)

For $n\in\mathbb{Z}_{+}$, let $\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}^{{\dagger}}:=(\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{0}^{{\dagger}},\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{1}^{{\dagger}},\dots,\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{n}^{{\dagger}})$ denote a probability vector such that

[TABLE]

where $\mathrm{row}_{j}(\,\cdot\,)$ denotes the $j$-th row of the matrix in the parentheses. Furthermore, if a parameter set $(\bm{v},b,\mathbb{C})\in\cal{V}$ of Condition 1 satisfies Condition 2, then

[TABLE]

and

[TABLE]

Remark 2.7

The symbols $\bm{\alpha}_{n}^{{\dagger}}$, $j_{n}^{{\dagger}}$, and $\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}^{{\dagger}}$ correspond to $\bm{\alpha}_{n}^{*}$  $j_{n}^{*}$, and $\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}^{*}$, respectively, in the original notation of [16].

Remark 2.8

Constructing the MIP-form solution (2.17) (and thus (1.9)) requires the sequence $\{\bm{\alpha}_{n}^{{\dagger}}\}$. Each $\bm{\alpha}_{n}^{{\dagger}}$ is an optimal solution for Problem 2.3 with index $n$. Therefore, we only have to solve a single optimization problem for each $n\in\mathbb{Z}_{+}$.

3 A new MIP-form solution

This section presents a new MIP-form solution for $\bm{\pi}$. To construct the new MIP-form solution, we first formulate an LFP problem different from Problem 2.3. We then provide a way to find its optimal solution. Finally, we show the new MIP-form solution for $\bm{\pi}$.

We introduce some definitions to describe our LFP problem for the new MIP-form solution. Let $\mathbb{Z}_{\geq m}:=\{m,m+1,\dots\}$ for any integer $m$. We then define $\bm{u}_{n,\mathbb{K}}^{*}:=(u_{n,\mathbb{K}}^{*}(i))_{i\in\mathbb{M}_{n}}$, $n\in\mathbb{Z}_{\geq K}$, as

[TABLE]

where the second equality is due to (2.7). We also define $\mathbb{I}_{n}^{+}$, $n\in\mathbb{Z}_{+}$, as

[TABLE]

where $(\,\cdot\,)_{j}$ denotes the $j$-th element of the (row or column) vector in the parentheses. Since the generator $\bm{Q}$ is irreducible, $(\bm{e}^{\top}\bm{Q}_{n+1,n})_{j}>0$ for at least one $j\in\mathbb{M}_{n}$ and thus $\mathbb{I}_{n}^{+}\neq\varnothing$. Furthermore, since $\bm{\pi}_{n+1}>\bm{0}$ and $\bm{Q}_{n+1,n}\geq\bm{O},\neq\bm{O}$,

[TABLE]

which is used later.

The following is our LFP problem, which is formulated for each $n\in\mathbb{Z}_{\geq K}$.

Problem 3.1

[TABLE]

Remark 3.2

The objective function of Problem 3.1 is well-defined, as with Problem 2.3 (see Remark 2.4).

An optimal solution for Problem 3.1 is given by Lemma 3.3 below.

Lemma 3.3

For $n\in\mathbb{Z}_{\geq K}$, fix $j_{n}^{*}\in\mathbb{I}_{n}^{+}\neq\varnothing$ such that

[TABLE]

and let $\bm{\alpha}_{n}^{*}:=(\alpha_{n}^{*}(j))_{j\in\mathbb{M}_{n}}$ denote a unit row vector such that

[TABLE]

The vector $\bm{\alpha}_{n}^{*}$ is then an optimal solution for Problem 3.1, and the optimal (and thus maximum) value $r_{n}(\bm{\alpha}_{n}^{*})=u_{n,\mathbb{K}}^{*}(j_{n}^{*})/u_{n}^{*}(j_{n}^{*})$ is positive.

Proof.

First, we prove that $\bm{\alpha}_{n}^{*}$ is an optimal solution for Problem 3.1. It follows from (3.10) and (3.11) that

[TABLE]

which leads to $u_{n,\mathbb{K}}^{*}(j)\leq\xi_{n}u_{n}^{*}(j)$ for all $j\in\mathbb{I}_{n}^{+}$. Thus, for any feasible solution $\bm{\alpha}_{n}$ satisfying (3.7)–(3.9), we have

[TABLE]

Combining this result with (3.6) and (3.12) yields

[TABLE]

which shows that $\bm{\alpha}_{n}^{*}$ is an optimal solution for Problem 3.1.

Next, we prove $r_{n}(\bm{\alpha}_{n}^{*})>0$ by contradiction. To this end, suppose that $r_{n}(\bm{\alpha}_{n}^{*})=0$, or equivalently, that $\bm{\alpha}_{n}\bm{u}_{n,\mathbb{K}}^{*}=0$ for any $\bm{\alpha}_{n}$ satisfying (3.7)–(3.9). We then have

[TABLE]

Therefore, using (3.5), (3.1), and (2.9), we obtain

[TABLE]

which contradicts $\bm{\pi}>\bm{0}$. The proof has been completed. ∎

The next theorem shows that the optimal solution $\bm{\alpha}_{n}^{*}$ for Problem 3.1 leads to an MIP-form solution for $\bm{\pi}$, which is different from the existing one (2.17) based on Problem 2.3.

Theorem 3.4

For each $n\in\mathbb{Z}_{\geq K}$, let $\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}^{*}=(\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{0}^{*},\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{1}^{*},\dots,\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{n}^{*})$, where $\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{k}^{*}$, $k\in\mathbb{Z}_{[0,n]}$, is a $1\times M_{k}$ vector obtained by replacing $\bm{\alpha}_{n}$ in (2.10) with the optimal solution $\bm{\alpha}_{n}^{*}$ for Problem 3.1, i.e.,

[TABLE]

where $j_{n}^{*}\in\mathbb{J}_{n}^{*}$ defined in (3.10). We then have

[TABLE]

and therefore

[TABLE]

Remark 3.5

We only have to solve a single optimization problem (Problem 3.1) for each $n\in\mathbb{Z}_{+}$ to construct the new MIP-form solution (3.14), as with the existing one (2.17) (see Remark 2.8).

Proof of Theorem 3.4.  According to Theorem B.10, it suffices to show that

[TABLE]

To do this, we define $\widetilde{\bm{\alpha}}_{n}:=(\widetilde{\alpha}_{n}(j))_{j\in\mathbb{M}_{n}}$ as

[TABLE]

Equations (3.5) and (3.16) imply that

[TABLE]

and thus $\widetilde{\bm{\alpha}}_{n}$ is a feasible solution for Problem 3.1. Recall here that $\bm{\alpha}_{n}^{*}$ maximizes the objective function $r_{n}(\,\cdot\,)$ (see Lemma 3.3). Therefore, it follows from (3.6), (3.1), and (3.13) that

[TABLE]

It also follows from (3.16), (3.1), and (2.8) that

[TABLE]

where the second equality is due to (2.9). Combining (3.17) and (3.18) leads to

[TABLE]

Consequently, (3.15) holds. The proof has been completed.

Theorem 3.4 shows that the new MIP-form solution (3.14) is obtained by finding the optimal solution $\bm{\alpha}_{n}^{*}$ of Problem 3.1, that is, by finding an element in $\mathbb{J}_{n}^{*}\subseteq\mathbb{I}_{n}^{+}$. This task can be lightened by the following theorem.

Theorem 3.6

The set $\mathbb{J}_{n}^{*}$ is given by

[TABLE]

where $\mathbb{O}_{n}^{+}$ is defined as

[TABLE]

Proof.

Lemma 3.3 states that, for any $j\in\mathbb{J}_{n}^{*}$, $u_{n,\mathbb{K}}^{*}(j)/u_{n}^{*}(j)=r_{n}(\bm{\alpha}_{n}^{*})>0$. This implies that $\mathbb{J}_{n}^{*}$ does not include $j$ such that $u_{n,\mathbb{K}}^{*}(j)=0$. It thus suffices to show that

[TABLE]

It follows from (3.1) and (2.11d) that

[TABLE]

It also follows from (3.20) that

[TABLE]

These two equations show that (3.21) holds, which completes the proof. ∎

4 Advantages of the new MIP-form solution

This section explains the advantages of our new MIP-form solution (3.14) over the existing one (2.17). We first point out the drawbacks of the existing MIP-form solution. We then show that our new solution (3.14) is quasi-algorithmically constructible (see Notion 1.1) unlike the existing one.

The existing MIP-form solution (2.17) has three drawbacks, compared with our new one. First, the existing solution (2.17) requires a parameter set $(\bm{v},b,\mathbb{C})\in\cal{V}$ of Condition 1 such that $\bm{v}$ satisfies Condition 2 (see Proposition 2.6). In some cases, Condition 2 does not hold, and such an unfavorable case is provided in Appendix A. Second, finding a parameter set $(b,\mathbb{C},\bm{v})\in\cal{V}$ may not be easy. Finally, it is of infinite computational complexity to obtain the optimal solution $\bm{\alpha}_{n}^{{\dagger}}$ for Problem 2.3. This is because the objective function of Problem 2.3 includes the infinite sums $\sum_{\ell=n+1}^{\infty}\bm{Q}_{k,\ell}\bm{v}_{\ell}$, $k\in\mathbb{Z}_{[0,n]}$ in (2.14).

Owing to these drawbacks, especially, to the last one, the existing MIP-form solution (2.17) for $\bm{\pi}$ is not quasi-algorithmically constructible. Nevertheless, constructing $\bm{\pi}$ via the existing solution (2.17) is formally expressed in an algorithm style equipped with a stopping criterion, as the sequential update algorithm [16, Algorithm 1]. This is described in Algorithm 1 below.

Remark 4.1

In the original description of Algorithm 1, the following operation is inserted as the first step: “Find $\bm{v}>\bm{0}$, $b>0$, and $\mathbb{C}\in\mathbb{S}$ such that Conditions 1 and 2 hold” (see [16, Algorithm 1]). However, in general, this cannot be performed algorithmically. Rather, $\bm{v}>\bm{0}$, $b>0$, and $\mathbb{C}\in\mathbb{S}$ are input parameters of the algorithm, as described in Algorithm 1 above.

Remark 4.2

In general, we have to truncate the infinite sums $\sum_{\ell=n+1}^{\infty}\bm{Q}_{k,\ell}\bm{v}_{\ell}$, $k\in\mathbb{Z}_{[0,n]}$ in order to compute $\bm{y}_{n}$ (see Remark 2.5 for an exceptional case).

In contrast, our new MIP-form solution (3.14) is quasi-algorithmically constructible. The new solution uses the sequence of probability vectors $\{\bm{\alpha}_{n}^{*}\}$, and each of them is an optimal solution for Problem 3.1. This problem is formulated without finding such a parameter set $(\bm{v},b,\mathbb{C})\in\cal{V}$ satisfying Condition 2. Moreover, the objective function of Problem 3.1 consists of only $\bm{u}_{n}^{*}$ and $\bm{u}_{n,\mathbb{K}}^{*}$ of finite computational complexity. Indeed, the finite computational complexity of $\bm{u}_{n}^{*}$ is guaranteed by the recursion (2.12), and that of $\bm{u}_{n,\mathbb{K}}^{*}$ is guaranteed by the following recursion:

[TABLE]

which follows from (3.1) and (3.22). Therefore, it is of finite computational complexity to obtain the optimal solution $\bm{\alpha}_{n}^{*}$ for Problem 3.1. In summary, constructing our new MIP-form solution (3.14) requires $\{\bm{\alpha}_{n}^{*};n\in\mathbb{Z}_{\geq K}\}$, $\{\bm{u}_{n}^{*};n\in\mathbb{Z}_{+}\}$, and $\{\bm{U}_{n,k}^{*};n\in\mathbb{Z}_{+},k\in\mathbb{Z}_{[0,n]}\}$, which are computed by iterating the recursive procedure consisting of an increasing but finite number of operations per iteration. Consequently, our new solution (3.14) for $\bm{\pi}$ is quasi-algorithmically constructible.

Shown below is an algorithm-style description of computing our new MIP-form solution (3.14) with a stopping criterion. This is the sequential update algorithm for our new solution.

Remark 4.3

To compute the probability vector $\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}^{*}=(\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{0}^{*},\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{1}^{*},\dots,\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{n}^{*})$, Step (3.d.i) performs finding an element $j_{n}^{*}$ of $\mathbb{J}_{n}^{*}$. Finding an element $j_{n}^{*}$ of $\mathbb{J}_{n}^{*}$ is equivalent to solving Problem 3.1, and it requires constructing the sets $\mathbb{I}_{n}^{+}$ and $\mathbb{O}_{n}^{+}$. This construction can be easily performed based on the probabilistic interpretation: The set $\mathbb{I}_{n}^{+}$ consists of the phases of $\mathbb{L}_{n}$ which accept direct (incoming) transitions from $\mathbb{L}_{n+1}$; the set $\mathbb{O}_{n}^{+}$ consists of the phases of $\mathbb{L}_{n}$ which are the starting points of (outgoing) paths leaving $\mathbb{L}_{n}$ eventually and reaching $\mathbb{L}_{n-1}$ while avoiding $\overline{\mathbb{S}}_{n}=\cup_{k=n+1}^{\infty}\mathbb{L}_{k}$. Note that the set $\mathbb{O}_{n}^{+}$ is obtained by identifying the positive rows of $\bm{U}_{n}^{*}\bm{Q}_{n,n-1}$, which is a crucial component of the recursion (2.11d) of $\{\bm{U}_{n,k}^{*}\}$.

Remark 4.4

The choice of finite set $\mathbb{K}\subset\mathbb{Z}_{+}$ can impact on the convergence speed of Algorithm 2. However, it would be difficult to discuss theoretically an optimal set $\mathbb{K}$. A reasonable choice of $\mathbb{K}$ would be $\mathbb{Z}_{[0,K]}=\{0,1,\dots,K\}$.

Remark 4.5

A single run of Algorithm 2 generates the probability vector $\hskip 0.50003pt{}_{(n_{\ell})}\widehat{\bm{\pi}}^{*}=(\hskip 0.50003pt{}_{(n_{\ell})}\widehat{\bm{\pi}}_{0}^{*},\hskip 0.50003pt{}_{(n_{\ell})}\widehat{\bm{\pi}}_{1}^{*},\dots,\hskip 0.50003pt{}_{(n_{\ell})}\widehat{\bm{\pi}}_{n_{\ell}}^{*})$ with $\ell$ being equal to the value fixed on completion of all the operations. Recall that $\hskip 0.50003pt{}_{(n_{\ell})}\widehat{\bm{\pi}}^{*}$ is an LBC-LA truncation approximation to the stationary distribution vector $\bm{\pi}$. Thus, $\sum_{k=1}^{n_{\ell}}k^{m}\hskip 0.50003pt{}_{(n_{\ell})}\widehat{\bm{\pi}}_{k}^{*}$, $m\in\mathbb{N}$ can be considered an approximation to the moment vector $\sum_{k=1}^{\infty}k^{m}\bm{\pi}_{k}$ of the stationary distribution, though, as the order $m$ is larger, a smaller $\varepsilon>0$ would need to be chosen (as the parameter of the stopping criterion) in order to achieve a satisfactory accuracy of the approximation.

Finally, in the two subsequent paragraphs, we mention the advantages of Algorithm 2 based on our new MIP-form solution (3.14), comparing with the existing algorithms [6, 10, 13, 19, 20] for UBH-MCs. These existing algorithms are classified into two types: (i) approximation by conditional stationary distribution [6, 20]; (ii) approximation by level-homogenizing [10, 13, 19]. Both types are originally designed to compute a single approximation to the stationary distribution, but performing them iteratively can generate a convergent sequence of approximations to the stationary distribution. Considering such applications of the existing algorithms, we compare them with our algorithm in the following.

The algorithms [6, 20] compute a single conditional stationary distribution $\bm{\pi}^{(n)}$, as an approximation to $\bm{\pi}$, in a single run. To generate each approximation $\bm{\pi}^{(n)}$, the algorithms [6, 20] have to perform a large (theoretically infinite) amount of preliminary computations involving the levels above $n$. The preliminary numerical results can be used to construct better approximations $\bm{\pi}^{(N)}$ with $N>n$, but to do this actually with satisfactory accuracy requires further preliminary computations involving higher levels. In contrast, Algorithm 2 computes each tentative solution $\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}^{*}$ without any information on the higher levels above $n$ except $\bm{Q}_{n+1,n}$.

The other algorithms [10, 13, 19] conduct level-homogenizing the transition law above some level, say level $n$, in order to compute an approximation to $\bm{\pi}$, denoted by $\bm{\pi}(n)$. More specifically, the approximation $\bm{\pi}(n)$ requires, as the input for computing itself, a (level-independent) $G$-matrix obtained by level-homogenizing the transition law above level $n$. Thus, each approximation $\bm{\pi}(n)$ is computed from scratch, which is time-consuming. In contrast, Algorithm 2 sequentially constructs tentative solutions $\{\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}^{*}\}$ convergent to the stationary distribution vector $\bm{\pi}$, making the most of the components of the “old” tentative solutions.

5 Special cases free from solving Problem 3.1

This section discusses special cases where an MIP-form solution for $\bm{\pi}$ is constructed without solving Problem 3.1. We first show a basic theorem on this matter, and then derive its corollary that implies what cases are free from solving Problem 3.1. Finally, we mention representative examples of such favorable cases.

The following theorem provides the basis for our discussion here.

Theorem 5.1

Let $N$ be a nonnegative integer, and let $\bm{\varphi}_{n}$, $n\in\mathbb{Z}_{\geq N+1}$ denote a $1\times M_{n}$ probability vector such that

[TABLE]

where $\{\varepsilon_{n}\in[0,1);n\in\mathbb{Z}_{\geq N+1}\}$ is a sequence such that $\lim_{n\to\infty}\varepsilon_{n}=0$. Furthermore, for $n\in\mathbb{Z}_{\geq N}$, let $\hskip 0.50003pt{}_{(n)}\breve{\bm{\pi}}:=(\hskip 0.50003pt{}_{(n)}\breve{\bm{\pi}}_{0},\hskip 0.50003pt{}_{(n)}\breve{\bm{\pi}}_{1},\dots,\hskip 0.50003pt{}_{(n)}\breve{\bm{\pi}}_{n})$ denote a probability vector such that

[TABLE]

We then have

[TABLE]

Proof.

Fix $n\in\mathbb{Z}_{\geq N}$ arbitrarily, and note that (5.3) holds if

[TABLE]

Indeed, the triangle inequality and (1.3) yield

[TABLE]

and then, combining this and (5.4) results in (5.3).

To complete the proof, we show that (5.4) holds. Equation (5.1) implies that

[TABLE]

Applying (5.5) to (5.2) yields

[TABLE]

Furthermore, substituting (2.9) into the above inequality leads to

[TABLE]

and thus

[TABLE]

which shows that (5.4) holds. The proof has been completed. ∎

Remark 5.2

Substituting

[TABLE]

into the matrix product form (2.10) of the LBC-LA truncation approximation $\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}=(\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{0},\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{1},\dots,\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}_{n})$, and using (2.6), we obtain $\hskip 0.50003pt{}_{(n)}\widehat{\bm{\pi}}=\hskip 0.50003pt{}_{(n)}\breve{\bm{\pi}}$. Therefore, $\hskip 0.50003pt{}_{(n)}\breve{\bm{\pi}}$ is an LBC-LA truncation approximation to $\bm{\pi}$.

Theorem 5.1 implies in what cases we do not have to solve Problem 3.1. However, Theorem 5.1 requires us to identify $\bm{\pi}_{n}/(\bm{\pi}_{n}\bm{e})$ to some extent, which is not easy in general. We thus provide a more effective (but more restrictive) result that can actually contribute to the construction of MIP-form solutions without solving Problem 3.1.

Corollary 5.3

Suppose that there exists some $N\in\mathbb{Z}_{+}$ such that $\mathbb{M}_{n}=\mathbb{M}:=\{1,2,\dots,M\}\subset\mathbb{N}$ for all $n\in\mathbb{Z}_{\geq N+1}$. Furthermore, there exists some $1\times M$ probability vector $\bm{\varpi}$ such that

[TABLE]

We then have

[TABLE]

Proof.

According to (5.6) and Theorem 5.1, it suffices to show that (5.1) holds with $\bm{\varphi}_{n}=\bm{\varpi}$ for all $n\in\mathbb{Z}_{\geq N+1}$. Equation (5.6) implies that there exists some $\{\delta_{n}\in[0,1);n\in\mathbb{Z}_{\geq N+1}\}$ such that $\lim_{n\to\infty}\delta_{n}=0$ and

[TABLE]

This inequality is equivalent to

[TABLE]

We now fix

[TABLE]

We then have $\lim_{n\to\infty}\varepsilon_{n}=0$ and

[TABLE]

Thus, (5.7) yields

[TABLE]

which shows that (5.1) holds with $\bm{\varphi}_{n}=\bm{\varpi}$ for all $n\in\mathbb{Z}_{\geq N+1}$. ∎

A typical example covered by Corollary 5.3 is a UBH-MC with block monotonicity (see [15, Definition 3.2]). If the generator $\bm{Q}$ is block monotone, then, for all $k\in\mathbb{Z}_{+}$, $\sum_{\ell=\max(k-1,0)}^{\infty}\bm{Q}_{k,\ell}$ is constant and thus is interpreted as the generator of the underlying Markov chain on the phase set $\mathbb{M}$, which controls the level process (see [15, Lemma 3.1]). In this case, $\bm{\varpi}$ is equal to the stationary distribution vector of the underlying Markov chain.

Another example is the (level-independent) M/G/1-type Markov chain. For this chain, several sufficient conditions are established for the tail asymptotics of $\bm{\pi}=(\bm{\pi}_{0},\bm{\pi}_{1},\dots)$ (see [7, 8, 9, 14]) such that, for some $c>0$, $\bm{\mu}>\bm{0}$, and random variable $Y$ on $\mathbb{Z}_{+}$,

[TABLE]

Moreover, (assuming additional conditions, if required), we can obtain (see, e.g., [8, Sections 4 and 5])

[TABLE]

which implies that (5.6) holds.

6 Concluding Remarks

We have established a quasi-algorithmic construction of the exact (not approximate) and whole (not partial) stationary distribution vector $\bm{\pi}$ in upper block-Hessenberg Markov chains (UBH-MCs). The core of this theoretical construction is that all we have to do is to solve just one linear fractional programing (LFP) problem (Problem 3.1) in each iteration for the construction. We have also presented some special cases free from solving the LFP problem. To find other such special cases is an interesting future task. A possible case would be the class of UBH-MCs with asymptotically block-Toeplitz structure. The generator $\bm{Q}$ of such a chain satisfies (1.1) and the following: $M_{n}=M\in\mathbb{N}$ for all sufficiently large $n$ and

[TABLE]

where $\sum_{k=-1}^{\infty}\bm{Q}_{k}$ is an essential $Q$-matrix. In [10], this special UBH-MC is referred to as a multi-dimensional asymptotically quasi-Toeplitz Markov chain.

Appendix A An example violating Condition 2

This section presents an example of UBH $Q$-matrices for which the existing MIP-form solution in [16] does not hold. To describe that example, we assume that $\bm{Q}$ is the generator of a specific M/G/1-type Markov chain with power-like level increments. For such a generator $\bm{Q}$, we show that Condition 2 does not hold for any parameter set $(\bm{v},b,\mathbb{C})$ of Condition 1 and therefore the existing MIP-form solution is not established.

Let $\mathbb{S}=\mathbb{Z}_{+}\times\mathbb{M}=\mathbb{Z}_{+}\times\{1,2,\dots,M\}$, and suppose that $\bm{Q}=(q(k,i;\ell,j))_{(k,i;\ell,j)\in\mathbb{S}}$ is an M/G/1-type generator given by

[TABLE]

where $\bm{A}_{k}:=(A_{k,i.j})_{i,j\in\mathbb{M}}$, $k\in\mathbb{Z}_{\geq-1}$, is an $M\times M$ matrix. We then assume the following.

Assumption A.1

(i) The generator $\bm{Q}$ in (A.6) is irreducible; (ii) $\sum_{k=-1}^{\infty}\bm{A}_{k}$ is an essential $Q$-matrix with stationary distribution vector $\bm{\varpi}$; (iii) $\bm{\varpi}\sum_{k=-1}^{\infty}k\bm{A}_{k}\bm{e}<0$; and (iv) $\lim_{k\to\infty}k^{3}\bm{A}_{k}=\bm{C}_{A}$ for some nonnegative matrix $\bm{C}_{A}\neq\bm{O}$.

We confirm that there exists some $(\bm{v},b,\mathbb{C})\in\cal{V}$ of Condition 1 under Assumption A.1. Uniformizing the generator $\bm{Q}$ (see, e.g., [2, Problem II.4.1]) yields an M/G/1 type stochastic matrix that satisfies the stability condition [2, Chapter XI, Proposition 3.1] of GI/G/1-type Markov chains (including M/G/1-type ones) due to the conditions (i)–(iii) of Assumption A.1. Therefore, Condition 1 holds for the generator $\bm{Q}$ satisfying Assumption A.1 (see, e.g., [11, Theorem 1.1]).

We confirm that Condition 2 does not hold for any $(\bm{v},b,\mathbb{C})\in\cal{V}$ of Condition 1 under Assumption A.1. Fix $(\bm{v},b,\mathbb{C})\in\cal{V}$ arbitrarily. Since $\mathbb{C}$ is a finite subset of the state space $\mathbb{S}=\mathbb{Z}_{+}\times\mathbb{M}$, there exists some $N\in\mathbb{Z}_{+}$ such that $\mathbb{C}\subseteq\mathbb{S}_{N}=\mathbb{Z}_{[0,N]}\times\mathbb{M}$. Thus, fix $N\in\mathbb{Z}_{+}$ as such, and let $\{(X_{t},J_{t});t\geq 0\}$ denote an M/G/1-type Markov chain on state space $\mathbb{S}$ with the generator $\bm{Q}$ satisfying Assumption A.1. It then follows from [11, Theorem 1.1] that there exists some $\gamma>0$ such that

[TABLE]

where $T_{\mathbb{C}}(1)=\inf\{t\geq 1:(X_{t},J_{t})\in\mathbb{C}\}$ and $T_{\mathbb{C}}=\inf\{t>0:(X_{t},J_{t})\in\mathbb{C}\}$. It also follows from $\mathbb{C}\subseteq\mathbb{S}_{N}$ and the M/G/1-type structure of $\bm{Q}$ that there exists some $\delta>0$ such that

[TABLE]

Combining (A.7) and (A.8), we have

[TABLE]

Furthermore, from [8, Theorem 4.1.1] and Assumption A.1 (especially, the condition (iv)), we have

[TABLE]

for some constant $c>0$. Using (A.6), (A.9), and (A.10), we obtain

[TABLE]

Therefore, Condition 2 does not hold for any $(\bm{v},b,\mathbb{C})\in\cal{V}$ in the present example.

Appendix B Convergent approximations to the stationary distribution vector of an essential $Q$-matrix

The purpose of this section is to provide the fundamental results needed in the proof of the main theorem (Theorem 3.4). This section consists of two subsections. Section B.1 describes several notions associated with essential $Q$-matrices. Section B.2 presents a necessary and sufficient condition for the convergence of a sequence of approximations to the stationary distribution vector of an essential $Q$-matrix.

B.1 Definition of basic notions

This subsection introduces the notions used in the next subsection. We first redefine the symbols $\mathbb{S}$ and $\bm{Q}$ introduced in the body of this paper. We next define several notions, such as essential $Q$-matrices, subinvariant measures, irreducibility, and recurrence. We then present a proposition on the recurrence and subinvariant measures of an irreducible essential $Q$-matrix.

Let $\mathbb{S}$ denote an arbitrary countable set, and let $\bm{Q}:=(q(i,j))_{i,j\in\mathbb{S}}$ denote an arbitrary $Q$-matrix (see, e.g., [1, page 64]), that is, a diagonally dominant matrix such that

[TABLE]

The following is the definition of essential $Q$-matrices.

Definition B.1

The $Q$-matrix $\bm{Q}$ is said to be essential if and only if the following hold for all $i\in\mathbb{S}$: (i) $q(i,i)$ is finite; (ii) $\sum_{j\in\mathbb{S}}q(i,j)=0$; (iii) $q(i,i)<0$. Note that any essential $Q$-matrix can be interpreted as the infinitesimal generator of a continuous-time Markov chain. Thus, an essential $Q$-matrix may be referred to as an infinitesimal generator or generator, depending on the context.

Remark B.2

The conditions (i) and (ii), respectively, imply that $\bm{Q}$ is stable and conservative (see, e.g., [4, Definition 13.3.10]).

Remark B.3

An essential $Q$-matrix is not necessarily regular (or equivalently, non-explosive). Indeed, provided that $\bm{Q}$ is an essential $Q$-matrix, $\bm{Q}$ is regular if and only if, for any $\lambda>0$, the system of equations

[TABLE]

has no bounded solution other than $\bm{x}=\bm{0}$ (see, e.g., [1, Chapter 2, Theorem 2.7] and [4, Theorem 13.3.11]).

To proceed further, we assume that $\bm{Q}$ is an essential $Q$-matrix, and then define $\{\Phi(t);t\geq 0\}$ as a Markov chain on $\mathbb{S}$ having this essential $\bm{Q}$ as its generator. For later use, we also define $\bm{P}$ as a stochastic matrix such that

[TABLE]

where $\mathrm{diag}\{-\bm{Q}\}$ is a diagonal matrix whose diagonal elements are identical to those of $(-\bm{Q})$. Note that $\bm{P}$ is interpreted as the transition probability matrix of an embedded discrete-time Markov chain for $\{\Phi(t)\}$ with generator $\bm{Q}$ (see [4, Section 13.3.2]). Hence, we call $\bm{P}$ the embedded transition probability matrix of $\{\Phi(t)\}$.

For the essential $Q$-matrix $\bm{Q}$, we define subinvariant measures and stationary distribution vectors.

Definition B.4

Let $\bm{\mu}:=(\mu(j))_{j\in\mathbb{S}}$ denote an arbitrary nonnegative and nonzero vector.

1. (i)

If $\bm{\mu}\bm{Q}\leq\bm{0}$, $\bm{\mu}$ is said to be a subinvariant measure of $\bm{Q}$ or the Markov chain $\{\Phi(t)\}$. Furthermore, if a subinvariant measure $\bm{\mu}$ satisfies $\bm{\mu}\bm{Q}=\bm{0}$, $\bm{\mu}$ is said to be invariant or stationary. 2. (ii)

If an invariant measure $\bm{\mu}$ satisfies $\bm{\mu}\bm{e}=1$, $\bm{\mu}$ is said to be a stationary distribution vector of $\bm{Q}$ or the Markov chain $\{\Phi(t)\}$.

Finally, we provide the definition of irreducibility, recurrence, and their related notions, and present a proposition that summarizes basic results on the recurrence and subinvariant measures of an irreducible essential $Q$-matrix.

Definition B.5

1. (i)

An essential $Q$-matrix $\bm{Q}$ and its Markov chain $\{\Phi(t)\}$ are irreducible (resp. transient, recurrent) if and only if the embedded transition probability matrix $\bm{P}$ in (B.1) is irreducible (resp. transient, recurrent) (see [4, Definitions 13.4.1 and 13.4.2]). 2. (ii)

An irreducible essential $Q$-matrix $\bm{Q}$ and its Markov chain $\{\Phi(t)\}$ are ergodic (i.e., positive recurrent) if and only if there exists a stationary distribution vector, or equivalently, a summable invariant measure unique up to constant multiples (see [4, Definition 13.4.8 and Theorem 13.4.10]).

Proposition B.6

Suppose that $\bm{Q}$ is an irreducible essential $Q$-matrix. The following statements hold:

1. (i)

There always exists a subinvariant measure of $\bm{Q}$. 2. (ii)

Any subinvariant measure of $\bm{Q}$ is positive, that is, its elements are all positive. 3. (iii)

If $\bm{Q}$ is recurrent, then it has an invariant measure, which is unique up to constant multiples. 4. (iv)

Any subinvariant measure of recurrent $\bm{Q}$ is, in fact, an invariant measure. 5. (v)

The matrix $\bm{Q}$ has a subinvariant measure that is not invariant if and only if it is transient.

Proof.

Let $\bm{\mu}=(\mu_{i})_{i\in\mathbb{S}}$ be a nonnegative and nonzero vector, and let

[TABLE]

It then follows from (B.1) and (B.2) that

[TABLE]

Thus, $\bm{\eta}$ is a subinvariant (resp. an invariant) measure of the embedded transition probability matrix $\bm{P}$ if and only if $\bm{\mu}\bm{Q}\leq\bm{0}$ (resp. $\bm{\mu}\bm{Q}=\bm{0}$) or equivalently, $\bm{\mu}$ is a subinvariant (resp. an invariant) measure of the $Q$-matrix $\bm{Q}$ (see [18, Definition 5.3]). Furthermore, since the $Q$-matrix $\bm{Q}$ is essential (that is, $-q(i,i)>0$ for all $i\in\mathbb{S}$),

[TABLE]

Therefore, the present proposition is an immediate consequence of [18, Theorem 5.4] (together with its corollary) and [18, Lemmas 5.5 and 5.6]. That implication is as follows:

[TABLE]

The proof has been completed. ∎

B.2 Convergence condition of approximations

This subsection proves a single theorem (Theorem B.10), which provides a necessary and sufficient condition for the convergence of a sequence of approximations to the stationary distribution of an essential $Q$-matrix. That necessary and sufficient condition is crucial to the proof of Theorem 3.4, the main theorem of this paper.

We begin with the following assumption.

Assumption B.7

1. (i)

The essential $Q$-matrix $\bm{Q}$ is ergodic with the unique stationary distribution vector $\bm{\pi}:=(\pi(j))_{j\in\mathbb{S}}>\bm{0}$ (the uniqueness and positivity of $\bm{\pi}$ is due to Proposition B.6). 2. (ii)

For all $n\in\mathbb{Z}_{+}$, $\bm{Q}_{n}:=(q_{n}(i,j))_{i,j\in\mathbb{S}}$ is a stable and conservative $Q$-matrix (see Remark B.2) such that $\lim_{n\to\infty}q_{n}(i,j)=q(i,j)$ for all $i,j\in\mathbb{S}$. 3. (iii)

Each $\bm{Q}_{n}$ has at least one stationary distribution vector, denoted by $\bm{\pi}_{n}:=(\pi_{n}(j))_{j\in\mathbb{S}}$, which can be considered an approximation to $\bm{\pi}$.

Remark B.8

The vector $\bm{\pi}_{n}$ in Assumption B.7 is redefined here and thus is completely different from “$\bm{\pi}_{n}$” used as the subvector of $\bm{\pi}$ in the body of this paper.

Under Assumption B.7, we have the next lemma, which is used to prove the theorem of this subsection.

Lemma B.9

Suppose that Assumption B.7 holds. Fix $j_{0}\in\mathbb{S}$ arbitrarily, and let $\mathbb{H}$ denote an infinite subset of $\mathbb{Z}_{+}$ such that $\{\pi_{n}(j_{0});n\in\mathbb{H}\}$ converges to some $\alpha\in[0,\infty)$, that is,

[TABLE]

Under these conditions, we have the statements (i) and (ii) below:

1. (i)

It holds that

[TABLE] 2. (ii)

Furthermore, if $\alpha>0$, then

[TABLE]

where, for any vector $\bm{x}:=(x(j))_{j\in\mathbb{S}}$, $\bm{x}^{\mathbb{A}}:=(x^{\mathbb{A}}(j))_{j\in\mathbb{S}}$ denotes a vector such that

[TABLE]

Proof.

We begin with the proof of the statement (i). To prove (B.4) and (B.5), it suffices to show that, for all infinite $\mathbb{H}^{\prime}\subseteq\mathbb{H}$,

[TABLE]

Indeed, for any $j\in\mathbb{S}$, there exists some infinite $\mathbb{H}_{j}\subseteq\mathbb{H}$ such that

[TABLE]

Note that $\mathbb{H}_{j}\subseteq\mathbb{H}$ and $\mathbb{H}\subseteq\mathbb{H}$ (any set is a subset of itself). Therefore, if (B.7) holds for all infinite $\mathbb{H}^{\prime}\subseteq\mathbb{H}$, then

[TABLE]

Combining (B.8) and (B.9) yields

[TABLE]

which shows that (B.4) and (B.5) hold.

To complete the proof of (B.4) and (B.5), we prove that (B.7) holds for all infinite $\mathbb{H}^{\prime}\subseteq\mathbb{H}$. Let $\mathbb{H}^{\prime}$ be an arbitrary infinite subset of $\mathbb{H}$, and let

[TABLE]

From (B.3) and (B.10), we then have

[TABLE]

Furthermore, using (B.10), $q(j,j)=\lim_{n\to\infty}q_{n}(j,j)$ (due to Assumption B.7 (ii)), $\bm{\pi}_{n}\bm{Q}_{n}=\bm{0}$, and Fatou’s lemma, we obtain

[TABLE]

which leads to

[TABLE]

It thus follows from Proposition B.6 (iii) and (iv) that if $\bm{\mu}:=(\mu(j))_{j\in\mathbb{S}}\neq\bm{0}$ then $\bm{\mu}$ is a unique (up to constant multiples) invariant measure of the ergodic $Q$-matrix $\bm{Q}$. Therefore, unifying both cases that $\bm{\mu}\neq\bm{0}$ and $\bm{\mu}=\bm{0}$, we can see that there exists some $c\geq 0$ such that

[TABLE]

Substituting (B.12) into (B.11) yields

[TABLE]

Combining this with (B.10) and (B.12) results in

[TABLE]

In addition, it follows from $\bm{\pi}_{n}\bm{e}=\bm{\pi}\bm{e}=1$, Fatou’s lemma, and (B.13) that

[TABLE]

This result and (B.13) imply (B.7). Consequently, (B.4) and (B.5) have been proved.

We move on to the proof of the statement (ii). Suppose $\alpha>0$, and let $\mathbb{A}$ be a nonempty and finite $\mathbb{A}\subset\mathbb{S}$. Since $\mathbb{A}$ is finite, it follows from (B.4) that

[TABLE]

where the positivity of the second limit is due to $\alpha>0$, $\bm{\pi}>\bm{0}$, and $\mathbb{A}\neq\varnothing$. The above two equations yield

[TABLE]

which implies that (B.6) holds. The proof has been completed. ∎

The following theorem presents a necessary and sufficient condition that the sequence $\{\bm{\pi}_{n}\}$ of approximations converges to the stationary distribution vector $\bm{\pi}$.

Theorem B.10

Under Assumption B.7, $\lim_{n\to\infty}\|\bm{\pi}_{n}-\bm{\pi}\|_{1}=0$ if and only if

[TABLE]

Proof.

We note that if $\lim_{n\to\infty}\|\bm{\pi}_{n}-\bm{\pi}\|_{1}=0$ holds then

[TABLE]

which implies (B.14). Therefore, to complete the proof, we prove that “(B.14) $\Rightarrow$ (B.15)”, and then prove that “(B.15) $\Rightarrow\lim_{n\to\infty}\|\bm{\pi}_{n}-\bm{\pi}\|_{1}=0$”.

We prove that “(B.14) $\Rightarrow$ (B.15)”. To this end, suppose that (B.14) holds while (B.15) does not hold, that is,

[TABLE]

Let $\mathbb{H}$ be an arbitrary infinite subset $\mathbb{H}\subset\mathbb{Z}_{+}$ such that

[TABLE]

It then follows from Lemma B.9 (i) that

[TABLE]

It also follows from (B.17) and the finiteness of $\mathbb{A}$, and $\mathbb{H}\subset\mathbb{Z}_{+}$ that

[TABLE]

which contradicts (B.14). Therefore, (B.14) implies (B.15).

We prove that “(B.15) $\Rightarrow\lim_{n\to\infty}\|\bm{\pi}_{n}-\bm{\pi}\|_{1}=0$”. For any fixed $j\in\mathbb{S}$, there exist some infinite subset $\mathbb{H}_{j}\subset\mathbb{Z}_{+}$ such that

[TABLE]

Thus, applying Lemma B.9 (i) to the sequence $\{\bm{\pi}_{n};n\in\mathbb{H}_{j}\}$ yields

[TABLE]

Combining this and (B.15) leads to

[TABLE]

Note here that $\bm{\pi}_{n}$ and $\bm{\pi}$ are probability vectors. Therefore, due to the dominated convergence theorem, (B.18) implies that $\lim_{n\to\infty}\|\bm{\pi}_{n}-\bm{\pi}\|_{1}=0$. It has been proved that “(B.15) $\Rightarrow\lim_{n\to\infty}\|\bm{\pi}_{n}-\bm{\pi}\|_{1}=0$”, which completes the proof. ∎

Remark B.11

Lemma B.9 (i) is the $Q$-matrix-version of [21, Lemma 2.1] for stochastic matrices. Furthermore, Lemma B.9 (ii) and Theorem B.10 are the $Q$-matrix-versions of Corollary 2.2 (i) and (ii), respectively, in [21].

Acknowledgments

The author thanks Dr. Masakiyo Miyazawa for valuable comments on the presentation of the notion of quasi-algorithmic constructions. The research of the author was supported in part by JSPS KAKENHI Grant Number JP21K11770.