Intermittently Observable Markov Decision Processes

TL;DR

This paper addresses decision-making in Markov Decision Processes with unreliable, intermittent state information, proposing new formulations and algorithms to find near-optimal policies efficiently despite information losses.

Contribution

It introduces a belief MDP and a tree MDP formulation for intermittent observations, along with finite-state approximations and a nested value iteration algorithm for improved efficiency.

Findings

Finite-state approximations enable near-optimal policy computation.

Nested value iteration outperforms standard methods in speed.

Numerical results confirm the effectiveness of proposed algorithms.

Abstract

This paper investigates MDPs with intermittent state information. We consider a scenario where the controller perceives the state information of the process via an unreliable communication channel. The transmissions of state information over the whole time horizon are modeled as a Bernoulli lossy process. Hence, the problem is finding an optimal policy for selecting actions in the presence of state information losses. We first formulate the problem as a belief MDP to establish structural results. The effect of state information losses on the expected total discounted reward is studied systematically. Then, we reformulate the problem as a tree MDP whose state space is organized in a tree structure. Two finite-state approximations to the tree MDP are developed to find near-optimal policies efficiently. Finally, we put forth a nested value iteration algorithm for the finite-state approximations, which is proved to be faster than standard value iteration. Numerical results demonstrate the effectiveness of our methods.

Tables

Table 1. Table 1 : Computation time and iterations of different algorithms for solving TA( L 𝐿 L ).

		NVI	VI	R-NVI
	Time (s)	16	44	63
$\left|𝒮\right|=40,\left|𝒜\right|=3$ $\rho =0.7,L=6$	Iteration	60	48	288
	Time (s)	86	259	310
$\left|𝒮\right|=80,\left|𝒜\right|=4$ $\rho =0.8,L=5$	Iteration	62	53	265
	Time (s)	103	268	301
$\left|𝒮\right|=100,\left|𝒜\right|=5$ $\rho =0.9,L=4$	Iteration	58	49	196
	Time (s)	417	1316	1461
$\left|𝒮\right|=100,\left|𝒜\right|=5$ $\rho =0.9,L=5$	Iteration	53	47	235
	Time (s)	180	580	649
$\left|𝒮\right|=200,\left|𝒜\right|=3$ $\rho =0.9,L=5$	Iteration	65	57	285

Table 2. Table 2 : Performance of TA( L 𝐿 L ) and TA( n , L 𝑛 𝐿 n,L ), L = 2 , n = 4 formulae-sequence 𝐿 2 𝑛 4 L=2,n=4 .

		TA($L$)	TA($L+n$)	TA($n,L$)
	Value	366	368	368
$\rho =0.9$	Time (s)	0.48	34	0.64
	Value	310	318	318
$\rho =0.8$	Time (s)	0.52	34	0.65
	Value	196	215	215
$\rho =0.6$	Time (s)	0.45	30	0.72
	Value	155	175	175
$\rho =0.5$	Time (s)	0.44	30	0.80

Table 3. Table 3 : Values of TA( L 𝐿 L ) policies ( ρ = 0.9 𝜌 0.9 \rho=0.9 ).

	TA(2)	TA(3)	TA(6)
$\left|𝒮\right|=20,\left|𝒜\right|=4$	69	69	69
$\left|𝒮\right|=30,\left|𝒜\right|=3$	90	90	90
$\left|𝒮\right|=40,\left|𝒜\right|=4$	192	193	193
$\left|𝒮\right|=50,\left|𝒜\right|=5$	395	397	397

Equations

$$P_a(i,j)=P(j|i,a)=\mathrm{Pr}\left(s_{t+1}=j|s_t=i,a_t=a\right),\forall a\in \mathcal{A}\text{ and }\forall i,j\in \mathcal{S}.$$

$$\frac{V(s)\text{buildrel}\Delta }{=\max E\left[\sum _{t=0}^{\infty }{\beta }^{t}r(s_t,a_t)|s_0=s\right],s\in \mathcal{S}.}$$

$$J(\theta )=\sum _{s\in \mathcal{S}}\theta (s)V(s),\theta \in {\Delta }_{\mathcal{S}},$$

$${\Delta }_{\mathcal{S}}\triangleq \left\{\theta \in {\mathbb{R}}^{|\mathcal{S}|}:\sum _{i=1}^{|\mathcal{S}|}\theta (i)=1,\theta (i)\ge 0,\forall i\right\}.$$

$$V(s)=\underset{a\in \mathcal{A}}{\max }\left\{r(s,a)+\beta \sum _{y\in \mathcal{S}}P(y|s,a)V(y)\right\},s\in \mathcal{S}.$$

$$\mathrm{Pr}({\gamma }_t=1)=\rho \in (0,1],\forall t.$$

$$\frac{{\mathbf{b}}_t\text{buildrel}\Delta }{=\mathrm{Pr}\left(s_t|s_{t_1},s_{t_2},\cdots ,s_{t_k},a_0,a_1,\cdots ,a_{t-1}\right)=\mathrm{Pr}\left(s_t|s_{t_k},a_{t_k},a_{t_k+1},\cdots ,a_{t-1}\right).}$$

$$\frac{T({\mathbf{b}}_t|{\mathbf{b}}_{t-1},a_{t-1})\text{buildrel}\Delta }{=\mathrm{Pr}\left({\mathbf{b}}_t|{\mathbf{b}}_{t-1},a_{t-1}\right)=\{\begin{array}{ll}1-\rho ,& \text{ if }{\mathbf{b}}_t=P_{a_{t-1}}^{\top }{\mathbf{b}}_{t-1}\\ \rho [P_{a_{t-1}}^{\top }{\mathbf{b}}_{t-1}{]}_i,& \text{ if }{\mathbf{b}}_t={\mathbf{e}}_i,i\in \mathcal{S}\\ 0,& \mathrm{otherwise}\end{array}.}$$

$$R({\mathbf{b}}_t,a_t)=\sum _{s\in \mathcal{S}}{\mathbf{b}}_t(s)r(s,a_t),{\mathbf{b}}_t\in \mathcal{W},a_t\in \mathcal{A}.$$

$$\varphi (\mathbf{b},\rho )=\underset{a\in \mathcal{A}}{\max }\left\{R(\mathbf{b},a)+\beta \sum _{{\mathbf{b}}^{\prime }\in \mathcal{W}}T\left({\mathbf{b}}^{\prime }|\mathbf{b},a\right)\varphi ({\mathbf{b}}^{\prime },\rho )\right\},\mathbf{b}\in \mathcal{W},$$

$$\frac{{\Omega }_{\theta }\text{buildrel}\Delta }{=\left\{\prod _{k=0}^L{P}_{a_k}^{\top }\mathbf{b}:a_k\in \mathcal{A},\mathbf{b}\in \left\{\theta ,{\mathbf{e}}_i:i\in \mathcal{S}\right\},L=0,1,2,\cdots \right\}.}$$

$${\varphi }^{\pi }[\rho ]=R_{\pi }+\beta T_{\rho }^{\pi }{\varphi }^{\pi }[\rho ],$$

$$\varphi (\mathbf{b},1)=\underset{a\in \mathcal{A}}{\max }\left\{\sum _{s\in \mathcal{S}}\mathbf{b}(s)r(s,a)+\beta {\mathbf{b}}^{\top }{P}_{a}V\right\}.$$

$$\varphi [1]-\varphi [\rho ]\le {\left(I-\beta T_{\rho }^{{\pi }_1}\right)}^{-1}\eta ,$$

$$\eta (\mathbf{b})=\beta (1-\rho )\left[J(\tau )-\varphi (\tau ,1)\right],\mathbf{b}\in \mathcal{W}.$$

$$g(h)=g(s,u_1,u_2,\cdots ,u_n)=\prod _{k=1}^n{P}_{u_k}^{\top }{\mathbf{e}}_s=P_{u_n}^{\top }{P}_{u_{n-1}}^{\top }\cdots P_{u_1}^{\top }{\mathbf{e}}_s,h\in \mathcal{H}$$

$$\frac{{\mathcal{G}}_{n+1}\text{buildrel}\Delta }{=\left\{(h,a):h\in {\mathcal{G}}_n,a\in \mathcal{A}\right\}.}$$

$$\frac{D(h^{\prime }|h,a)\text{buildrel}\Delta }{=\mathrm{Pr}\left(h^{\prime }|h,a\right)=\{\begin{array}{ll}1-\rho ,& \text{if }{h}^{\prime }=(h,a)\\ \rho [P_a^{\top }g(h){]}_i,& \text{if }{h}^{\prime }=i\in \mathcal{S}\\ 0,& \text{ otherwise.}\end{array}}$$

$$\frac{D_L(h^{\prime }|h,a)\text{buildrel}\Delta }{=\mathrm{Pr}\left(h^{\prime }|h,a\right)=\{\begin{array}{ll}1-\rho ,& \text{ if }{h}^{\prime }=h\in {\mathcal{G}}_L\\ \rho {\mathbf{b}}_a(i),& \text{ if }{h}^{\prime }=i\in \mathcal{S}\\ 0,& \text{ otherwise.}\end{array}}$$

$$\underset{h\in {\mathcal{G}}_k}{\max }\left|{\phi }_L(h)-\phi (h)\right|\le {\delta }_L\left(\frac{{\beta }^{L+1}{(1-\rho )}^{L+1}}{1-\beta }+\sum _{t=L-k}^{L-1}{\beta }^{t+1}{(1-\rho )}^{t+1}\right),k=0,1,\cdots ,L$$

$$\phi (h)=\underset{a\in \mathcal{A}}{\max }\left\{\bar{R}(h,a)+\beta \sum _{i\in \mathcal{S}}D\left(i|h,a\right)\phi (i)+\beta D\left({\tau }_a|h,a\right)\phi ({\tau }_a)\right\},$$

$$\left|{\mathcal{H}}_L\right|=\left|{\mathcal{G}}_0\right|\frac{{\left|\mathcal{A}\right|}^{L+1}-1}{\left|\mathcal{A}\right|-1}.$$

$${\mathcal{H}}_L^n=\left(\underset{i=0}{\overset{n}{\cup }}{\mathcal{Z}}_i\right)\cup \left(\underset{i=n+1}{\overset{L+n}{\cup }}{\mathcal{G}}_i^{-}\right),$$

$$\frac{{\mathcal{Z}}_i\text{buildrel}\Delta }{=\{\begin{array}{ll}{\mathcal{G}}_0,& i=0,\\ \left\{\left(h,{\mu }_L^{i-1}(h)\right):h\in {\mathcal{Z}}_{i-1}\right\},& 1\le i\le n,\end{array}}$$

$$\frac{{\mathcal{G}}_{n+i}^{-}\text{buildrel}\Delta }{=\{\begin{array}{ll}\left\{\left(h,a\right):a\in \mathcal{A},h\in {\mathcal{Z}}_n\right\},& i=1,\\ \left\{(h,a):a\in \mathcal{A},h\in {\mathcal{G}}_{n+i-1}^{-}\right\},& i>1.\end{array}}$$

$${\mathcal{A}}_h^n=\{\begin{array}{ll}\left\{{\mu }_L^{n-1}(h)\right\},& \text{if }h\in \underset{i=0}{\overset{n-1}{\cup }}{\mathcal{Z}}_i\\ \mathcal{A},& \text{othereise.}\end{array}$$

$$\frac{D_L^n(h^{\prime }|h,a)\text{buildrel}\Delta }{=\mathrm{Pr}\left(h^{\prime }|h,a\right)=\{\begin{array}{ll}1-\rho ,& \text{ if }{h}^{\prime }=h\in {\mathcal{G}}_{L+n}^{-}\\ \rho {\mathbf{b}}_a(i),& \text{ if }{h}^{\prime }=i\in \mathcal{S}\\ 0,& \text{ otherwise.}\end{array}}$$

$$Q(h,a)=\bar{R}(h,a)+\beta \sum _{i\in \mathcal{S}}D\left(i|h,a\right)\phi (i)+\beta D\left({\tau }_a|h,a\right)\phi ({\tau }_a).$$

$$\frac{{ϵ}_h\text{buildrel}\Delta }{=\min \left\{Q(h,\mu (h))-Q(h,a)|a\in \mathcal{A}\text{ is non-optimal}\right\}.}$$

$$ϵ(n)\triangleq \min \{{ϵ}_h/2:h\in {\mathcal{H}}_n\}.$$

Full text

Keywords: MDP, state information losses, truncated approximation, structural results, nested value iteration.

1 Introduction

Markov decision process (MDP) is a widely adopted model for sequential decision-making in discrete-time stochastic control problems. At each time step, the process is in some state $s$, and a controller is responsible for selecting an action $a$ from the set of available actions of state $s$. Upon the execution of action $a$, the process stochastically transitions to a new state $s^{\prime}$ at the next time step and generates a reward $r(s,a)$. Solving the MDP consists in finding an optimal policy of selecting actions for the controller to maximize the total reward over a time horizon. It is well-known that the current state of the process is sufficient for computing the optimal action at any time step (Puterman, 1994). We thus refer to a policy as a mapping from state space to action space.

At every time point of decision-making, the controller needs the current state information to determine the action to be applied. In the classical setting, the current state information is assumed to be always available to the controller. This assumption, however, is not valid in many practical applications. A typical example is a situation where the controller relies on a remote sensor for perceiving the state information of the process; at each time step, the remote sensor observes the current state of the process and transmits the information to the controller via wireless communication. Such a situation is ubiquitous nowadays as wireless sensor network technology is being rapidly developed and deployed (Yick et al., 2008; Kandris et al., 2020). However, despite its many advantages, wireless communication is generally not reliable. In particular, information transmissions over a wireless communication channel may occasionally fail due to channel fading and environmental interference (Tse and Viswanath, 2005). As a result, the state information can not always be delivered successfully from the remote sensor to the controller; hence the controller has to select actions for the MDP in the presence of state information losses. We refer to this problem as intermittently observable MDP (IOMDP) and investigate it in this paper from theoretical and algorithmic perspectives.

Studies on MDPs with imperfect information transmissions can be traced back to Brooks and Leondes (1972), who investigated MDPs with one-step delayed state information. Continuing along this line of research, Katsikopoulos and Engelbrecht (2003) extended the framework and showed that an MDP with delays (whether constant or random delays) can be reduced to an MDP without delays, differing only in the size of state space. Adlakha et al. (2012) studied a group of coupled MDPs with delayed state information and provided a bound on the finite history of observations and control needed for the optimal control. In addition, state information delays have also been considered in partially observable MDPs (POMDPs) (Kim and Jeong, 1987; Bander and White III, 1999) and decentralized control problems (Hsu and Marcus, 1982; Varaiya and Walrand, 1978).

While delayed state information has been extensively investigated, intermittent state information has not attracted particular attention in MDP studies. Perhaps this is because IOMDPs naturally fall into the category of POMDPs. Specifically, if the state information is not received by the controller at some time step, then the previously received state information could serve as a noisy observation that implies a distribution of the current state. The mainstream method for treating a POMDP is to reformulate it as a fully observable MDP by constructing belief states (Astrom, 1965; Krishnamurthy, 2016). The resulting MDP is thus sometimes called belief MDP (Kaelbling et al., 1998). Under this formulation, it can be proved that the value function of the belief MDP is a piecewise linear and convex function of the belief state (Smallwood and Sondik, 1973; White III and Harrington, 1980; Araya et al., 2010). This nice property has been extensively exploited to develop algorithms for finding optimal policies, resulting in the one-pass algorithm (Smallwood and Sondik, 1973), the linear support algorithm (Cheng, 1988), incremental pruning (Cassandra et al., 2013), the duality-based approach (Zhang, 2010), and others. In addition, there are also many algorithms that approximate the exact value iteration solution, including the point-based value iteration (Pineau et al., 2003; Porta et al., 2006), heuristic search value iteration (Smith and Simmons, 2012), and others (Shani et al., 2007; Kurniawati et al., 2008; Poupart et al., 2011).

Although there is a rich set of algorithms for POMDPs, finding optimal or near-optimal policies is still computationally complex. A major reason is the so-called curse of dimensionality — given a problem with $n$ physical states, the belief state space of the associated POMDP is the $(n-1)$-dim probability simplex, which is a continuous space. As a result, general POMDP algorithms are inefficient for solving IOMDPs. Fortunately, this paper finds that the special structure of the problem can be exploited to find near-optimal policies efficiently.

In this paper, we model the transmissions of state information as a Bernoulli process whose parameter (i.e., the state information reception probability) is determined by the communication environment. We first adopt the belief MDP formulation to analyze the effect of state information losses on the optimal value. Then, we reformulate the IOMDP using the sufficient history—a segment of history information that is sufficient for making the optimal decision. The set of sufficient histories can be organized in a tree structure. We thus call the new formulation a tree MDP. Based on the tree MDP formulation, we propose two efficient finite-state approximations for the problem to find near-optimal policies.

The contributions of this paper are as follows:

• •

We show that the optimal value monotonically increases with the state information reception probability and provide a bound for the performance regret caused by state information losses.

• •

We reformulate the IOMDP as a tree MDP and propose a truncated approximation to the tree MDP for finding near-optimal policies. A theoretical bound for the approximation error is derived to better understand the method.

• •

We find that a majority of the states of the tree MDP are redundant in the sense that they will never be visited. This motivates us to put forth the high-order truncated approximation—a modified truncated approximation that could identify the redundant states and omit them when computing policies. It is more efficient and scalable than the original truncated approximation because of the reduction of state space.

• •

We propose a variant of value iteration, called nested value iteration, for the truncated approximation of the tree MDP. We show theoretically that the nested value iteration converges to the optimal value function faster than the standard value iteration. It also applies to the high-order truncated approximation.

The rest of the paper is organized as follows. Section 2 presents the preliminaries and problem formulation. Section 3 establishes fundamental structural results. Section 4 introduces the tree MDP formulation and proposes the truncated approximation. Section 5 develops the high-order truncated approximation. Section 6 proposes the nested value iteration algorithm. Section 7 demonstrates the experiment results. Finally, section 8 concludes this paper.

2 Problem Statement

2.1 Preliminary: MDP

An MDP is defined by a tuple $(\mathcal{S},\mathcal{A},P,r,\beta)$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $P$ is the transition kernel, $r:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ is the reward function, $\beta\in[0,1)$ is the discount factor. This paper assumes that both $\mathcal{S}$ and $\mathcal{A}$ are finite sets. For simplicity, we index the states and actions by letting ${\cal S}=\{1,2,\cdots,|{\cal S}|\}$ and ${\cal A}=\{1,2,\cdots,|{\cal A}|\}$. Let $s_{t}$ and $a_{t}$ denote the state and action at time $t$. The following expressions will be used interchangeably:

[TABLE]

We will use ${P_{a}}={[{P_{a}}(i,j)]_{i,j}}$ to denote the transition matrix associated with action $a$. Solving the MDP consists in finding an optimal policy that maximizes the expected total discounted reward

[TABLE]

More generally, given a distribution $\theta$ of the initial state, we can define

[TABLE]

where $\Delta_{\cal S}$ is the ($|\mathcal{S}|-1$)-dim probability simplex defined as

[TABLE]

At each time $t$, the state information $s_{t}\in\mathcal{S}$ is transmitted to the controller, based on which the controller computes an action $a_{t}$ and applies $a_{t}$ to the process. Upon the execution of $a_{t}$, the process generates a reward $r(s_{t},a_{t})$ and transitions to state $s_{t+1}$ at the next time step. In the classical setting, the transmissions of $s_{t}$ are always timely and reliable. Then the optimal policy for the MDP can be determined by the Bellman equation:

[TABLE]

2.2 Intermittently Observable MDP

Consider a scenario where the controller relies on a remote sensor for perceiving the state information of the process. Suppose that the controller and the remote sensor are physically separated and communicate via an unreliable wireless channel. Due to environmental interference and channel fading, transmissions over the wireless channel may occasionally fail. As a result, the state information observed by the sensor may not be always delivered successfully to the controller. This paper proposes the IOMDP framework to address the problem of selecting actions for the MDP with intermittent state information.

Formally, define a Bernoulli random variable $\gamma_{t}$ as an indicator for the transmission of $s_{t}$. Let $\gamma_{t}=1$ if $s_{t}$ is successfully transmitted to the controller before the time point of determining $a_{t}$ and $\gamma_{t}=0$ otherwise. We assume there is no delayed arrival of state information, i.e., the controller will never receive $s_{t}$ if $\gamma_{t}=0$. Further, assume that $\{{\gamma_{t}}:t=0,1,2,\cdots\}$ is a sequence of i.i.d. random variables and that

[TABLE]

Quantity $\rho$ is referred to as the state information reception probability (SIRP). An IOMDP with SIRP $\rho$ is defined by the tuple $(\mathcal{S},\mathcal{A},\rho,P,r,\beta)$.

To analyze the IOMDP, we reformulate the problem as a belief MDP denoted by ${\cal B}(\rho)=({\cal W},{\cal A},\rho,T,R,\beta)$. Specifically, ${\cal W}$ is the set of belief states, $T$ is the transition kernel of belief states, $R$ is the reward function. The remaining symbols are of the same meaning as before. The belief MDP is a special kind of POMDP, in which we use a belief state to represent a probability distribution over the state space $\mathcal{S}$. Let $t_{i}$ denote the time index of the $i$-th successfully transmitted state information. Due to possible transmission failures, $t_{i+1}-t_{i}$ may be greater than 1. Suppose there are in total $k$ successful transmissions up to time $t$ (i.e., $k=\sum_{i=0}^{t}\gamma_{i}$), then $\{s_{t_{i}}:1\leq i\leq k\}$ is the set of all the state information received by the controller up to time $t$. The belief state at time $t$, denoted by $\mathbf{b}_{t}$, is a sufficient statistic for the given history:

[TABLE]

The second equality above follows from the Markovian property. It means that the belief state depends on the latest received state information and the following actions. We will refer to the sequence $({s_{{t_{k}}}},{a_{{t_{k}}}},{a_{{t_{k}}+1}},\cdots,{a_{t-1}})$ as the sufficient history at time $t$. Note that ${{\bf{b}}_{t}}\in{\Delta_{\cal S}}$ with its $i$-th element, denoted by $\mathbf{b}_{t}(i)$, being the probability of $s_{t}=i$ conditioned on the sufficient history. Specially, if $t_{k}=t$ (i.e., the controller receives $s_{t}$), then $\mathbf{b}_{t}$ reduces to a one-hot vector. We will use $\mathbf{e}_{i}$ to denote the one-hot vector with the $i$-th element being 1 and other elements being 0. If $s_{t}$ is not delivered successfully, then $\mathbf{b}_{t}$ is determined by $\mathbf{b}_{t-1}$ and $a_{t-1}$. In particular, the transition probability of the belief MDP is given by

[TABLE]

where ${[P_{{a_{t-1}}}^{\top}{{\bf{b}}_{t-1}}]_{i}}$ denotes the $i$-th element of vector $P_{{a_{t-1}}}^{\top}{{\bf{b}}_{t-1}}$. Finally, the reward function of the belief MDP is given by

[TABLE]

The belief MDP can be viewed as a fully observable MDP; hence its optimal policy can be determined by the Bellman equation:

[TABLE]

where we use $\phi({\bf{b}},\rho)$ to denote the optimal value function of ${\cal B}(\rho)$. In the context that $\rho$ is fixed, we may omit $\rho$ and simply write the value function as $\phi({\bf{b}})$. For every $\mathbf{b}\in{\cal W}$ and $a\in{\cal A}$, $R(\mathbf{b},a)$ can be viewed as the expected one-step reward generated by the underlying MDP given that the current state follows a distribution $\mathbf{b}$. Therefore, $\phi({\bf{b}},\rho)$ is the maximum expected total discounted reward that can be obtained from the underlying MDP with an initial belief state $\mathbf{b}$ and SIRP $\rho$.

In principle, the belief state can be any element in ${\Delta_{\cal S}}$ (i.e., ${\cal W}={\Delta_{\cal S}}$). Then the belief MDP has a continuous state space. However, as implied by (4), given any initial belief state $\theta$, the set of possible belief states is countable, i.e.,

[TABLE]

Unless otherwise specified, we will consider ${\cal W}={\Omega_{\theta}}$ instead of ${\Delta_{\cal S}}$ to avoid technique issues related to measurability.

3 Structural Results

This section establishes structural results that are fundamental for understanding IOMDPs. The aim is to identify the relationship between the state information reception probability and the optimal value function.

Let $\Pi$ denote the set of stationary deterministic policies for the belief MDP. It is well-known that $\Pi$ contains at least one optimal policy; hence we will focus on policies in $\Pi$. For any $\pi\in\Pi$, let $\pi(\mathbf{b})$ denote the action taken by policy $\pi$ in belief state $\mathbf{b}$. Denote by ${\phi^{\pi}}(\cdot,\rho)$ the value function of ${\cal B}(\rho)$ under policy $\pi$ and $T^{\pi}_{\rho}$ the associated transition matrix. Their relationship can be expressed in vector form as

[TABLE]

where ${\phi^{\pi}}[\rho]={[{\phi^{\pi}}({\bf{b}},\rho)]_{{\bf{b}}\in{\cal W}}}$ is the vector form of the value function over the belief state space ${\cal W}$, and ${R_{\pi}}={[R({\bf{b}},\pi({\bf{b}}))]_{{\bf{b}}\in{\cal W}}}$ is the vector of rewards associated with policy $\pi$. Likewise, we will denote by $\phi[\rho]$ the vector form of the optimal value function of ${\cal B}(\rho)$.

We first introduce a useful lemma. For any $\rho\in(0,1]$, ${\cal B}(\rho)$ is a POMDP whose reward function is linear with the belief state. The piecewise linearity and convexity of the optimal value function have been extensively exploited in POMDPs of this kind (Smallwood and Sondik, 1973). Formally,

Lemma 1

For any $\rho\in(0,1]$, $\phi({\bf{b}},\rho)$ is a piecewise linear and convex function of $\mathbf{b}\in\Delta_{\cal S}$.

With Lemma 1, we are ready to establish a key result that reveals the relationship between the optimal value function and the SIRP.

Theorem 2

For any $\mathbf{b}\in{\cal W}$, $\phi({\bf{b}},\rho)$ is an increasing function of $\rho\in(0,1]$.

**Proof ** See Appendix A.

Theorem 2 is consistent with our intuition. It shows that the optimal value function of the belief MDP is monotonically increasing with the state information reception probability. In other words, the more state information the controller can receive, the greater its expected total discounted reward from the underlying MDP.

The following result is useful in robustness and worst-case performance analysis.

Corollary 3

For any $\rho\in(0,1]$, suppose that $\pi_{\rho}\in\Pi$ is an optimal policy for ${\cal B}(\rho)$. Then for any $v\in(0,\rho)$ and $u\in(\rho,1]$, ${\phi^{{\pi_{\rho}}}}[v]\leq{\phi^{{\pi_{\rho}}}}[\rho]\leq{\phi^{{\pi_{\rho}}}}[u]$.

**Proof ** The second inequality has been established in the proof of Theorem 2. Since $\rho>v$, it follows from Theorem 2 that ${\phi^{{\pi_{\rho}}}}[\rho]=\phi[\rho]\geq\phi[v]\geq{\phi^{{\pi_{\rho}}}}[v]$.

Although straightforward, the lemma below is interesting and worth being highlighted. If we apply a value iteration algorithm to solve ${\cal B}(\rho)$ with any $\rho<1$, $\phi(\cdot,1)$ is a good starting point for the value iteration. As we know, a good starting point may considerably speed up the value iteration algorithm. Since the underlying MDP is much easier to be solved than the belief MDP, the following result can be used in practical computations to reduce computation time.

Lemma 4

For $\rho=1$, the optimal value function $\phi(\cdot,1)$ can be derived from $V$. In particular,

$\phi({{\bf{e}}_{i}},1)=V(i)$, $\forall i\in{\cal S}$;

• 2.

For ${\bf{b}}\notin\{{{\bf{e}}_{i}}:i\in{\cal S}\}$,

[TABLE]

**Proof ** The initial belief state being ${\bf e}_{i}$ means that the controller has observed that the initial state of the underlying MDP is $s_{0}=i$. Since $\rho=1$, the controller can receives $s_{t}$ for all $t$. As a result, the belief state always remains in the set $\{{{\bf{e}}_{i}}:i\in{\cal S}\}$ and the belief MDP reduces to the original MDP. Hence $\phi({{\bf{e}}_{i}},1)=V(i),\forall i\in{\cal S}$. Statement 2 follows immediately from the Bellman equation and statement 1.

Clearly, $\phi({{\bf{e}}_{i}},1)=V(i)=J({{\bf{e}}_{i}})$ for all $i\in{\cal S}$. It is interesting that, for any ${\bf{b}}\notin\{{{\bf{e}}_{i}}:i\in{\cal S}\}$, $\phi({\bf{b}},1)$ given by Lemma 4 is not equal to $J({\bf b})$ defined in (2). Both of them consider that the initial state of the underlying MDP follows a distribution ${\bf b}$. However, $J({\bf b})$ in the classical setting assumes that the controller can observe exactly the initial state $s_{0}$ before it computes the first action $a_{0}$; hence $a_{0}$ depends on the specific realization of $s_{0}$ and is always optimal. By contrast, $\phi({\bf{b}},1)$ corresponds to the case that the controller does not know the exact initial state and needs to determine $a_{0}$ only based on ${\bf b}$; consequently, $a_{0}$ is likely to be sub-optimal for a particular realization of $s_{0}$. Therefore, $\phi({\bf{b}},1)\leq J({\bf{b}})$ and the difference $J({\bf{b}})-\phi({\bf{b}},1)$ is the performance regret of the belief MDP generated by the uncertainty at the first time step.

The above discussion is useful for understanding the next theorem, where we provide a bound for the performance regret caused by state information losses.

Theorem 5

For any $\mathbf{b}\in{\cal W}$, $\phi({\bf{b}},\rho)$ is continuous at any $\rho\in(0,1)$. In addition, denote by $\pi_{1}$ an optimal policy for ${\cal B}(1)$. Then, for any $\rho\in(0,1]$,

[TABLE]

where $\eta:{\cal W}\to\mathbb{R}$ is given by (let $\tau=P_{{\pi_{1}}({\bf{b}})}^{\top}{\bf{b}}$)

[TABLE]

**Proof ** See Appendix A.

Basically, the upper bound for the performance regret is obtained by applying the optimal policy of ${\cal B}(1)$ (i.e., $\pi_{1}$) to the belief MDP ${\cal B}(\rho)$ and comparing the resulting value function with $\phi(\cdot,1)$. Start from a belief state ${\bf b}$, ${\cal B}(1)$ and ${\cal B}(\rho)$ generate the same instantaneous reward $R({\bf b},\pi({\bf b}))$ when they are controlled by the same policy $\pi_{1}$; the performance regret comes from the second step. For ${\cal B}(1)$ under policy $\pi_{1}$, the belief state transitions to ${\bf e}_{i}$ with probability $\tau(i)$ for each $i\in{\cal S}$. By contrast, for ${\cal B}(\rho)$ under policy $\pi_{1}$, the belief state transitions to ${\bf e}_{i}$ with probability $\rho\tau(i)$ and transitions to $\tau$ with probability $1-\rho$. The regret comes from the latter transition, as illustrated above. We can think of the upper bound of $\phi[1]-\phi[\rho]$ stated in Theorem 5 as the expected total discounted regret of a Markov regret process (MRP). In particular, the MRP is governed by the transition matrix $T_{\rho}^{{\pi_{1}}}$, and the regret at state ${\bf b}$ is $\eta({\bf b})$.

4 Truncated Approximation

Although the belief MDP is much simpler than general POMDPs, it is still hard to solve exactly due to the countably infinite state space. We thus reformulate the IOMDP as a tree MDP, based on which we propose two finite-state approximations for finding near-optimal policies. In the following sections, we consider $\rho\in(0,1]$ to be fixed but arbitrary. We thus drop $\rho$ from notations to simplify the expressions. For example, $\phi({\bf{b}},\rho)$ will be written as $\phi({\bf b})$ and the associated vector form will be written as $\phi$.

4.1 Tree MDP

We next reformulate the IOMDP to ease expositing of the finite-state approximations. As defined in (3), the belief state at any time is determined by the sufficient history at the controller. In particular, a sufficient history, denoted by $h=(s,{u_{1}},{u_{2}},\cdots,{u_{n}})$, is a tuple consisting of a state $s\in{\cal S}$ and a sequence of $n$ ordered actions, where $n\in\{0,1,2,\cdots\}$ and $u_{k}\in{\cal A}$ for all $k$. When $n=0$, we will simply write $h=s$. Denote by ${\cal H}$ the set of all sufficient histories. Then according to the definition of belief states, ${\cal H}$ and ${\cal W}$ are connected by function $g:{\cal H}\to{\cal W}$ defined as

[TABLE]

where $\prod\nolimits_{k=1}^{n}{P_{{u_{k}}}^{\top}}=I$ for $n=0$. Note that function $g$ is not necessarily a one-to-one mapping. It is possible that $g(h_{1})=g(h_{2})$ for $h_{1}\neq h_{2}$.

The set ${\cal H}$ can be hierarchically organized in a tree with $|{\cal S}|$ roots and infinite layers. In particular, each element in ${{\cal G}_{0}}:={\cal S}$ corresponds to a root. For $n\in\{1,2,\cdots\}$, denote ${\cal G}_{n}$ the set of sufficient histories with $n$ actions in the tuple. Then set ${\cal G}_{n}$ consists of the nodes at layer $n$ of the tree. Every $h\in{\cal G}_{n}$ has $|{\cal A}|$ children, each corresponds to adding an extra action $a\in{\cal A}$ at the end of the tuple, i.e., $(h,a)$. We then define $\{{\cal G}_{n}:n=0,1,2,\cdots\}$ recursively by

[TABLE]

Fig.1 is an example of sufficient histories organized in a tree structure. Using the terminology in tree structures, $h$ will be referred to as the parent node of $(h,a)$. Except for the root nodes (i.e., those in ${\cal G}_{0}$), each node in the tree has a unique parent. In addition, ${\cal G}_{k}$ and ${\cal G}_{n}$ are disjoint for any $n\neq k$ and ${\cal H}=\cup_{k=0}^{\infty}{{\cal G}_{k}}$. For any positive integer $n$, let ${\cal H}_{n}=\cup_{k=0}^{n}{{\cal G}_{k}}$ denote the set of nodes from layer 0 to layer $n$.

Considering ${\cal H}$ as the state space, we can reformulate the IOMDP as an MDP denoted by ${\cal C}(\rho)=({\cal H},{\cal A},\rho,D,{\bar{R}},\beta)$. In particular, the reward function is $\bar{R}(h,a):=R\left({g(h),a}\right)$ and the transition kernel is given by (cf. eq.(4))

[TABLE]

We call ${\cal C}(\rho)$ a tree MDP because of the tree structure of its state space. To distinguish from the belief state, the state of the tree MDP at time $t$, denoted by $h_{t}\in{\cal H}$, will be referred to as the position state. Let $\varphi:{\cal H}\to\mathbb{R}$ denote the optimal value function of ${\cal C}(\rho)$. We can easily identify the relationship between the two formulations of the IOMDP, as stated in the lemma below. The proof is provided in Appendix B for the sake of completeness.

Lemma 6

For the two formulations ${\cal B}(\rho)$ and ${\cal C}(\rho)$,

$\phi({\bf b})=\varphi(h)$* if ${\bf b}=g(h)$;*

• 2.

$\varphi(h)=\varphi(h^{\prime})$* for any $h,h^{\prime}\in{\cal H}$ satisfying $g(h)=g(h^{\prime})$;*

• 3.

Denote by $\pi$ and $\mu$ the optimal policies for ${\cal B}(\rho)$ and ${\cal C}(\rho)$, respectively. Then $\pi({\bf b})=\mu(h)$ if ${\bf b}=g(h)$.

The two formulations, although similar, have different advantages. As shown in the previous section, the belief MDP formulation allows mathematical analysis of the optimal value function, thus establishing some fundamental properties of the IOMDP. By contrast, we find it convenient to develop finite-state approximations and compute near-optimal policies for the problem using the tree MDP formulation.

4.2 Truncated Approximation

Our first finite-state approximation for ${\cal C}(\rho)$ is inspired by a simple fact: starting from the root layer, the position state $h_{t}$ can arrive at layer $n$ only if the controller suffers $n$ consecutive transmission failures. The probability of such an event is $(1-\rho)^{n}$, which decreases exponentially as $n$ increases. When the probability of $h_{t}$ visiting ${\cal G}_{n}$ is small enough, the expected performance regret caused by taking a sub-optimal action in $h_{t}\in{\cal G}_{n}$ should be negligible.

Let us consider the following $L$-truncated approximation.

Definition 7** ($L$-Truncated Approximation, TA($L$))**

Given a positive integer $L$, the $L$-truncated approximation of ${\cal C}(\rho)$ is an MDP denoted by ${\cal C}{{}_{L}}(\rho)=({{\cal H}_{L}},{\cal A},\rho,{D_{L}},\bar{R},\beta)$. In particular, the state space of ${\cal C}_{L}(\rho)$ is ${{\cal H}_{L}}=\cup_{n=0}^{L}{{\cal G}_{n}}$. The action space ${\cal A}$, SIRP $\rho$, reward function $\bar{R}$, and discount factor $\beta$ are identical to those of ${\cal C}(\rho)$. The transition kernel $D_{L}$ is defined as follows:

• (1)

For any $h\in{{\cal G}_{n}}$ with $n\leq L-1$, ${D_{L}}(h^{\prime}|h,a)=D(h^{\prime}|h,a)$ for all $h^{\prime}\in{\cal H}_{L}$ and $a\in{\cal A}$;

• (2)

For any $h\in{{\cal G}_{L}}$, let $\tau_{a}=(h,a)$ and ${\bf b}_{a}=g(\tau_{a})$, where $a\in{\cal A}$. Then

[TABLE]

The state space of TA($L$) consists of position states at the first $L+1$ layers. Therefore, for any $h\in{{\cal G}_{L}}$, $(h,a)\in{{\cal G}_{L+1}}$ is not a state of ${\cal C}_{L}(\rho)$. We thus redefine the transition probabilities for $h\in{{\cal G}_{L}}$ in ${\cal C}_{L}(\rho)$. Fig.2 compares the transition probabilities, in ${\cal C}(\rho)$ and ${\cal C}_{L}(\rho)$, of taking a particular action in $h\in{{\cal G}_{L}}$. TA($L$) can be viewed as an adaptation for practical operations. That is, the controller does not update the position state (and the associated belief state) after it experiences more than $L$ consecutive transmission failures until it receives a new state observation. We will use $\varphi_{L}(\cdot)$ to denote the optimal value function of ${\cal C}_{L}(\rho)$. It turns out that $\varphi_{L}(h)$ is a good approximator of $\varphi(h)$ for part of $h\in{\cal H}_{L}$. The theorem below provides a bound for the approximation error of TA($L$).

Theorem 8

For any $\rho\in(0,1]$ and positive integer $L$,

[TABLE]

where ${\delta_{L}}=\mathop{\max}\limits_{x\in{{\cal G}_{L}}\cup{{\cal G}_{L+1}}}\varphi(x)$.

**Proof ** See Appendix B.

The upper bound presented in Theorem 8 is insightful. For $h\in{\cal G}_{k}$, we refer to $k$ as the layer index of $h$. According to Theorem 8, the error bound is an increasing function of $k\in\{0,1,\cdots,L\}$ for any fixed $L$, implying that the difference between the two value functions can be larger for $h$ in layers closer to the truncated layer (i.e., ${\cal G}_{L}$). The monotonicity of the error bound w.r.t. the layer index can be interpreted as follows. The approximation error comes from the truncated layer since the position states in that layer have different transition probabilities from ${\cal C}(\rho)$. Starting from the $k$-th layer, the position state can arrive at the truncated layer only if the controller experiences at least $L-k+1$ consecutive transmission failures, with the associated probability and discounting factor being $(1-\rho)^{L-k+1}$ and $(1-\beta)^{L-k+1}$, respectively. As a result, when $k=0$, the error bound reduces to ${\delta_{L}}{\beta^{L+1}}{(1-\rho)^{L+1}}/(1-\beta)$, which can be arbitrarily small if $L$ is large enough. While for $h\in{\cal G}_{L}$, the approximation error is dominated by the term $\beta(1-\rho){\delta_{L}}$, which may be non-negligible.

A near-optimal approximation for the value function of an MDP is useful because it can be used to derive a near-optimal policy. For the tree MDP ${\cal C}(\rho)$, the optimal action of $h\in{\cal G}_{k}$ is determined by the Bellman equation

[TABLE]

where $i\in{\cal G}_{0}$ and $\tau_{a}:=(h,a)\in{\cal G}_{k+1}$. Therefore, if $\varphi_{L}(h^{\prime})$ is very close to $\varphi(h^{\prime})$ for all $h^{\prime}\in{{\cal G}_{0}}\cup{{\cal G}_{k+1}}$, we can obtain a near-optimal action for every $h\in{\cal G}_{k}$ by replacing $\varphi(\cdot)$ with $\varphi_{L}(\cdot)$ in (11). We say that the resulting action is $\epsilon$-optimal for ${\cal C}(\rho)$ in position state $h$ if $\left|{{\varphi_{L}}(h^{\prime})-\varphi(h^{\prime})}\right|<\epsilon$ for all $h^{\prime}\in{{\cal G}_{0}}\cup{{\cal G}_{k+1}}$.

Denote by $\mu_{L}$ an optimal policy for ${\cal C}_{L}(\rho)$ and $\mu_{L}(h)$ the associated action for $h\in{\cal H}_{L}$. According to Theorem 8, given any $\epsilon>0$ and positive integer $n$, there exists a finite integer $L_{n}\geq n$ such that $\left|{{\varphi_{L}}(h)-\varphi(h)}\right|<\epsilon$ for all $L\geq L_{n}$ and $h\in\cup_{k=0}^{n}{{\cal G}_{k}}$. Consequently, for any $L\geq L_{n}$ and $h\in\cup_{k=0}^{n}{{\cal G}_{k}}$, $\mu_{L}(h)$ is $\epsilon$-optimal for ${\cal C}(\rho)$. With $\mu_{L}$, we can derive a policy for ${\cal C}(\rho)$ to control the underlying MDP in the presence of state information losses. In particular,

Definition 9** (TA($L$) Policy)**

For any positive integer $L$, define the TA($L$) policy for ${\cal C}(\rho)$ as follows:

• (1)

For $h\in{\cal H}_{L}$, the TA($L$) policy takes action $\mu_{L}(h)$;

• (2)

For $h\notin{\cal H}_{L}$, denote by $h^{\prime}\in{\cal G}_{L}$ the ancestor node of $h$ in ${\cal G}_{L}$. Then the TA($L$) policy takes action $\mu_{L}(h^{\prime})$ in position state $h$.

Intuitively, the larger the value of $L$, the closer the TA($L$) policy is to the optimal policy. More analysis about the TA($L$) policy will be provided in the next section.

5 High-order Truncated Approximation

Although TA($L_{n}$) can be used to determined an $\epsilon$-optimal action for each $h\in\cup_{k=0}^{n}{{\cal G}_{k}}$, we typically have $L_{n}>n$ for small $\epsilon$. Hence a relatively large $L_{n}$ is needed to obtain a satisfactory policy. This makes the method inefficient and usually prohibitively complex since the state space of TA($L$) increases exponentially with $L$:

[TABLE]

As a result, TA($L$) is limited to IOMDPs with large $\rho$ and small action spaces. We next put forth a more efficient and scalable approach to finding near-optimal policies for ${\cal C}(\rho)$. The following concepts are important for developing the approach.

Definition 10

Given a policy $\mu$, a position state of ${\cal C}(\rho)$ is said to be reachable under policy $\mu$ if it can be visited within a finite time with a positive probability. A position state is redundant if it is not reachable.

As discussed around (8), every position state $h$ has $|{\cal A}|$ children, among which $(h,a)$ can be visited only if the controller takes action $a$ in $h_{t}=h$ and fails to receive $s_{t+1}$. Therefore, using a deterministic policy, each position state has only one reachable child. That is, each layer in the tree contains exactly $|{\cal S}|$ reachable position states. If we fix the optimal action for $h$ to be, say $\mu(h)$, then $(h,a)$ will never be visited if $a\neq\mu(h)$ (except for the initial position states). That is, under the optimal policy, $(h,\mu(h))$ is reachable, and the remaining children of $h$ are redundant. As far as computing the optimal policy is concerned, we can omit the redundant position states and remove them from the state space.

While the number of reachable position states at each layer is fixed (i.e, $|{\cal S}|$), the total number of position states increases exponentially over layers—most of them are redundant. This is the source of the inefficiency and poor scalability of TA($L$). The computation complexity can be significantly reduced if we can remove the redundant position states when computing a policy.

Let us consider the following idea to compute $\epsilon$-optimal actions for $h\in\cup_{k=0}^{n}{{\cal G}_{k}}$. Instead of solving for TA($L_{n}$) with a large $L_{n}$, we first select an $L_{0}<L_{n}$ so that $L_{0}$ is large enough for computing $\epsilon$-optimal actions for all $h\in{\cal G}_{0}$ by solving TA($L_{0}$). Upon determining an $\epsilon$-optimal action for every $h\in{\cal G}_{0}$, we can identify the reachable position states in ${\cal G}_{1}$. The next step is to compute $\epsilon$-optimal actions for the reachable position states in ${\cal G}_{1}$—we do not compute actions for the redundant position states since they will never be reached. For this purpose, we construct a modified TA($L_{0}+1)$, which will be referred to as TA($1,L_{0}$), by removing the redundant position states in ${\cal G}_{1}$ and their descendants at other layers from the state space of TA($L_{0}+1$). Solving for TA($1,L_{0}$) is considerably easier than TA($L_{0}+1$) due to the reduction of state space, but it still provides $\epsilon$-optimal actions for the reachable position states in ${\cal G}_{1}$. Doing so repeatedly, we can compute $\epsilon$-optimal actions for reachable position states layer by layer. We call this method high-order truncated approximation, where the $n$-th order truncated approximation refers to repeating the above procedure $n$ times and will be denoted by TA($n,L$). We will justify this method after providing a formal definition below.

We formally define TA($n,L$) in a recursive way for $n\in\{0,1,2,\cdots\}$ and a fixed $L\in\{1,2,3,\cdots\}$. In particular, TA($n,L$) is an MDP denoted by ${\cal C}^{n}_{L}(\rho)$ with ${\cal C}^{0}_{L}(\rho)\triangleq{\cal C}_{L}(\rho)$. Let $\mu^{n}_{L}$ stand for the optimal policy for ${\cal C}^{n}_{L}(\rho)$. As discussed before, with a proper $L$, $\mu^{0}_{L}(h)=\mu_{L}(h)$ is an $\epsilon$-optimal action for ${\cal C}(\rho)$ in position state $h\in{\cal G}_{0}$. Our aim is to define a series of $\{{\cal C}^{n}_{L}(\rho):n=1,2,3,\cdots\}$ such that, for every reachable $h$ in ${\cal G}_{n}$, $\mu^{n}_{L}(h)$ is an $\epsilon$-optimal action for ${\cal C}(\rho)$. Meanwhile, the redundant position states in ${\cal G}_{n}$ and their descendants at other layers will be excluded from the state space of ${\cal C}^{n}_{L}(\rho)$ to reduce computation complexity.

Mathematically, ${\cal C}_{L}^{n}(\rho)=({\cal H}_{L}^{n},\{{\cal A}_{h}^{n}\},\rho,D_{L}^{n},\bar{R},\beta)$ is an MDP with state space

[TABLE]

where ${{\cal Z}_{i}}\subseteq{{\cal G}_{i}}$ is the set of reachable position states in ${\cal G}_{i}$ under the policy $\mu^{i-1}_{L}$:

[TABLE]

and ${\cal G}_{n+i}^{-}\subset{{\cal G}_{n+i}}$ is the set of descendants of ${\cal Z}_{n}$ at layer $(n+i)$:

[TABLE]

Fig. 3 gives an example of the state space of TA($n,L$) for $n=1,L=3$. The action space for any $h\in{\cal H}^{n}_{L}$ is given by

[TABLE]

The first line of the above definition means to fix the action for the already determined reachable states at layers 0 to $n-1$. With this restriction, we have $\mu_{L}^{n}(h)=\mu_{L}^{n-1}(h)$ for all $h\in\cup_{i=0}^{n-1}{{\cal Z}_{i}}$. The SIRP $\rho$, reward function $\bar{R}$, and discount factor $\beta$ are identical to those of ${\cal C}(\rho)$. The transition kernel $D^{n}_{L}$ is defined as follows:

• (1)

For $h\in\left({\cup_{i=0}^{n}{{\cal Z}_{i}}}\right)\cup\left({\cup_{i=n+1}^{L+n-1}{\cal G}_{i}^{-}}\right)$, $D_{L}^{n}(h^{\prime}|h,a)=D(h^{\prime}|h,a)$ for any $h^{\prime}\in{\cal H}^{n}_{L}$ and $a\in{\cal A}^{n}_{h}$;

• (2)

For $h\in{\cal G}_{L+n}^{-}$, let $\tau_{a}=(h,a)$ and ${\bf b}_{a}=g(\tau_{a})$, where $a\in{\cal A}$. Then

[TABLE]

This completes the definition of ${\cal C}^{n}_{L}(\rho)$. Note that $|{{\cal Z}_{i}}|=|{{\cal G}_{0}}|$ for all $1\leq i\leq n$ and that ${\cal H}_{L}^{n}\subseteq{{\cal H}_{L+n}}$ can be determined with $\mu^{n-1}_{L}$. Starting from $n=0$ and ${\cal Z}_{0}={\cal G}_{0}$, we compute the optimal policy $\mu^{n}_{L}$ for ${\cal C}^{n}_{L}(\rho)$ and then use $\mu^{n}_{L}$ and ${\cal Z}_{n}$ to identify ${\cal Z}_{n+1}$ (the set of reachable position states in ${\cal G}_{n+1}$); then the descendants of ${\cal Z}_{n+1}$ at the following $L-1$ layers (i.e., ${\cal G}_{n+i}^{-}$ for $2\leq i\leq L+1$) can be identified. On this basis, we construct ${\cal C}^{n+1}_{L}(\rho)$ to compute the actions for position states in ${\cal Z}_{n+1}$. Doing so iteratively with a proper $L$ yields the optimal actions of the reachable position states layer by layer (see Theorem 13). We can stop when $(1-\rho)^{n+1}$ is small enough and control ${\cal C}(\rho)$ using the TA($n,L$) policy derived from $\mu^{n}_{L}$:

Definition 11** (TA($n,L$) Policy)**

For any positive integers $L$ and $n$, define the TA($n,L$) policy for ${\cal C}(\rho)$ as follows:

• (1)

For $h\in{\cal H}^{n}_{L}$, the TA($n,L$) policy takes action $\mu^{n}_{L}(h)$;

• (2)

For $h\notin{\cal H}^{n}_{L}$, denote by $h^{\prime}\in{\cal G}_{L+n}$ the ancestor node of $h$ in ${\cal G}_{L+n}$. Then the TA($n,L$) policy takes action $\mu^{n}_{L}(h^{\prime})$ in position state $h$.

We next establish the relationship between ${\cal C}^{n}_{L}(\rho)$ and ${\cal C}_{L+n}(\rho)$, and then analyze the TA($L$) and TA($n,L$) policies. Denote by $\varphi_{L}^{n}:{\cal H}_{L}^{n}\to\mathbb{R}$ the optimal value function of ${\cal C}^{n}_{L}(\rho)$. The following lemma is useful.

Lemma 12

For any fixed $L$, if ${\cal C}^{n}_{L}(\rho)$ and ${\cal C}_{L+n}(\rho)$ have the same optimal action for every $h\in\cup_{i=0}^{n-1}{{\cal Z}_{i}}$, then ${\varphi_{L+n}}(h)=\varphi_{L}^{n}(h)$ for any $h\in{\cal H}^{n}_{L}$.

**Proof ** See Appendix C.

For the tree MDP ${\cal C}(\rho)$, the Bellman equation (11) can be expressed as $\varphi(h)=\mathop{\max}\limits_{a\in{\cal A}}\left\{{Q(h,a)}\right\}$, where $Q:{\cal H}\times{\cal A}\to\mathbb{R}$ is the state-action value function:

[TABLE]

Denote by $\mu(h)\in{\cal A}$ an optimal action of $h$. Let

[TABLE]

Since the action space is a finite set, we have $\epsilon>0$ for any $h$. As mentioned before, given the optimal policy $\mu_{L}$ for ${\cal C}_{L}(\rho)$ and the associated optimal value function $\varphi_{L}(h)$, we say that $\mu_{L}(h)$ is an $\epsilon$-optimal action for ${\cal C}(\rho)$ in position state $h\in{\cal G}_{k}$ if $\left|{{\varphi_{L}}(h^{\prime})-\varphi(h^{\prime})}\right|<\epsilon$ for all $h^{\prime}\in{{\cal G}_{0}}\cup{{\cal G}_{k+1}}$. The following fact can be easily verified:

Fact 1 *For any $h\in{\cal H}_{L-1}$ in the tree MDP ${\cal C}(\rho)$, if $\mu_{L}(h)$ is an $\epsilon$-optimal action with $\epsilon\leq\epsilon_{h}/2$, then, in fact, ${\mu_{L}}(h)=\arg{\max_{a\in{\cal A}}}\left\{{Q(h,a)}\right\}$ is optimal for ${\cal C}(\rho)$. *

According to Fact 1, define

[TABLE]

We show in the following theorem that, with a proper $L$, solving for ${\cal C}^{n}_{L}(\rho)$ yields $\epsilon(n)$-optimal (thus optimal) actions for all reachable position states at layers 0 to $n$. Recall that $\mu^{n}_{L}$ and $\mu_{L+n}$ denote the optimal policies for ${\cal C}^{n}_{L}(\rho)$ and ${\cal C}_{L+n}(\rho)$, respectively.

Theorem 13

For any positive integer $n$, given an $L$ satisfying

[TABLE]

where $\delta(n,L)=\mathop{\max}\limits_{x\in{{\cal H}_{L+n}}}\varphi(x)$. Then,

• (1)

$\mu_{L+n}(h)$* is an optimal action for any $h\in\cup_{k=0}^{n}{{\cal G}_{k}}$ in ${\cal C}(\rho)$.*

• (2)

$\mu^{n}_{L}(h)$* is an optimal action for any $h\in\cup_{k=0}^{n}{{\cal Z}_{k}}$ in ${\cal C}(\rho)$.*

**Proof ** See Appendix C.

Theorem 13 shows that both TA($L+n$) and TA($n,L$) can be used to obtain optimal actions for reachable position states at layers 0 to $n$. However, it is easy to see that TA($n,L$) is much more efficient than TA($L+n$). The cardinality of the state space of ${\cal C}^{n}_{L}(\rho)$ is increasing linearly with $n$, that is,

[TABLE]

where $\left|{{{\cal G}_{0}}}\right|=\left|{\cal S}\right|$. By contrast, (12) shows that $|{{\cal H}_{L+n}}|$ increases exponentially with $n$ for any fixed $L$. Although solving ${\cal C}^{n}_{L}(\rho)$ needs to solve ${\cal C}^{k}_{L}(\rho)$ for all $k\leq n$, the overall computation complexity of TA($n,L$) is still significantly lower than that of TA($L$). In addition, if we use a value iteration algorithm to solve ${\cal C}^{n}_{L}(\rho)$, then the optimal value function of ${\cal C}^{n-1}_{L}(\rho)$ is a good starting point for the value iteration. Since ${\varphi}^{n-1}_{L}(h)$ is close to ${\varphi}^{n}_{L}(h)$ for $h\in\cup_{i=0}^{n-1}{{\cal Z}_{i}}$. It is well-known that the value iteration converges quickly if the starting point is near the optimum.

6 Nested Value Iteration

We have shown in the previous section that solving for TA($L$) and TA($n,L$) could offer near-optimal policies for the IOMDP. This section presents a variant of the value iteration algorithm, called nested value iteration (NVI), to compute the optimal policy for TA($L$) efficiently. It also applies to TA($n,L$).

The algorithm is inspired by the special structure of the tree MDP. As indicated by (11), position states at the root layer (i.e., ${\cal G}_{0}$) play a central role in the Bellman equation since their values directly affect the values of all position states. Intuitively, for a non-zero $\rho$, the larger the layer index of $h$, the weaker the influence of $h$’s value on the overall value function. The basic idea of NVI is to update the values of important position states more frequently than the less important ones, as shown in Algorithm 1.

In particular, based on the tree structure of the state space of ${{\cal C}_{L}}(\rho)$, we define a collection of nested sets as follows:

[TABLE]

Note that ${\cal H}_{l}\subset{\cal H}_{l+1}$ for all $l$ and the largest set is ${\cal H}_{L}$—the entire state space of ${{\cal C}_{L}}(\rho)$. As commented in Algorithm 1, NVI consists of outer and inner iterations. Each outer iteration consists of $L$ inner iterations, where the $l$-th inner iteration only updates the values of position states in set ${\cal H}_{L-l+1}$ ($1\leq l\leq L$). Consequently, position states in ${\cal G}_{0}$ are updated in every inner iteration, while position states in ${\cal G}_{L}$ are updated once in every outer iteration (i.e., every $L$ inner iterations).

The following theorem is useful for understanding the NVI algorithm.

Theorem 14

Let $\{\psi^{n}\}$ denote the sequence of functions generated by the nested value iteration algorithm with input ${{\cal C}_{L}}(\rho)$. Then

• (1)

$\psi^{n}$* converges in max norm to the optimal value function $\varphi_{L}$;*

• (2)

Let $n_{k}=kL+1$, then for all $k\in\{0,1,2,\cdots\}$,

[TABLE]

where $i\in\{0,1,\cdots,L\}$ and $c(i)=\max\{L-i,1\}$.

**Proof ** See Appendix D.

Theorem 14 is useful for interpreting the advantage of NVI to the standard value iteration. For $h\in{\cal G}_{0}$, the convergence rate of NVI is identical to that of standard value iteration in general cases (Puterman, 1994), that is,

[TABLE]

However, the computation complexity (in terms of the number of states to be updated) of $L$ consecutive inner iterations is significantly lower than that of $L$ standard iterations. That is, NVI speeds up the convergence of position states in the upper layers of the tree structure. The fast convergence of root position states, in turn, would speed up the convergence of other position states due to the strong influence of root position states on the overall value function. For example, the value of $h\in{\cal G}_{L}$ is updated once in each outer iteration; its convergence rate over outer iterations, according to Theorem 14, is ${\beta^{L}}\rho+\beta(1-\rho)$, which is smaller than the rate $\beta$ in standard value iteration.

Although Algorithm 1 is a natural form of NVI, modifying Algorithm 1 by defining the collection of nested sets in a different way may also perform well. The basic principles are: (i) the root position states in the tree should be contained in the smallest set so that they can be updated more frequently than others; (ii) the largest set should be the entire state space so that all states can be updated at least once in each outer iteration. It is also worth noting that the order of inner iterations within each outer iteration matters. We call the current order descending because the updating set shrinks from ${\cal H}_{L}$ to ${\cal H}_{1}$ in each outer iteration. Be careful that adopting the ascending order may lead to a worse convergence rate. This can be seen by using a similar argument as in the proof of Theorem 14. We do not provide details here since it is not the focus of this paper. Numerical results will be provided in the next section to validate the importance of descending order of inner iterations.

Finally, extending NVI to solve TA($n,L$) is straightforward. For ${\cal C}^{n}_{L}(\rho)$ with $n\geq 1$, the action of $h\in\cup_{i=0}^{n-1}{{\cal Z}_{i}}$ is already determined, hence there is no need to update the value of these position states. The values of root position states are identical to that of ${\cal C}^{0}_{L}(\rho)$, which can be used to update the value of other position states. We thus can define the collection of nested sets as follows:

[TABLE]

Replacing ${\cal H}_{l}$ with ${\cal H}_{n,l}$ in Algorithm 1, NVI can be used to solve TA($n,L$). Our previous discussions also imply that using the optimal value function of ${\cal C}^{n-1}_{L}(\rho)$ as the starting point would help NVI converge quickly to the optimal value function of ${\cal C}^{n}_{L}(\rho)$, as the values of root position states are already optimum.

7 Numerical Results

We present numerical results to demonstrate the effectiveness of our methods. Throughout this section, the state information transmissions of all experiments were simulated by Bernoulli trials generated by Matlab on PC. At each time step, the controller of the IOMDP receives the current state information only if the outcome of the Bernoulli trail is 1 (another outcome is 0). The probability of the outcome being 1 in each Bernoulli trial corresponds to the transmission success probability (i.e., SIRP) of the experiment.

We first compare NVI with standard value iteration (VI) in a group of randomly generated MDPs, as shown in Table 1. The results of reverse NVI (R-NVI) are added to demonstrate the importance of the descending order of inner iterations. Specifically, R-NVI refers to NVI with ascending order for inner iterations (i.e., the $i$-th inner iteration of each outer iteration updates states in ${\cal H}_{i}$), as discussed in Section 6. For each experiment in Table 1, the transition matrices and the reward function of the underlying MDP were generated randomly with given state and action spaces. We then constructed the associated TA($L$) and computed its optimal policy using the three algorithms. In all experiments, NVI and R-NVI converge to the same value function as VI, leading to the same policy. It shows that NVI converges significantly faster than the other two algorithms. We also list the number of iterations needed for each algorithm to converge; for NVI and R-NVI, the numbers of inner iterations are counted. As we can see, the number of iterations of NVI is slightly larger than that of VI. Note that the states in ${\cal G}_{L}$ are updated every $L$ inner iterations. This means that NVI finds the optimal values for position states in ${\cal G}_{L}$ with much less updates than VI. The experiments verify our analysis under Theorem 14 that the fast convergence of root position states would speed up the convergence of other position states, illustrating why NVI is faster than VI. By contrast, the number of iterations of R-NVI is considerably larger than that of VI (the former is exactly $L$ times that of the latter in all experiments). In summary, NVI is more efficient than VI, while R-NVI is not. In the remaining part of this section, TA($L$) and TA($n,L$) are solved using NVI in all experiments.

We next evaluate the effectiveness of TA($L$) and TA($n,L$) for solving IOMDPs. The experiments were carried out in an MDP with $|{\cal S}|=9,|{\cal A}|=4$. The parameters of the underlying MDP are provided in Appendix F. As shown in Table 2, we consider 4 values of $\rho$. For each $\rho$, we solved the associated TA($L$) using NVI and obtained the TA($L$) policy to control the IOMDP with SIRP $\rho$. We then collected the total discounted reward over a horizon of $10^{5}$ time steps. The value of each group is the empirical average total discounted reward of $2\times 10^{4}$ independent runs. The results of TA($L+n$) and TA($n,L$) were obtained by the same procedure. In all experiments, the policies generated by TA($L+n$) and TA($n,L$) are identical; hence their values are equal. We also present the computation time of each method in the table, where the computation time of TA($n,L$) includes the time of solving TA($k,L$) for all $0\leq k\leq n$. The results show that TA($L$) with a small $L$ can not generate a satisfactory policy when $\rho$ is small, but we can obtain a better policy by increasing $L$. However, the computation time of TA($L$) increases explosively with $L$. Fortunately, TA($n,L$) can generate the same policy as TA($L+n$) using much less computation time.

We carried out more experiments for the last group ($\rho=0.5$) of Table 2 to show how $L$ and $n$ affect the performance of TA($L$) and TA($n,L$). Fig. 4(a) shows that the value of the TA($L$) policy increases with $L$ and eventually becomes stable after $L\geq 6$. This verifies our analysis in Section 5—the larger the value of $L$, the TA($L$) policy is optimal for more position states, hence the closer the TA($L$) policy is to the optimal policy. Given an initial position state in ${\cal G}_{0}$, the value difference between the TA($L$) policy and the optimal policy diminishes quickly as $L$ increases. The disadvantage of TA($L$) is also obvious: the computation time increases exponentially with $L$. In Fig. 4(b), we fix the value of $L$ and examine how the parameter $n$ affects the value and the computation time of the TA($n,L$) policy. Three different $L$ are compared. TA($n,L$) shows excellent performance in this experiment—it achieves the same value as TA($L+n$) with much less computation time. It is worth noting that the computation time of TA($n,L$) only increases linearly with $n$, showing good scalability of this method.

The above experiments show that, compared with TA($n,L$), TA($L$) is much more inefficient and unscalable. However, we can show that TA($L$) with small $L$ is good enough in some scenarios. The experiments were conducted in IOMDPs generated randomly as before. We set the SIRP to be $\rho=0.9$ and the initial position state belong to ${\cal G}_{0}$ in all these experiments. Table 3 shows the values of multiple TA($L$) policies, where the value of each group is obtained by the same procedure as in Table 2. In these experiments, the performance of TA(2) is (nearly) the same as that of TA(6), implying that TA(2) is already near-optimal. Considering that TA($L$) is easier to implement than TA($n,L$), it may be a good choice to use TA($L$) when a small $L$ is good enough.

8 Conclusion

This paper studied intermittently observable MDPs. We assumed that the transmissions of state information follow a Bernoulli lossy process. The problem of selecting actions for the MDP in the presence of state information losses has been investigated systematically. Two methods of formulating the IOMDP were used for different purposes. We first formulated the problem as a belief MDP to analyze the effect of state information losses. We proved that the expected total discounted reward is a continuous and increasing function of the state information reception probability. An upper bound for the performance regret caused by state information losses is derived. Then, we reformulated the IOMDP as a tree MDP to develop finite-state approximations, TA($L$) and TA($n,L$), for the problem. We showed analytically that TA($L$) could well approximate the optimal value function for part of the states, thus allowing it to generate a near-optimal policy. Inspired by the approximation error bound of TA($L$), we further developed TA($n,L$), which is more efficient than TA($L$) because it excludes the redundant states when computing policies. In addition, we also proposed a nested value iteration algorithm for TA($L$) and TA($n,L$). Convergence analysis is provided for interpreting the efficiency of the algorithm. Finally, we validated the effectiveness of the proposed methods by numerical results.

From the perspective of communication, the transmission model of state information depends on the environment and the communication protocol adopted by the system. The Bernoulli process considered in this paper is a proper transmission model if the system adopts a time-division multiple access (TDMA) protocol and operates in a stationary environment. In this case, the SIRP can be easily measured by counting the empirical success rate of a number of transmissions. A natural direction of extending the IOMDP framework studied in this paper is to generalize the transmission model. (1) If the communication environment is time-varying (e.g., the sensor or the controller moves significantly over time), then the SIRP may change dynamically over time. For problems of this kind, we may need to take the worst-case performance into account when making decisions. The structural results established in Section 3 may be exploited to analyze these problems. (2) If the system adopts other protocols, we may need a more complicated transmission model (e.g., a Markov model is usually used for transmissions over WiFi networks). These present interesting questions for future research.

Acknowledgments and Disclosure of Funding

This work was supported in part by the General Research Funds (Project No. 14200221) established under the University Grant Committee of the Hong Kong Special Administrative Region, China.

Appendix A

A.1 Proof of Theorem 2

In this part we prove Theorem 2 from Section 3. For notation simplicity, $T_{\rho}^{\pi}({\bf{x}},{\bf{y}})$ will stand for the probability of the belief state transitioning from ${\bf{x}}\in{\cal W}$ to ${\bf{y}}\in{\cal W}$, i.e.,

[TABLE]

**Proof **[Theorem 2] Consider $0<\rho\leq v\leq 1$. Suppose that $\pi_{\rho}\in\Pi$ is an optimal policy for ${\cal B}(\rho)$. Let $T_{\rho}^{{\pi_{\rho}}}$ denote the transition matrix of ${\cal B}(\rho)$ under policy $\pi_{\rho}$. Then

[TABLE]

Note that policy $\pi_{\rho}$ can also be applied to the belief MDP ${\cal B}(v)$. We have

[TABLE]

From (15) and (16),

[TABLE]

It follows that

[TABLE]

Note that ${\left({I-\beta T_{v}^{{\pi_{\rho}}}}\right)^{-1}}$ exists and is non-negative (i.e., all entries are non-negative) because $T_{v}^{{\pi_{\rho}}}$ is a stochastic matrix. Specifically,

[TABLE]

Let $\eta=\left({T_{v}^{{\pi_{\rho}}}-T_{\rho}^{{\pi_{\rho}}}}\right){\phi^{{\pi_{\rho}}}}[\rho]$. Just like ${\phi^{{\pi_{\rho}}}}(\cdot,\rho)$, $\eta$ can be viewed as a function on set ${\cal W}$. For an arbitrary $\mathbf{b}\in{\cal W}$, let $\tau=P_{{\pi_{\rho}}({\bf{b}})}^{\top}{\bf{b}}$. Then we can express $\eta$ in component notation as

[TABLE]

Note that $\tau=\sum\limits_{i\in{\cal S}}{\tau(i){{\bf{e}}_{i}}}$ and that ${\phi^{{\pi_{\rho}}}}({\bf{x}},\rho)=\phi({\bf{x}},\rho)$ is a convex function of ${\bf x}\in{\cal W}$. It follows that

[TABLE]

Therefore, $\eta({\bf{b}})\geq 0$ for any ${\bf{b}}\in{\cal W}$. The product of a non-negative matrix and a non-negative vector is clearly non-negative. That is,

[TABLE]

It follows immediately that $\phi[v]\geq{\phi^{{\pi_{\rho}}}}[v]\geq{\phi^{{\pi_{\rho}}}}[\rho]=\phi[\rho]$. This completes the proof.

A.2 Proof of Theorem 5

This part proves Theorem 5 from Section 3.

**Proof **[Theorem 5] Consider an arbitrary $\rho\in(0,1)$. Let $v\in[\rho,1]$ and $\pi_{v}$ denote an optimal policy for ${\cal B}(v)$. It follows from Theorem 2 that ${\phi^{{\pi_{v}}}}[v]=\phi[v]\geq\phi[\rho]\geq{\phi^{{\pi_{v}}}}[\rho]$. Therefore,

[TABLE]

where $||\cdot||_{\infty}$ denotes the max norm. Using a similar argument as in (18) yields

[TABLE]

Let $\eta=\left({T_{v}^{{\pi_{v}}}-T_{\rho}^{{\pi_{v}}}}\right)\phi[v]$. Then for any ${\bf b}\in{\cal W}$, let $\tau=P_{{\pi_{v}}({\bf{b}})}^{\top}{\bf{b}}$, we have

[TABLE]

Since the reward function is bounded, the value function $\phi(\cdot,v)$ is also bounded. The convexity of $\phi(\cdot,v)$ implies that the term within the square brackets is non-negative. As a result,

[TABLE]

is non-negative and bounded. It follows that

[TABLE]

where ${\bf e}$ is the vector with all elements being 1. Clearly, for any $\epsilon>0$,

[TABLE]

Using a similar argument could show that the above conclusion holds for $v\leq\rho$. This proves the continuity of $\phi({\bf b},\rho)$.

The bound for $\phi[1]-\phi[\rho]$ is obtained by substituting $v=1$ into (21) and (22). Using Lemma 4 yields the expression of $\eta$ presented in Theorem 5.

Appendix B

B.1 Proof of Lemma 6

**Proof ** For any $h\in{\cal H}$ and $a\in{\cal A}$, let ${\bf b}=g(h)$, $\tau_{a}=(h,a)$, and ${\bf b}_{a}=g(\tau_{a})$. Note that

[TABLE]

Then by definition, we have

[TABLE]

The Bellman equation for ${\cal C}(\rho)$ can be written as

[TABLE]

Note also that the Bellman equation for ${\cal B}(\rho)$ is

[TABLE]

Clearly, (25) and (26) are of the same form for any ${\bf b}=g(h)$. Since the Bellman equation has a unique solution, we conclude that $\phi({\bf b})=\varphi({h})$ if ${\bf b}=g(h)$. This proves statement 1. Statement 2 follows immediately: for any $h,h^{\prime}\in{\cal H}$ satisfying $g(h)=g(h^{\prime})$, we have $\varphi(h^{\prime})=\phi({\bf{b}})=\varphi(h)$. In addition, statement 1 implies that $\phi({{\bf{e}}_{i}})=\varphi(i)$ for all $i\in{\cal S}$ and $\phi({{\bf{b}}_{a}})=\varphi({\tau_{a}})$ for any $a\in{\cal A}$. Then statement 3 can be verified using (25) and (26).

B.2 Proof of Theorem 8

**Proof ** We first construct an auxiliary MDP, denoted by ${\cal C}^{\prime}(\rho)=({\cal H},{\cal A},\rho,M,\bar{R},\beta)$ to facilitate the analysis of the approximation error of TA($L$). In particular, the belief state space ${\cal H}$ , action space ${\cal A}$, SIRP $\rho$, reward function ${\bar{R}}$, and discount factor $\beta$ are identical to those of ${\cal C}(\rho)$. The transition kernel $M$ is defined as follows:

• (1)

For any $h\in{\cal G}_{n}$ with $n\neq L$, $M(h^{\prime}|h,a)=D(h^{\prime}|h,a)$, where $h^{\prime}\in{\cal H}$ and $a\in{\cal A}$;

• (2)

For any $h\in{\cal G}_{L}$, let $\tau_{a}=(h,a)$ and ${\bf b}_{a}=g(\tau_{a})$, where $a\in{\cal A}$. Then

[TABLE]

On the one hand, since ${\cal C}(\rho)$ and ${\cal C}^{\prime}(\rho)$ have the same state and action spaces, they share the same set of deterministic stationary policies, say $\Pi_{h}$. On the other hand, note that ${\cal H}={{\cal H}_{L}}\cup\left({\cup_{n=L+1}^{\infty}{{\cal G}_{n}}}\right)$ and that the probability of the position state transitioning from ${\cal H}_{L}$ to $\left({\cup_{n=L+1}^{\infty}{{\cal G}_{n}}}\right)$ is 0. If the initial position state is $h_{0}\in{\cal H}_{L}$, then ${\cal C}^{\prime}(\rho)$ reduces to ${\cal C}_{L}(\rho)$. Denote by $\psi^{\pi}(\cdot)$ the value function of ${\cal C}^{\prime}(\rho)$ under policy $\pi\in\Pi_{h}$ and $\psi(\cdot)$ the optimal value function of ${\cal C}^{\prime}(\rho)$. Then

[TABLE]

As we will show below, it is more convenient to compare two MDPs with the same state and action spaces. Hence ${\cal C}^{\prime}(\rho)$ will serve as a bridge that connects ${\cal C}(\rho)$ and ${\cal C}_{L}(\rho)$ so that we can characterize the approximation error by comparing $\varphi$ with $\psi$.

For any $\pi\in\Pi_{h}$, using the same argument as in (18) yields

[TABLE]

Let ${\xi^{\pi}}=\left({{D^{\pi}}-{M^{\pi}}}\right){\varphi^{\pi}}$ and $h^{\prime}=(h,\pi(h))$. Then ${\xi^{\pi}}:{\cal H}\to\mathbb{R}$ is given by

[TABLE]

The RHS of (27) can be viewed as the expected total discounted reward of a Markov reward process whose reward function is ${\xi^{\pi}}$ and the Markov chain is governed by the transition kernel $M^{\pi}$. Denote by $\{x_{t}:t=0,1,2,\cdots\}$ the Markov reward process. We then can write $\varphi^{\pi}-\psi^{\pi}$ in component form as

[TABLE]

Substituting (28) into (29) yields

[TABLE]

For any $h\in{\cal G}_{k}$, it can be proved that(see Appendix E for the proof)

[TABLE]

Note that the above quantity is independent of policy $\pi$. Since $\varphi^{\pi}\geq 0$ for all $\pi$ and $\rho$, we have

[TABLE]

The inequality follows from the fact that $h\in{\cal G}_{L}$ and $h^{\prime}\in{\cal G}_{L+1}$. Define

[TABLE]

Then ${\left\|{{\xi^{\pi}}}\right\|_{\infty}}\leq\beta(1-\rho){\delta_{L}}$ for all $\pi$. Therefore, for any $\pi\in\Pi_{h}$,

[TABLE]

Now, denote by $\pi$ and $\pi^{\prime}$ the optimal policies for ${\cal C}(\rho)$ and ${\cal C}^{\prime}(\rho)$, respectively. From (31), for any $h\in{\cal G}_{k},k=0,1,\cdots,L$,

[TABLE]

Putting together (32) and (33) yields

[TABLE]

Since $\psi(h)=\varphi_{L}(h)$ for all $h\in{\cal H}_{L}$, the desired result follows immediately.

Appendix C

In this Appendix we prove theoretical results from Section 5.

C.1 Proof of Lemma 12

**Proof ** Let $\mu_{L+n}$ denote the optimal policy for ${\cal C}_{L+n}(\rho)$. According to our construction, ${\cal C}^{n}_{L}(\rho)$ and ${\cal C}_{L+n}(\rho)$ having the same optimal action for every $h\in\cup_{i=0}^{n-1}{{\cal Z}_{i}}$ means that

[TABLE]

In addition, each $h\in\cup_{i=0}^{n-1}{{\cal Z}_{i}}$ in MDP ${\cal C}^{n}_{L}(\rho)$ has only one available action, i.e., $\mu_{L+n}(h)$.

The quantity $\varphi_{L+n}(h)$ is the expected total discounted reward generated by ${\cal C}_{L+n}(\rho)$ with policy $\mu_{L+n}$ and initial position state $h$. We thus can express $\varphi_{L+n}(h)$ as

[TABLE]

For ${\cal C}_{L+n}(\rho)$ with initial position state $h$ and being governed by policy $\mu_{L+n}$, ${{\cal H}_{L+n}}-{\cal H}_{L}^{n}$ is a set of redundant position states. Therefore, for any $h\in{\cal H}^{n}_{L}$ and $x\notin{\cal H}^{n}_{L}$,

[TABLE]

Denote by $\kappa$ a policy for ${\cal C}^{n}_{L}(\rho)$ with $\kappa(h)={\mu_{L+n}}(h)$ for any $h\in{\cal H}_{L}^{n}$. Then (35) and (36) implies that

[TABLE]

The second line of (37) is the expected total discounted reward generated by ${\cal C}^{n}_{L}(\rho)$ with policy $\kappa$ and initial position state $h$. It follows that ${\varphi_{L+n}}(h)\leq\varphi_{L}^{n}(h)$ for all $h\in{\cal H}^{n}_{L}$.

We next show ${\varphi_{L+n}}(h)\geq\varphi_{L}^{n}(h)$ for all $h\in{\cal H}^{n}_{L}$. Denote by $\Pi^{c}$ the set of all deterministic policies for ${\cal C}_{L+n}(\rho)$. Given the optimal policy $\mu_{L+n}$, define $\Pi_{n}^{c}=\{\kappa\in{\Pi^{c}}:\kappa(h)={\mu_{L+n}}(h),\forall h\in\cup_{i=0}^{n-1}{{\cal Z}_{i}}\}$. Then for any $h\in{\cal H}^{n}_{L}$,

[TABLE]

The desired result follows immediately.

C.2 Proof of Fact 1

Let

[TABLE]

Then $\mu_{L}(h)$ being an $\epsilon$-optimal action implies that

[TABLE]

Note that ${\mu_{L}}(h)=\arg{\max_{a\in{\cal A}}}\left\{{Q_{L}(h,a)}\right\}$. Let ${a_{h}}=\arg{\max_{a\in{\cal A}}}\left\{{Q(h,a)}\right\}$. Suppose that $\mu_{L}(h)\neq a_{h}$, we must have

[TABLE]

where the first and third inequalities follow from (38). The above implies that

[TABLE]

Therefore, $\mu_{L}(h)=a_{h}$ if $\epsilon\leq\epsilon_{h}/2$. This completes the proof.

C.3 Proof of Theorem 13

**Proof ** For $k\in\{0,1,\cdots,n\}$, define

[TABLE]

Then according to Theorem 8, the approximation error of $\varphi_{L+k}(h)$ for $h\in{\cal G}_{k+1}$ is bounded by

[TABLE]

where $k\in\{0,1,\cdots,n\}$. The penultimate inequality follows from the fact that $O_{L}(k)$ decreases with $k$:

[TABLE]

Denote by $\mu_{L+k}$ the optimal policy for ${\cal C}_{L+k}(\rho)$. Since the error bound increases monotonically with the layer index, (39) also holds for any $h\in{\cal G}_{i},i=0,1,\cdots,k$. Then according to Fact 1, we have

• (a)

$\mu_{L+k}(h)$ is an optimal action for $h\in\cup_{i=0}^{k}{{\cal G}_{i}}$ in ${\cal C}(\rho)$, $k\in\{0,1,\cdots,n\}$.

• (b)

${\mu_{L+k}}(h)={\mu_{L+k+1}}(h)$ for $h\in\cup_{i=0}^{k}{{\cal G}_{i}}$, $k\in\{0,1,\cdots,n-1\}$.

• (c)

$\mu_{L}^{k}(h)=\mu_{L}^{k+1}(h)$ for $h\in\cup_{i=0}^{k}{{\cal Z}_{i}}$, $k\in\{0,1,\cdots,n-1\}$.

Note that (b) follows from (a); (c) was discussed when defining TA($L$). Clearly, statement (1) of the theorem is a special case of (a).

We next prove statement (2) using (a), (b), and (c). Note that ${\mu_{L}}=\mu_{L}^{0}(h)$. It follows that $\mu_{L}^{1}(h)=\mu_{L}^{0}(h)={\mu_{L}}(h)={\mu_{L+1}}(h)$ for $h\in{{\cal G}_{0}}={{\cal Z}_{0}}$; that is, ${\cal C}^{1}_{L}(\rho)$ and ${\cal C}_{L+1}(\rho)$ have the same optimal action for every $h\in{\cal Z}_{0}$. Then according to Lemma 12, $\varphi_{L}^{1}(h)={\varphi_{L+1}}(h)$ for all $h\in{\cal H}^{1}_{L}$. This fact, together with (39) (the case of $k=1$), implies that $\mu^{1}_{L}(h)$ is an optimal action for $h\in{{\cal Z}_{0}}\cup{{\cal Z}_{1}}$ in ${\cal C}(\rho)$. That is, ${\mu_{L+1}}(h)=\mu_{L}^{1}(h)$ for $h\in{{\cal Z}_{0}}\cup{{\cal Z}_{1}}$. Repeating the above argument until the case of $k=n$ yields statement (2).

Appendix D

This Appendix proves Theorem 14 from Section 6.

**Proof ** For any bounded real-valued function $\psi:{\cal H}_{L}\to\mathbb{R}$, define the operator $F_{k}$ as

[TABLE]

where $k\in\{1,2,\cdots,L\}$. That is, $F_{k}$ is the operator corresponding to the $(L-k+1)$-th inner iteration within an outer iteration. It is well-known that $F_{L}$ is a contraction mapping and

[TABLE]

where $a\in{\cal A}$ is the greedy action with respect to $\psi^{n}$. For $h\in{\cal G}_{l}$ and $v=\min\{l+1,L\}$, substituting $D_{L}(h^{\prime}|h,a)$ into (40) yields

[TABLE]

It can be proved by a similar argument that the above inequality holds for any $F_{k}$ and $h\in{\cal H}_{k}$.

Let $n=kL+i$ counts the overall number of inner iterations, then $\psi^{n}$ is the result of the $i$-th inner iteration of the $k$-th outer iteration. We have

[TABLE]

Let $m=kL+1$ for an arbitrary $k$. Then

[TABLE]

According to (41), for $h\in{\cal G}_{l}$,

[TABLE]

For the first term of (42),

[TABLE]

Using the argument repeatedly yields

[TABLE]

For the second term of (42), ${F_{k}}{\psi^{n}}(h)={\psi^{n-1}}(h)$ if $h\notin{\cal H}_{k}$. We thus have

[TABLE]

Statement (2) can be obtained from (42)-(44): for $h\in{\cal G}_{l}$,

[TABLE]

The convergence property follows immediately.

Appendix E

This part proves eq. (30) from the proof of Theorem 8 (Appendix B.2). Given that $h\in{\cal G}_{k}$. For $n=0,1,2,\cdots,L$, define

[TABLE]

Since $\{x_{t}:t=0,1,2,\cdots\}$ is governed by the transition kernel $M^{\pi}$, we have

[TABLE]

From (47) and (48), for $n\in\{1,2,\cdots,L-1\}$ and $t\geq n$,

[TABLE]

It follows that, for $t\geq L$,

[TABLE]

Then we can derive

[TABLE]

Given that $x_{0}=h\in{\cal G}_{k}$, it is easy to verify that

[TABLE]

Since $p_{L}(L)=(1-\rho)^{L}$, the first term in (50) is 0. It follows from (50) and (51) that

[TABLE]

This completes the proof.

Appendix F

This Appendix provides an introduction to the experimental MDP in Section 7 (Table 2 and Fig. 4).

Consider a system with a car moving in the map shown in Fig. 5. The car has four actions at any position: move left, move right, move up, and move down. The aim is to control the car to keep moving along a clockwise direction in the white region ($1\to 2\to\cdots\to 8\to 1\to\cdots$). The task is terminated once the car enters the gray region (including the gray region in the central part and the gray region in the marginal part).

Given a positive integer $n$, we use $[n]=\{1,2,\cdots,n\}$ to denote the set of integers between 1 and $n$. Mathematically, the MDP is formulated as follows:

• •

State space ${\cal S}=[9]$, where states 1-8 correspond to positions 1-8 in the white region and state 9 corresponds to the gray region (terminated state).

• •

Action space ${\cal A}=[4]$: action 1, move left; action 2, move down; action 3, move right; action 4, move up.

• •

Transition kernel. For any state $s\in[8]$, denote by $m_{s}\in{\cal A}$ the clockwise action of state $s$ (that is, move left in states 1 and 2, move down in states 3 and 4, move right in states 5 and 6, move up in states 7 and 8) and $m^{-}_{s}\in{\cal A}$ the anti-clockwise action. Denote by $s_{t}$ and $a_{t}$ the state and action at time $t$, respectively. The transition kernel is defined as follows: (1) for $s_{t}=9$, $\Pr({s_{t+1}}=j|{s_{t}}=9,{a_{t}})=1$ if $j=9$ and $\Pr({s_{t+1}}=j|{s_{t}}=9,{a_{t}})=0$ otherwise; (2) for $s_{t}\in[8]$,

[TABLE]

[TABLE]

[TABLE]

• •

Reward function: $r(s,a)=20$ if $s\in[8],a=m_{s}$ and $r(s,a)=0$ otherwise.

• •

Discount factor: $\beta=0.95$.