Finite-Memory Strategies in POMDPs with Long-Run Average Objectives

TL;DR

This paper demonstrates that in POMDPs with long-run average objectives, decision makers can use finite-memory strategies to achieve near-optimal results, enabling recursive approximation of the long-term value.

Contribution

It proves the existence of finite-memory approximately optimal strategies in POMDPs with long-run average objectives, a significant theoretical advancement.

Findings

Finite-memory strategies are approximately optimal in POMDPs with long-run average objectives.

The long-run value approximation is recursively enumerable.

The value function exhibits a weak continuity property with respect to transition functions.

Abstract

Partially observable Markov decision processes (POMDPs) are standard models for dynamic systems with probabilistic and nondeterministic behaviour in uncertain environments. We prove that in POMDPs with long-run average objective, the decision maker has approximately optimal strategies with finite memory. This implies notably that approximating the long-run value is recursively enumerable, as well as a weak continuity property of the value with respect to the transition function.

Equations

$${\gamma }_{\infty }^{p_1}(\sigma ):={\mathbb{E}}_{\sigma }^{p_1}\left(\underset{n\to +\infty }{\mathrm{liminf}}\frac{1}{n}\sum _{m=1}^n{G}_m\right),$$

$$v_{\infty }(p_1):=\underset{\sigma \in \Sigma }{\sup }{\gamma }_{\infty }^{p_1}(\sigma ).$$

$$v_{\infty }(p_1)$$

$$=\underset{\lambda \to 0}{\lim }\underset{\sigma \in \Sigma }{\sup }{\mathbb{E}}_{\sigma }^{p_1}\left(\sum _{m\ge 1}\lambda (1-\lambda {)}^{m-1}{G}_m\right)$$

$$=\underset{\sigma \in \Sigma }{\sup }\underset{n\to +\infty }{\mathrm{liminf}}{\mathbb{E}}_{\sigma }^{p_1}\left(\frac{1}{n}\sum _{m=1}^n{G}_m\right).$$

$${\gamma }_{\infty }^{p_1}(\sigma )\ge v_{\infty }(p_1)-\epsilon .$$

$$v_{\infty }(p_1)=\underset{\sigma \in \Sigma }{\sup }{\mathbb{E}}_{\sigma }^{p_1}\left(\underset{n\to \infty }{\mathrm{liminf}}\frac{1}{n}\sum _{m=1}^n{G}_m\right).$$

$$d(q,q^{\prime })=\underset{\begin{array}{c}k\in \mathcal{K},a\in \mathcal{A}\\ k^{\prime }\in \mathcal{K},s\in \mathcal{S}\end{array}}{\max }\left\{\frac{q(k,a)(k^{\prime },s)}{q^{\prime }(k,a)(k^{\prime },s)},\frac{q^{\prime }(k,a)(k^{\prime },s)}{q(k,a)(k^{\prime },s)}\right\}-1.$$

$$d(q,q^{\prime })\le \delta \Rightarrow {\gamma }_{\infty }^{\prime p_1}(\sigma )\ge {\gamma }_{\infty }^{p_1}(\sigma )-\eta ,$$

$${\gamma }_{\infty }^{\prime k}(\sigma )\ge {\gamma }_{\infty }^k(\sigma )-4|\mathcal{K}||\mathcal{S}||\mathcal{M}|d(q,q^{\prime }).$$

$${\gamma }_{\infty }^{\prime p_1}(\sigma )\ge {\gamma }_{\infty }^{p_1}(\sigma )-4|\mathcal{K}||\mathcal{S}||\mathcal{M}|d(q,q^{\prime }).$$

$$v_{\infty }^{\prime }(p_1)\ge {\gamma }_{\infty }^{\prime p_1}({\sigma }_{\epsilon })\ge v_{\infty }(p_1)-2\epsilon .$$

$$\underset{q^{\prime }\to q}{\mathrm{liminf}}{v}_{\infty }^{\prime }(p_1)\ge v_{\infty }(p_1),$$

$$\sigma =(wait{)}^{2^{0^2}}(change)(wait{)}^{2^{1^2}}\cdots (change)(wait{)}^{2^{N^2}}\cdots$$

$$1=\underset{\epsilon \to 0}{\mathrm{liminf}}{v}_{\infty }(p_1\mid \epsilon )>v_{\infty }(p_1\mid \epsilon =0)=\frac{1}{2}.$$

$$H_m:=((A_1,S_1),(A_2,S_2),\dots ,(A_{m-1},S_{m-1})).$$

$$P_m(k):={\mathbb{P}}_{\sigma }^{p_1}(K_m=k\mid H_m).$$

$${\mathbb{P}}_{{\sigma }^{\epsilon }}^{p_1}({\Vert P_{m_{\epsilon }}-P^{\ast }\Vert }_1\le \epsilon )\ge 1-\epsilon .$$

$$\left(\frac{1}{n}\sum _{m=1}^n{G}_m\right)\underset{n\to \infty }{\to }{\gamma }_{\infty }^k(\sigma ){\mathbb{P}}_{\sigma }^k-a.s.$$

$${\gamma }_{\infty }^{p^{\ast }}({\sigma }^{\prime })\ge {\gamma }_{\infty }^{p^{\ast }}(\sigma )-\epsilon .$$

$${\gamma }_{\infty }^{p_1}({\sigma }^0)$$

$$\ge {\mathbb{E}}_{{\sigma }^{\epsilon }}^{p_1}\left({\gamma }_{\infty }^{P^{\ast }}({\sigma }^{\prime })\right)-2\epsilon$$

$$\ge {\mathbb{E}}_{{\sigma }^{\epsilon }}^{p_1}\left({\gamma }_{\infty }^{P^{\ast }}(\sigma )\right)-3\epsilon$$

$$={\mathbb{E}}_{{\sigma }^{\epsilon }}^{p_1}\left(v_{\infty }(P^{\ast })\right)-3\epsilon$$

$$\ge v_{\infty }(p_1)-4\epsilon$$

$${\mathcal{B}}_m^i(h_m):=\bigcup _{k_1\in {\mathcal{K}}_i}\{k\in \mathcal{K}:{\mathbb{P}}_{\sigma }^{k_1}(H_m=h_m)>0,{\mathbb{P}}_{\sigma }^{k_1}(K_m=k|H_m=h_m)>0\}.$$

$$B_m^i:={\mathcal{B}}_m^i(H_m),$$

$$\mathrm{supp}(P_m)=\bigcup _{i\in [1..I]}{B}_m^i.$$

$$\sigma [h_m](h_{m^{\prime }}):=\sigma (h_m,h_{m^{\prime }}).$$

$$\forall k\in B_m^i{\gamma }_{\infty }^k({\sigma }_m)={\gamma }_{\infty }^i$$

$$\left(\frac{1}{n}\sum _{m^{\prime }=m}^{m+n-1}{G}_{m^{\prime }}\right)\underset{n\to \infty }{\to }{\gamma }_{\infty }^{k^{\prime }}(\sigma )={\gamma }_{\infty }^i{\mathbb{P}}_{\sigma }^{k^{\prime }}-a.s..$$

$$\left(\frac{1}{n}\sum _{m^{\prime }=1}^n{G}_{m^{\prime }}\right)\underset{n\to \infty }{\to }{\gamma }_{\infty }^{k^{\prime }}(\sigma )={\gamma }_{\infty }^i{\mathbb{P}}_{{\sigma }_m}^k-a.s.,$$

Full text

1 Introduction

In a Partially Observable Markov Decision Process (POMDP), at each stage, the decision-maker chooses an action that determines, together with the current state, a stage reward and the distribution over the next state. The state dynamic is imperfectly observed by the decision-maker, who receives a stage signal on the current state before playing. Thus, POMDPs generalize the Markov Decision Process (MDP) model of Bellman [3].

POMDPs are widely used in prominent applications such as in computational biology [10], software verification [8], reinforcement learning [16], to name a few. Even special cases of POMDPs, namely, probabilistic automata or blind MDPs, where there is only one signal, is also a standard model in several applications [20, 19, 7].

In many of these applications, the duration of the problem is huge. Thus, considerable attention has been devoted to the study of POMDPs with long duration. A standard way is to consider the long-run objective criterion, where the total reward is the expectation of the inferior limit average reward (see [1] for a survey). The value for this problem is known to coincide with several classical definitions of long-run values (asymptotic value, uniform value, general uniform value, long-run average value, uncertain-duration process value [23, 21, 22, 18, 27]) and has been characterized in [22]. In this paper, we will simply name this common object value. Thus, strong results are available concerning the existence and characterization of the value.

This is in sharp contrast with the study of long-run optimal strategies. Indeed, before our work, little was known about the sophistication of strategies that approximate the value. It has been shown that:(i) stationary strategies approximate the value in MDPs [4]; and (ii) belief-stationary strategies approximate the value in blind MDPs [23] and POMDPs with an ergodic structure [6].

Our main contributions are:

• •

Strategy complexity. We show that for every POMDP with long-run average objectives, for every $\varepsilon>0$, there is a finite-memory strategy (i.e. generated by a finite state automaton) that achieves expected reward within $\varepsilon$ of the optimal value. In the case of blind MDP finite memory is equivalent to finite recall (i.e. decisions are defined using only the last actions), but finite recall cannot achieve $\varepsilon$-approximations in general POMDPs.

• •

Computational complexity. An important consequence of our above result is that the decision version of the approximation problem for POMDPs with long-run average objectives (see Definition 3.1) is recursively enumerable (r.e.) but not decidable. Our results on strategy complexity imply the recursively enumerable upper bound and the lower bound is a consequence of [17].

• •

Value property. The long-run reward of a finite-memory strategy is robust upon small perturbations of the transition function, where the notion of perturbation over the transition function is defined as in Solan [25] and Solan and Vieille [26]. This implies lower semi-continuity of the value function upon such small perturbations. This result is tight in the sense that there is an example with a discontinuous value function (see Example 4.4).

A natural question would be to ask for an upper bound on the size of the memory needed to generate $\varepsilon$-optimal strategies, in terms of the data of the POMDP. In fact, a previous undecidability result [17] shows that such an upper bound can not exist (see Subsection 3.1). Thus, the existence of $\varepsilon$-optimal strategies with finite memory is, in some sense, the best possible result one can have in terms of strategy complexity.

2 Model and statement of results

2.1 Model

Throughout the paper we mostly use the following notation: (i) sets are denoted by calligraphic letters, e.g. $\mathcal{A},\mathcal{H},\mathcal{K},\mathcal{S}$; (ii) elements of these sets are denoted by lowercase letters, e.g. $a,h,k,s$; and (iii) random elements with values in these sets are denoted by uppercase letters, e.g. $A,H,K,S$. For a set $\mathcal{C}$, denote $\Delta(\mathcal{C})$ the set of probability measure distributions over $\mathcal{C}$, and $\delta_{c}$ the Dirac measure at some element $c\in\mathcal{C}$. We will slightly abuse notation by not making a distinction between a probability measure (which can be evaluated on events) and its corresponding probability density (which can be evaluated on elements).

Consider a POMDP $\Gamma=(\mathcal{K},\mathcal{A},\mathcal{S},q,g)$, with finite state space $\mathcal{K}$, finite action set $\mathcal{A}$, finite signal set $\mathcal{S}$, transition function $q\colon\mathcal{K}\times\mathcal{A}\rightarrow\Delta(\mathcal{K}\times\mathcal{S})$ and reward function $g\colon\mathcal{K}\times\mathcal{A}\rightarrow[0,1]$.

Given $p_{1}\in\Delta(\mathcal{K})$, called initial belief, the POMDP starting from $p_{1}$ is denoted by $\Gamma(p_{1})$ and proceeds as follows:

• •

An initial state $K_{1}$ is drawn from $p_{1}$. The decision-maker knows $p_{1}$ but does not know $K_{1}$.

• •

At each stage $m\geq 1$, the decision-maker takes some action $A_{m}\in\mathcal{A}$. This action determines a stage reward $G_{m}\coloneqq g(K_{m},A_{m})$, where $K_{m}$ is the (random) state at stage $m$. Then, the pair $(K_{m+1},S_{m})$ is drawn from $q(K_{m},A_{m})$. The next state is $K_{m+1}$ and the decision-maker is informed of the signal $S_{m}$, but neither of the reward $G_{m}$ nor of the state $K_{m+1}$.

At stage $m$, the decision-maker remembers all the past actions and signals, which is called history before stage $m$. Let $\mathcal{H}_{m}\coloneqq(\mathcal{A}\times\mathcal{S})^{m-1}$ be the set of histories before stage $m$, with the convenient notation $(\mathcal{A}\times\mathcal{S})^{0}\coloneqq\{\emptyset\}$. A strategy is a mapping $\sigma\colon\cup_{m\geq 1}\mathcal{H}_{m}\to\mathcal{A}$. The set of strategies is denoted by $\Sigma$. The randomness introduced by the transition function, $q\colon\mathcal{K}\times\mathcal{A}\rightarrow\Delta(\mathcal{K}\times\mathcal{S})$, suggests that a history $h_{m}\in\mathcal{H}_{m}$ can occur under many sequences of states $(k_{1},k_{2},\ldots,k_{m-1})$. The infinite sequence $(k_{1},a_{1},s_{1},k_{2},a_{2},s_{2},\ldots)$ is called a play, and the set of all plays is denoted by $\Omega$.

For $p_{1}\in\Delta(\mathcal{K})$ and $\sigma\in\Sigma$, define $\mathbb{P}^{p_{1}}_{\sigma}$ the law induced by $\sigma$ and the initial belief $p_{1}$ on the set of plays of the game $\Omega=(\mathcal{K}\times\mathcal{A}\times\mathcal{S})^{\mathbb{N}}$, and $\mathbb{E}^{p_{1}}_{\sigma}$ the expectation with respect to this law. For simplicity, identify $\mathcal{K}$ with the set of extremal points of $\Delta(\mathcal{K})$.

Let

[TABLE]

and

[TABLE]

The term $\gamma_{\infty}^{p_{1}}(\sigma)$ is the long-term reward given by strategy $\sigma$ and $v_{\infty}(p_{1})$ is the optimal long-term reward, called value, defined as the supremum long-term reward over all strategies.

Remark 2.1*.*

It has been shown that $v_{\infty}$ coincides with the limit of the value of the $n$-stage problem and $\lambda$-discounted problem, as well as the uniform value and weighted uniform value (see [23, 21, 22, 27]). In particular, we have:

[TABLE]

Remark 2.2*.*

In the literature, the concept of strategy that we defined is often called pure strategy, by contrast with behavior strategies that use randomness by allowing strategies of the form $\sigma\colon\cup_{m\geq 1}\mathcal{H}_{m}\to\Delta(\mathcal{A})$. By Kuhn’s theorem, enlarging the set of pure strategies to behaviour strategies does not change $v_{\infty}$ (see [27, 11]), and thus does not change our results.

Definition 2.3** (Blind MDP).**

A POMDP is called blind MDP if the signal set is a singleton.

Note that in a blind MDP, signals do not convey any relevant information. Therefore, a strategy is simply an infinite sequence of actions $(a_{1},a_{2},\dots)\in\mathcal{A}^{\mathbb{N}}$.

2.2 Contribution

We start by defining several classes of strategies. Recall that $\Gamma(p_{1})$ is the POMDP $\Gamma$ starting from $p_{1}$, which is known to the player.

Definition 2.4** ($\varepsilon$-optimal strategy).**

Let $p_{1}\in\Delta(\mathcal{K})$ and $\varepsilon>0$. A strategy $\sigma\in\Sigma$ is $\varepsilon$-optimal in $\Gamma(p_{1})$ if

[TABLE]

Definition 2.5** (finite-memory strategy).**

A strategy $\sigma$ is said to have finite memory if it can be modeled by a finite-state transducer. Formally, $\sigma=(\sigma_{u},\sigma_{a},\mathcal{M},m_{0})$, where $\mathcal{M}$ is a finite set of memory states, $m_{0}$ is the initial memory state, $\sigma_{a}:\mathcal{M}\to\mathcal{A}$ is the action selection function and $\sigma_{u}\colon\mathcal{M}\times\mathcal{A}\times\mathcal{S}\to\mathcal{M}$ is the memory update function.

Definition 2.6** (Finite-recall strategy).**

A strategy $\sigma$ is said to have finite recall if there exists a constant $M>0$ such that for all $h_{M}\in\mathcal{H}_{M}$, and for all $m>M$ and $h_{m-M}\in\mathcal{H}_{m-M}$, we have that $\sigma(h_{m-M},h_{M})$ does not depend on $h_{m-M}$.

Remark 2.7*.*

For blind MDPs, finite-recall and finite-memory strategies coincide with the set of eventually periodic strategies: a strategy $\sigma=(a_{1},a_{2},\ldots)$ is eventually periodic if there exists $T\geq 1$ and $N\geq 1$ such that for all $m\geq N$, $a_{m+T}=a_{m}$. This property does not extend to general POMDPs (see Proposition 2.12): any finite-recall strategy has finite-memory, but the inverse is not true.

Remark 2.8*.*

Finite-memory strategies and finite-recall strategies have been investigated in the Shapley zero-sum stochastic game model [24]. In this framework, none of these strategies is enough to approximate the value, and a long-standing open problem is whether finite-memory strategies with a clock are good enough (see [14, 13] for more details on this topic).

Our main result is the following theorem.

Theorem 2.9**.**

For every POMDP $\Gamma$, initial belief $p_{1}$ and $\varepsilon>0$, there exists an $\varepsilon$-optimal finite-memory strategy in $\Gamma(p_{1})$.

Remark 2.10*.*

A previous complexity result [17] shows that the size of the memory can not be bounded from above in terms of the data of the POMDP (see Subsection 3.1).

Corollary 2.11**.**

For every blind MDP $\Gamma$, initial belief $p_{1}$ and $\varepsilon>0$, there exists an $\varepsilon$-optimal finite-memory strategy in $\Gamma(p_{1})$, and thus the strategy is eventually periodic and has finite recall.

Lastly, finite-recall is not enough to ensure $\varepsilon$-optimality in general POMDPs.

Proposition 2.12**.**

There exists a POMDP, and $\varepsilon>0$, with no $\varepsilon$-optimal finite-recall strategy.

The rest of the paper is organized as follows. Section 3 explains the consequences of our result in terms of complexity and model robustness. Section 4 introduces examples used to prove negative results and to illustrate our techniques. Section 5 introduces two key lemmata, and shows that they imply Theorem 2.9. Section 6 proves one of the two lemmata and develops what we call super-support based strategies in details. Missing proofs are in the appendices.

3 Consequences of the results

3.1 Complexity

Decidability. A decision problem consists in deciding between two options given an input (accepting or rejecting) and its complexity is characterized by Turing machines. A Turing machine takes an input and, if it halts, it either accepts or rejects it. If it halts for all possible inputs in a finite number of steps, then the Turing machine is considered an algorithm. An algorithm solves a decision problem if it takes the correct decision for all inputs. The class of decision problems that are solvable by an algorithm is called decidable. Two natural generalizations of decidable problems are: recursively enumerable (r.e.) and co-recursively enumerable (co-r.e.). The decision problems in r.e. (resp., co-r.e.) are those for which there is a Turing machine that accepts (resp., rejects) every input that should be accepted (resp., rejected) according to the problem, but, on other inputs, it needs not to halt.

Notice that the class of decidable problems is the intersection of r.e. and co-r.e. In this work, the algorithmic problem of interest is the following.

Definition 3.1** (Decision version of approximating the value).**

Let $p_{1}\in\Delta(\mathcal{K})$. Given $x\in[0,1]$, $\varepsilon>0$ such that $v_{\infty}(p_{1})>x+\varepsilon$ or $v_{\infty}(p_{1})<x-\varepsilon$, the problem consists in deciding which one is the case: to accept means to prove that $v_{\infty}(p_{1})>x+\varepsilon$ holds, while to reject means to prove the opposite.

Previous results and implication of our result. It is known that the decision version of the approximation problem is not decidable [17] (even for blind MDPs). However, the complexity characterization has been open. Thanks to Theorem 2.9, we can design a Turing machine that accepts every input that should be accepted for this problem.

Consider playing a finite-memory strategy $\sigma$. Then, the dynamics of the game can be described by a finite Markov chain. Therefore, the reward obtained by playing $\sigma$ (i.e. $\gamma^{p_{1}}_{\infty}(\sigma)$) can be deduced from its stationary measure, which can be computed in polynomial time by solving a linear programming problem [12, Section 2.9, page 70]. Our protocol checks the reward given by every finite-memory strategy to approximate the value of the game $v_{\infty}(p_{1})$. By Theorem 2.9, if $v_{\infty}(p_{1})>x+\varepsilon$ holds, a finite-memory strategy that achieves a reward strictly greater than $(x+\varepsilon)$ will be eventually found and our protocol will accept the input. On the other hand, if $v_{\infty}(p_{1})<x-\varepsilon$, the protocol will never find out that this is the case because there are infinitely many finite-memory strategies, so it will not halt. Thus, our result establishes that the approximation version of the problem is in r.e., and the previously known results imply that the problem is not decidable. Formally, we have the following result.

Corollary 3.2**.**

The decision version of approximating the value is r.e. but not decidable.

Remark 3.3*.*

The former paragraph shows that no upper bound on the size of the memory used by $\varepsilon$-optimal strategies can be proved. Indeed, if such a bound existed, one could modify the previous algorithm in the following way: reject the input if every finite-memory strategy of size lower than the bound has been enumerated. This would imply that the decision version of approximating the value is decidable, which is a contradiction.

3.2 Objective comparison

In this section, we contrast our results with other natural objectives.

Recall that the value of $\Gamma(p_{1})$ is defined as

[TABLE]

We say this is a liminf-average objective. Consider replacing $\liminf_{n\rightarrow\infty}\frac{1}{n}\sum_{m=1}^{n}G_{m}$ by: (i) $\limsup_{n\rightarrow\infty}\frac{1}{n}\sum_{m=1}^{n}G_{m}$, which we call limsup-average objective; (ii) $\limsup_{n\rightarrow\infty}G_{n}$, which we call limsup objective.

Proposition 3.4**.**

For both limsup-average and limsup objective, there exists a POMDP, and $\varepsilon>0$, with no $\varepsilon$-optimal finite-memory strategy.

This negative result, proved in Section 4.1.2, does not imply any computational complexity characterization for the limsup-average or limsup objective, and whether the approximation of the value problem for limsup-average objectives is recursively enumerable remains open. However, it shows that any approach based on finite-memory strategies cannot establish recursively enumerable bounds for the approximation problem.

Let us focus on the limsup objective. Limsup objective is arguably simpler than the liminf-average objective and, to formalize this statement, we can compare the complexity of the objects themselves irrespective of any particular context or model (such as POMDPs). The Borel hierarchy describes the complexity of an objective by the number of quantifier alternations needed to describe it. Its construction is similar to that of the Borel $\sigma$-algebra, or $\sigma$-field, and is defined as follows.

Definition 3.5** (Borel hierarchy).**

Consider $h_{m}\in\mathcal{H}_{m}=(\mathcal{A}\times\mathcal{S})^{m-1}$ a finite history of the game. The cylinder set generated by $h_{m}$ is given by $\{h_{m}\}\times(\mathcal{A}\times\mathcal{S})^{\mathbb{N}}$. Finite intersection, unions and complements of the cylinder sets generated by finite histories form the first level in the hierarchy. Countable unions of the first level form $\Sigma_{1}$ and countable intersections form $\Pi_{1}$. The next level is always obtained from the previous one: countable unions of $\Pi_{i}$ give $\Sigma_{i+1}$ and countable intersections of $\Sigma_{i}$ give $\Pi_{i+1}$. The nested sequence of family of problems $\{\Sigma_{i}\cup\Pi_{i}\}_{i\geq 1}$ is called Borel hierarchy.

For example, limsup objective can be described as countable intersection of countable unions of rewards: given a family of sets $(\mathcal{C}_{n})_{n\geq 1}$, $\limsup_{n\rightarrow\infty}\mathcal{C}_{n}=\cap_{n\geq 1}\cup_{m\geq n}\mathcal{C}_{m}$. The formal result is the following (see [9]).

Proposition 3.6**.**

The limsup objective is $\Pi_{2}$-complete, i.e. complete for the second level of the Borel hierarchy, whereas the liminf-average objective is $\Pi_{3}$-complete, i.e. complete for the third level of the Borel hierarchy.

While the notion of Borel hierarchy characterizes the topological complexity for objectives, a similar notion of Arithmetic hierarchy characterizes the computational complexity for decision problems.

Definition 3.7** (Arithmetic hierarchy).**

Denote $\Sigma_{0}^{1}$ the class of r.e. problems and $\Pi_{0}^{1}$ the co-r.e. problems. For $i>1$, define $\Sigma_{0}^{i}$ as the class of problems solved by Turing machines with access to oracles for $\Pi_{0}^{i-1}$ and $\Pi_{0}^{i}$ is similarly defined with oracles for $\Sigma_{0}^{i-1}$. The nested sequence of family of problems $\{\Sigma_{0}^{i}\cup\Pi_{0}^{i}\}_{i\geq 1}$ is called Arithmetic hierarchy.

By Corollary 3.2, we have that POMDPs with a liminf-average objective is in $\Sigma_{0}^{1}\setminus(\Sigma_{0}^{1}\cap\Pi_{0}^{1})$. On the other hand, it was shown in [2, 5] that POMDPs with limsup objective with boolean rewards is $\Sigma_{0}^{2}$-complete.

We conclude this section with a summary chart contrasting liminf-average and limsup objectives. The surprising result is the complexity switch: limsup objective has lower Borel hierarchy complexity but higher Arithmetic hierarchy complexity in the context of POMDPs.

3.3 Robust $\varepsilon$-optimal strategies

Consider a POMDP $\Gamma=(\mathcal{K},\mathcal{A},\mathcal{S},q,g)$. It is well known that the value function is continuous with respect to perturbations of the reward function $g$ and the initial belief $p_{1}$. Now, we show a robustness result concerning the transition function $q$.

In applications, just as in any stochastic model, the structure of the model is decided first, and then the specific probabilities are either estimated or fixed. The values of transition probabilities are approximations: an $\varepsilon$-perturbation of these probabilities are not expected to have an impact on the modelling. In our setting, the transitions are encoded in the function $q\colon\mathcal{K}\times\mathcal{A}\to\Delta(\mathcal{K}\times\mathcal{S})$ and we would expect some robustness against perturbations of the values it takes.

The notion of perturbation over $q$ is measured as in Solan [25] and Solan and Vieille [26], where perturbations in each transition probability are measured as relative differences, not additive differences. Formally, define the semimetric

[TABLE]

Under this notion, and taking $q$ and $q^{\prime}$ close to each other, we can prove the existence of strategies which are approximately optimal for the POMDP corresponding to $q$ and perform almost as well when they are applied to the POMDP corresponding to $q^{\prime}$. To formally state this notion of robustness, let us give the following definition.

Definition 3.8** (Robust strategies).**

Given a POMDP $\Gamma$ with transition function $q$, an initial belief $p_{1}\in\Delta(\mathcal{K})$, we say that $\sigma$ is a robust strategy for $\Gamma(p_{1})$ if the following condition holds: $\forall\eta>0$ $\exists\delta>0$ such that

[TABLE]

where $\gamma_{\infty}$ is the long-term reward in $\Gamma$ and $\gamma^{\prime}_{\infty}$ is the long-term reward in $\Gamma^{\prime}=(\mathcal{K},\mathcal{A},\mathcal{S},g,q^{\prime})$.

Lemma 3.9**.**

Any finite-memory strategy is robust. Thus, in any POMDP and for any $\varepsilon>0$, there exists a robust $\varepsilon$-optimal finite-memory strategy.

Proof.

Let $\Gamma=(\mathcal{K},\mathcal{A},\mathcal{S},g,q)$ be a POMDP. Consider $\sigma=(\sigma_{u},\sigma_{a},\mathcal{M},m_{0})$ a finite-memory strategy for $\Gamma(p_{1})$. Playing $\sigma$ from $p_{1}$ induces a Markov Chain $(Y_{n})_{n\geq 1}$ on $\mathcal{K}\times\mathcal{S}\times\mathcal{M}$. Define $\tilde{g}\colon\mathcal{K}\times\mathcal{M}\to[0,1]$ by $\tilde{g}(k,m)\coloneqq g(k,\sigma_{a}(m))$.

Now, consider the 0-Player stochastic game with reward $\tilde{g}$ and transitions given by the kernel of the Markov Chain $(Y_{n})_{n\geq 1}$. Let $s_{0}\in\mathcal{S}$ be any signal. By definition, for any $k\in\mathcal{K}$, the value of this stochastic game starting from $(k,s_{0},m_{0})$ coincides with $\gamma^{k}_{\infty}(\sigma)$. Using [25][Theorem 6, page 841], we deduce that

[TABLE]

Integrating $k$ over $p_{1}$ yields

[TABLE]

Taking $\delta=\eta(4|\mathcal{K}||\mathcal{S}||\mathcal{M}|)^{-1}$, we conclude that $\sigma$ is a robust strategy. By Theorem 2.9, for all $\varepsilon>0$, there exists an $\varepsilon$-optimal finite-memory strategy, which is thus robust. ∎

Corollary 3.10**.**

Let $\mathcal{K},\mathcal{A},\mathcal{S}$ be finite sets, $g\colon\mathcal{K}\times\mathcal{A}\rightarrow\mathbb{R}$ a reward function, and $p_{1}\in\Delta(\mathcal{K})$ an initial belief. The mapping from $(\Delta(\mathcal{K}\times\mathcal{S})^{\mathcal{K}\times\mathcal{A}},d)$ to $\mathbb{R}$ that maps each transition function $q$ to the value at $p_{1}$ of the POMDP $(\mathcal{K},\mathcal{A},\mathcal{S},g,q)$ is lower semi-continuous.

Proof.

Let $q\in\Delta(\mathcal{K}\times\mathcal{S})^{\mathcal{K}\times\mathcal{A}}$. Let $\Gamma=(\mathcal{K},\mathcal{A},\mathcal{S},q,g)$. By the previous lemma, for all $\varepsilon>0$, there exists $\sigma_{\varepsilon}$ a robust $\varepsilon$-optimal strategy in $\Gamma(p_{1})$. Take $\eta=\varepsilon$, by robustness of $\sigma_{\varepsilon}$, there exists $\delta>0$ such that, for all $q^{\prime}\in\Delta(\mathcal{K}\times\mathcal{S})^{\mathcal{K}\times\mathcal{A}}$, we have that if $d(q,q^{\prime})\leq\delta$, then $\gamma^{\prime p_{1}}_{\infty}(\sigma_{\varepsilon})\geq\gamma^{p_{1}}_{\infty}(\sigma_{\varepsilon})-\varepsilon$. Also, by $\varepsilon$-optimality of $\sigma_{\varepsilon}$, we have that $\gamma^{p_{1}}_{\infty}(\sigma_{\varepsilon})\geq v_{\infty}(p_{1})-\varepsilon$. Then,

[TABLE]

Taking $\varepsilon\to 0$, we conclude that

[TABLE]

and thus $v_{\infty}$ is lower semi-continuous with respect to $q$. ∎

Lower semi-continuity of the value function is the best result one can achieve in the following sense.

Proposition 3.11**.**

There is a POMDP such that the mapping from $(\Delta(\mathcal{K}\times\mathcal{S})^{\mathcal{K}\times\mathcal{A}},d)$ to $\mathbb{R}$ that maps each transition function $q$ to the value at $p_{1}$ of the POMDP $(\mathcal{K},\mathcal{A},\mathcal{S},g,q)$ is discontinuous.

4 Examples

In this section, we introduce examples to prove negative results (Propositions 2.12, 3.4 and 3.11) and to illustrate our techniques later on.

4.1 Negative results

Let us prove Propositions 2.12 and 3.4 by presenting an example for each statement.

4.1.1 Proof of Proposition 2.12

We will prove that there exists a POMDP and $\varepsilon>0$ with no $\varepsilon$-optimal finite-recall strategy by an explicit construction. Recall that a strategy has finite recall if it uses only a finite number of the last stages in the current history to decide the next action (see Definition 2.6). Therefore, our construction should have the property that, for any finite-recall strategy, there is a pair of finite histories such that:

The last stages are identical, i.e., the player did the same actions and received the same signals in the last part of both histories (but the starting point was different). 2. 2.

Taking the same decision in both histories leads to losing some reward that can not be compensated in the long-run. 3. 3.

The previous loss does not decrease to zero by increasing the amount of memory.

Example 4.1**.**

Consider the POMDP $\Gamma=(\mathcal{K},\mathcal{A},\mathcal{S},q,g)$ with five states: $k_{0},k_{1},\ldots,k_{4}$. The initial state is $k_{0}$ and players know it (formally, the initial belief is $\delta_{k_{0}}$). The state $k_{4}$ is an absorbing state from where it is impossible to get out and rewards are zero. The states $k_{1}$ and $k_{2}$ form a sub-game where the optimal strategy is trivial. This is the same for the state $k_{3}$. From $k_{0}$ a random initial signal is given indicating which sub-game the state moved to. The key idea is that there is an arbitrarily long sequence of actions and signals which can be gotten in both sub-games, but the optimal strategy behaves differently in each of them. Therefore, to forget the initial signal of the POMDP leads to at most half of the optimal value.

Figure 2 is a representation of $\Gamma$: first under action $a$ and then action $b$. Each state is followed by the corresponding reward, and the arrows include the probability for the corresponding transition along with the signal obtained.

The sub-game of $k_{1}$ and $k_{2}$ has a unique optimal strategy: play action $a$ until receiving signal $s_{2}$, then play action $b$ once and repeat. The value of this sub-game is $1$ and deviating from the prescribed strategy would lead to a long-run reward of [math]. Similarly, the value of the sub-game of $k_{3}$ has a unique optimal strategy: to always play action $a$. Again, the value of this sub-game is $1$ and playing any other strategy leads to a long-run reward of [math].

By the previous discussion, the value of this game starting from $k_{0}$ is $1$. On the other hand, the maximum value obtained by strategies with finite recall is only $1/2$, by playing, for example, always action $a$. Finite-recall strategies achieve at most $1/2$ because, no matter how much finite recall there is, by playing the game the decision-maker faces a history of having played always action $a$ and always receiving a signal $s_{1}$, except for the last signal which is $s_{2}$. Then, if action $b$ is played, the second sub-game is lost; if action $a$ is played, the first sub-game is lost. That is why, for any $0<\varepsilon<1/2$, there is no $\varepsilon$-optimal finite-recall strategy for this POMDP.

4.1.2 Proof of Proposition 3.4

We will show that for the limsup-average and limsup objectives there is a blind MDP where there is no $\varepsilon$-optimal finite-memory strategy. For both cases, the example is constructed with the following idea in mind. To achieve the optimal value, the decision-maker needs to play an action $a_{1}$ for some period, then play another action $a_{2}$ and repeat the process. The key is to require that the length of the period gets longer as the game progresses. This kind of strategy can not be achieved with finite-memory strategies.

For the limsup-average objective, the blind MDP example is due to Venel and Ziliotto [28] and is presented below.

Example 4.2**.**

Consider two states $k_{0}$ and $k_{1}$ and the player receives a reward only when the state is $k_{1}$. To reach $k_{1}$, the decision-maker can play action change and move between the two states. By playing action wait, the state does not change.

Figure 3 is a representation of the game.

Consider the initial belief $p_{1}=\frac{1}{2}\cdot\delta_{k_{0}}+\frac{1}{2}\cdot\delta_{k_{1}}$, the uniform distribution. It is easy to see that finite-memory strategies (or equivalently finite-recall strategies) can not achieve more than $1/2$. On the other hand, the value of this game with the limsup-average objective is $1$, and is guaranteed by the following strategy:

[TABLE]

Hence, finite-memory strategies do no guarantee any approximation for POMDPs with limsup-average objective.

For the limsup objective, the blind MDP example is the following.

Example 4.3**.**

Consider four states ($k_{0},k_{1},k_{2}$ and $k_{3}$) and two actions: wait ($w$) and change ($c$). The initial state is $k_{1}$ and players know it. In $k_{1}$, if $w$ is played, then the state moves to $k_{2}$ with probability $1/2$ and stays with probability $1/2$; if $c$ is played, then the absorbing state $k_{0}$ is reached. From state $k_{2}$, if we play $w$, we stay in the same state; if we play $c$, we move to state $k_{3}$. From $k_{3}$, the only state that has a positive reward, if we play any action, we return to the initial state $k_{1}$.

Figure 4 is a representation of the game.

In this blind MDP (see [5, 2]), for the limsup objective, for any $\varepsilon>0$, there is an infinite-memory strategy that guarantees $1-\varepsilon$, so the value of the game is $1$. On the other hand, applying any finite-memory strategy (or equivalently finite-recall strategies) yields a *limsup *reward of [math]. Hence, finite-memory strategies do no guarantee any approximation for POMDPs with limsup objective.

4.1.3 Proof of Proposition 3.11

We will show that there is a POMDP with discontinuous value with respect to the transition function. The idea is to have two possible scenarios where signals are slightly different. By analyzing a long sequence of signals, the player is able to identify which is the scenario of the current state and so take a better strategy. The following example considers a transition function parameterized by $\varepsilon\geq 0$.

Example 4.4**.**

Consider two states $(k_{u},k_{d})$ and three actions: up ($a_{u}$), down ($a_{d}$) and wait ($a_{w}$). Signals are relevant only for action $a_{w}$: under a non-symmetric transition function, they inform about the underlying state. More concretely, there are two signals $s_{u}$ and $s_{d}$. Playing actions $a_{u}$ or $a_{d}$ will give signal $s_{u}$ or $s_{d}$ respectively, adding no information. Playing action $a_{w}$ leads to signals $s_{u}$ and $s_{d}$ with slightly different probabilities if the state is $k_{u}$ or $k_{d}$. In terms of actions and rewards, $a_{u}$ leads to positive reward only if the state is $k_{u}$, similarly, $a_{d}$ leads to positive reward only if the state is $k_{d}$. Finally, $a_{w}$ leads to null reward in both states. Figure 5 is a representation of this POMDP where transitions are specified.

Consider an initial belief $p_{1}=1/2\cdot\delta_{k_{u}}+1/2\cdot\delta_{k_{d}}$. If $\varepsilon=0$, the value is $1/2$, achieved for example by the constant strategy $\sigma\equiv a_{u}$. Note that playing action $a_{w}$ leads to no information since the random signal the player receives is independent of the underlying state. In contrast, if $\varepsilon>0$, playing action $a_{w}$ reveals information about the underlying state. If action $a_{w}$ is played sufficiently many times, the player can estimate the state by comparing the number of signals $s_{d}$ against $s_{u}$: if $s_{d}$ appear more than $s_{u}$, then it is more probable that the underlying state is $k_{d}$. Therefore, by playing $a_{w}$, the player can estimate the state with increasing probability. That is why the value for $\varepsilon>0$ is $1$. This proves that this POMDP is discontinuous with respect to the transition function since

[TABLE]

4.2 Illustrative examples

We show an example of POMDP that will be analyzed in Section 6.2 in light of our technique. This example comes in two variants differing in sophistication.

4.2.1 Simple version

Let us explain the easiest version.

Example 4.5**.**

Consider two states ($k_{u},k_{d}$) and two actions: up ($a_{u}$) and down ($a_{d}$). All transitions are possible (including loops) and they do not depend on the action. Signals inform the player when the state changes. In terms of actions and rewards, by playing $a_{u}$ the player obtains a reward of $1$ only if the current state is $k_{u}$. Similarly, by playing $a_{d}$ the player obtains a reward of $1$ only if the state is $k_{d}$. Figure 6 is a representation of the game with specific transition probabilities.

Consider an initial belief $p_{1}=1/4\cdot\delta_{k_{u}}+3/4\cdot\delta_{k_{d}}$. During a play, the decision-maker can have two beliefs, $1/4\cdot\delta_{k_{u}}+3/4\cdot\delta_{k_{d}}$ or $3/4\cdot\delta_{k_{u}}+1/4\cdot\delta_{k_{d}}$, because the signals notify when there has been a change. The value of this game is $3/4$. An optimal strategy is to play action $a_{d}$ until getting a signal $s_{c}$, then playing action $a_{u}$ until getting a signal $s_{c}$, and repeat.

4.2.2 Involved version

Let us go to the more complex version. Now the transition between the two extremes includes more states, instead of being a direct jump.

Example 4.6**.**

Consider six states and four actions: up ($a_{u}$), down ($a_{d}$), left ($a_{l}$) and right ($a_{r}$). States can be separated into two groups: extremes ($k_{u}$ and $k_{d}$) and transitional ($k_{l_{1}},k_{r_{1}},k_{l_{2}}$ and $k_{r_{2}}$). Furthermore, transitional states can be divided into two groups: left states ($k_{l_{1}}$ and $k_{l_{2}}$) and right states ($k_{r_{1}}$ and $k_{r_{2}}$). Transitions are from extreme states to transitional states and from transitional to extremes. More precisely, excluding loops, only the following transitions are possible: from $k_{u}$ to either $k_{l_{1}}$ or $k_{r_{1}}$, then from these two to $k_{d}$, from $k_{d}$ to either $k_{l_{2}}$ or $k_{r_{2}}$ and then back to $k_{u}$. Signals are such that the player knows: (i) the state changed to an extreme state, or (ii) the state changed to a transitional state and the new state is with higher probability a left state or a right state. In terms of actions and rewards, each action has an associated set of states in which the reward is $1$ and the rest is [math]: by playing $a_{u}$ the reward is $1$ only if the current state is $k_{u}$, playing $a_{d}$ rewards only state $k_{d}$, $a_{l}$ rewards states $k_{l_{1}}$ and $k_{l_{2}}$, and $a_{r}$ rewards states $k_{r_{1}}$ and $k_{r_{2}}$. Figure 7 is a representation of this game with specific transition probabilities.

Consider an initial belief $p_{1}=1/4\cdot\delta_{k_{u}}+3/4\cdot\delta_{k_{d}}$. The value of the game is $21/32$. An optimal strategy is given by playing action $a_{d}$ until getting a signal $s_{l}$ or $s_{r}$. If the decision-maker got signal $s_{l}$, then play action $a_{l}$, otherwise, play action $a_{r}$. Repeat action $a_{l}$ or $a_{r}$ until getting the signal $s_{c}$. Then, play $a_{u}$ until getting a signal $s_{l}$ or $s_{r}$. When this happens, play $a_{l}$ or $a_{r}$ accordingly until getting signal $s_{c}$. And so, repeat the cycle.

The belief dynamic under this optimal strategy is the following. The initial belief is $p_{1}$, supported in the extreme states. By getting a signal $s_{w}$, the belief does not change. By getting signal $s_{l}$, the weight on $k_{u}$ distributes between states $k_{l_{1}}$ and $k_{r_{1}}$ in a proportion $3:1$ and the weight on $k_{d}$ distributes between $k_{l_{2}}$ and $k_{r_{2}}$ in the same way. By getting signal $s_{r}$, the distribution is similar, but the role of left states are interchanged with right states. Once the belief is in the transitional states, by playing the respective action (either $a_{l}$ or $a_{r}$), the belief does not change while receiving signal $s_{w}$. Upon receiving the signal $s_{c}$, the new belief is $3/4\cdot\delta_{k_{u}}+1/4\cdot\delta_{k_{d}}$. By symmetry of the POMDP, the dynamic is then similar until getting signal $s_{c}$ for a second time. At that time, the belief is equal to the initial distribution, namely $1/4\cdot\delta_{k_{u}}+3/4\cdot\delta_{k_{d}}$.

Remark 4.7*.*

For the decision-maker to have a finite-memory strategy, some quantity with finitely many options must be updated over time. A tentative idea is to compute the posterior belief, but it can take infinitely many values. In this example, using a belief partition is enough to encode an optimal strategy. In general, it is an open question if a belief partition is sufficient to achieve $\varepsilon$-optimal strategies.

5 Structure of the proof

In this section, we introduce two key lemmas and derive from them the proof of Theorem 2.9. We first define the history at stage $m$, which is all the information the decision-maker has at stage $m$.

Definition 5.1** ($m$-stage history).**

Given a strategy $\sigma\in\Sigma$ and an initial belief $p_{1}$, denote the (random) history at stage $m$ by

[TABLE]

The random variable $H_{m}$ takes values in $\mathcal{H}_{m}=(\mathcal{A}\times\mathcal{S})^{m-1}$.

Recall that we denote the state at stage $m$ by $K_{m}$, which takes values in $\mathcal{K}$; the signal at stage $m$ by $S_{m}$, which takes values in $\mathcal{S}$; and the action at stage $m$ by $A_{m}$, which takes values in $\mathcal{A}$. Note that the history at stage $m$ does not contain direct information about the states $K_{1},\ldots,K_{m}$.

The belief of the player at stage $m$ plays a key role in the study of POMDPs, and we formally define it as follows.

Definition 5.2** ($m$-stage belief).**

Given a strategy $\sigma\in\Sigma$ and an initial belief $p_{1}$, denote the belief at stage $m$ by $P_{m}$, which is given by, for all $k\in\mathcal{K}$,

[TABLE]

For fixed $\sigma$ and $p_{1}$, one can use Bayes rule to compute $P_{m}$. To avoid heavy notations, we omit the dependence of $P_{m}$ on $\sigma$ and $p_{1}$. For $p\in\Delta(\mathcal{K})$, denote the support of $p$ by $\operatorname{supp}(p)$, which is the set of $k\in\mathcal{K}$ such that $p(k)>0$.

The first ingredient of the proof of Theorem 2.9 is the following lemma.

Lemma 5.3**.**

For any initial belief $p_{1}$ and $\varepsilon>0$, there exists $m_{\varepsilon}\geq 1$, $\sigma^{\varepsilon}\in\Sigma$ and a (random) belief $P^{*}\in\Delta(\mathcal{K})$ (which depends on the history before stage $m_{\varepsilon}$) such that:

[TABLE] 2. 2.

There exists $\sigma\in\Sigma$, which depends on $P^{*}$, such that for all $k\in\operatorname{supp}(P^{*})$

[TABLE]

Moreover, $\gamma_{\infty}^{P^{*}}(\sigma)=v_{\infty}(P^{*})$ and $\mathbb{E}^{p_{1}}_{\sigma^{\varepsilon}}(v_{\infty}(P^{*}))\geq v_{\infty}(p_{1})-\varepsilon$.

This result is a consequence of Venel and Ziliotto [27, Lemma 33]. This previous work states the existence of elements $\mu^{*}\in\Delta(\Delta(\mathcal{K}))$ and $\sigma^{*}\in\Delta(\Sigma)$ with similar properties to those of $P^{*}\in\Delta(\mathcal{K})$ and $\sigma\in\Sigma$. In this sense, the present lemma can be seen as a deterministic version of this previous result. To focus on the new tools we introduce in this paper to prove Theorem 2.9, we relegate the proof and the explanation of the differences between the two lemmata to Appendix A.

Remark 5.4*.*

The first property of Lemma 5.3 follows immediately from [27, Lemma 33] by the type of convergence in this previous result. On the other hand, the second property requires the introduction of a certain Markov chain on $\mathcal{K}\times\mathcal{A}\times\Delta(\mathcal{K})$. This Markov chain is already present in the work [27] but was used for other purposes. Therefore, the proof consists mainly of recalling previous results and constructions.

Remark 5.5*.*

Note that $\mathbb{P}^{k}_{\sigma}$ represents the law on plays induced by the strategy $\sigma$, conditional on the fact that the initial state is $k$. This does not mean that we consider the decision-maker to know $k$. In the same fashion, $\gamma^{k}_{\infty}(\sigma)$ is the reward given by the strategy $\sigma$, conditional on the fact that the initial state is $k$. Even though $\sigma$ is optimal in $\Gamma(P^{*})$, this does not imply that $\sigma$ is optimal in $\Gamma(\delta_{k})$: we may have $\gamma^{k}_{\infty}(\sigma)<v_{\infty}(\delta_{k})$.

The importance of Lemma 5.3 comes from the fact that the average rewards converge almost surely to a limit that only depends on the initial state $k$. Intuitively, this result means that for any initial belief $p_{1}$, after a finite number of stages, we can get $\varepsilon$-close to a belief $P^{*}$ such that the optimal reward from $P^{*}$ is, in expectation, almost the same as from $p_{1}$, and moreover from $P^{*}$ there exists an optimal strategy that induces a strong ergodic behavior on the state dynamics. Thus, there is a natural way to build a $3\varepsilon$-optimal strategy $\tilde{\sigma}$ in $\Gamma(p_{1})$: first, apply the strategy $\sigma^{\varepsilon}$ for $m_{\varepsilon}$ stages, then apply $\sigma$. Since after $m_{\varepsilon}$ steps the current belief $P_{m_{\varepsilon}}$ is $\varepsilon$-close to $P^{*}$ with probability higher than $1-\varepsilon$, the reward from playing $\tilde{\sigma}$ is at least the expectation of $\gamma^{P^{*}}_{\infty}(\sigma)-2\varepsilon$, which is greater than $v_{\infty}(p_{1})-3\varepsilon$. Therefore, this procedure yields a $3\varepsilon$-optimal strategy. Nonetheless, $\sigma$ may not have finite memory, and thus $\tilde{\sigma}$ may not have either. The main difficulty of the proof is to transform $\sigma$ into a finite-memory strategy. We formalize this discussion below.

Definition 5.6** (ergodic strategy).**

Let $p^{*}\in\Delta(\mathcal{K})$. We say that a strategy $\sigma$ is ergodic for $p^{*}$ if the following holds for all $k\in\operatorname{supp}(p^{*})$

[TABLE]

From the previous discussion, we aim at proving the following result.

Lemma 5.7**.**

Let $p^{*}\in\Delta(\mathcal{K})$ and $\sigma$ be an ergodic strategy for $p^{*}$. For all $\varepsilon>0$, there exists a finite-memory strategy $\sigma^{\prime}$ such that

[TABLE]

This is our key lemma and the main technical contribution. The next section is devoted to explaining the technique used and proving it.

Proof of Theorem 2.9 assuming Lemmas 5.7 and 5.3.

Let $p_{1}$ be an initial belief and $\varepsilon>0$. Let $m_{\varepsilon}$, $\sigma^{\varepsilon}$, $P^{*}$ and $\sigma$ be given by Lemma 5.3. Define the strategy $\sigma^{0}$ by: playing $\sigma^{\varepsilon}$ until stage $m_{\varepsilon}$, then switch to the strategy $\sigma^{\prime}$ given by Lemma 5.7 for $\sigma$ and $p^{*}=P^{*}$. Note that $\sigma^{0}$ has finite memory. We have

[TABLE]

and the theorem is proved.

∎

6 Super-support and proof of Lemma 5.7

In this entire section, fix $p^{*}\in\Delta(\mathcal{K})$, which will be used as an initial belief, and $\sigma$ an ergodic strategy for $p^{*}$.

6.1 Notation

For $a,b\in\mathbb{R}$, denote the set $[a,b]\cap\mathbb{Z}$ by $[a\,..\,b]$.

Definition 6.1** (Value partition).**

Let $\sim$ be the equivalence relationship on $\operatorname{supp}(p^{*})$ defined by $k\sim k^{\prime}$ if and only if $\gamma^{k}_{\infty}(\sigma)=\gamma^{k^{\prime}}_{\infty}(\sigma)$. Let $\{\mathcal{K}_{1},\ldots,\mathcal{K}_{I}\}$ be the corresponding value partition.

Definition 6.2** (Super-support).**

For $i\in[1\,..\,I]$ and $m\geq 0$, define, for all $h_{m}\in\mathcal{H}_{m}$,

[TABLE]

In other words, $\mathcal{B}^{i}_{m}(h_{m})$ is the set of all reachable states at stage $m$ starting from some state in $\mathcal{K}_{i}$, playing the strategy $\sigma$ and obtaining history $h_{m}$ (if $H_{m}=h_{m}$ is possible). Denote

[TABLE]

the random set associated with $H_{m}$, and $B_{m}\coloneqq(B^{1}_{m},\ldots,B^{I}_{m})$ the super-support at stage $m$.

Remark 6.3*.*

Note that

[TABLE]

Therefore, the support of $P_{m}$ can be deduced from the super-support $B_{m}$. On the other hand, $B_{m}$ can not be deduced from $P_{m}$, and thus can not be deduced from the support of $P_{m}$. This justifies the vocabulary.

We will build a finite-memory $\varepsilon$-optimal strategy that plays by blocks. Each block has fixed finite length and, within each block, the strategy depends only on the history in the block and on the super-support at the beginning of the block. At the end of the block, the automaton computes the new super-support according to the block history and the previous super-support. Thus, the only difference with a bounded recall strategy is that our strategy keeps track of the super-support. Super-support is a type of origin information: it is related to the value partition, and therefore to where the current mass distribution comes from.

Definition 6.4** ($h_{m}$-shift).**

Let $m\geq 1$ and $h_{m}\in\mathcal{H}_{m}$. The $h_{m}$-shift of $\sigma$ is the strategy $\sigma[h_{m}]$ defined by, for all $m^{\prime}\geq 1$,

[TABLE]

We denote $\sigma_{m}\coloneqq\sigma[H_{m}]$, the corresponding random shift at stage $m$.

In other words, $\sigma[h_{m}]$ corresponds to the continuation of the strategy $\sigma$ conditional on the fact that the history of the first $m$ stages was $h_{m}$.

6.2 Illustration

The super-support captures specific information related to the beginning of the game: the origin of the current mass distribution (given by $P_{m}$) in terms of the initial value partition $(\mathcal{K}_{i})_{i\in[1\,..\,I]}$. There are finitely many possible super-supports and it is possible to keep track of the current super-support using Bayesian updating. Therefore, it is a good variable to be used in finite-memory strategies.

Let us recall our simple example of a POMDP, Example 4.5.

See 4.5

Finite-recall is enough to approximate the value of this POMDP: the decision-maker can recall the last action. Then, upon seeing the signal $s_{c}$, the player has to change actions. Recall that $p_{1}=1/4\cdot\delta_{k_{u}}+3/4\cdot\delta_{k_{d}}$. Therefore, an optimal strategy is given by playing $a_{d}$ until getting signal $s_{c}$, then playing $a_{u}$ until getting signal $s_{c}$ and repeat. This strategy is ergodic for $p_{1}$ and the corresponding value partition is given by $(\mathcal{K}_{1}=\{k_{u}\},\mathcal{K}_{2}=\{k_{d}\})$, because, if $K_{1}=k_{u}$, the long-run reward is [math] and, if $K_{1}=k_{d}$, the long-run reward is $1$. In this case, the super-support describes completely the belief $P_{m}$ since it keeps track of which state has the highest (or lowest) probability.

Although the example is simple, we can already see the difference between support strategies and super-support strategies. In this case, all strategies based on the current support (the support of $P_{m}$) are constant and therefore can achieve a long-run reward of at most $1/2$. On the other hand, super-support strategies can be optimal and achieve a long-run reward of $3/4$.

This example also shows that playing by blocks and defining the behaviour in each block by the current support (instead of the super-support) is not enough.

Let us analyze now our more complex POMDP example, Example 4.6.

See 4.6

Recall that $p_{1}=1/4\cdot\delta_{k_{u}}+3/4\cdot\delta_{k_{d}}$ and that an optimal strategy is given by playing action $a_{d}$ until getting a signal $s_{l}$ or $s_{r}$. If the decision-maker gets signal $s_{l}$, then play action $a_{l}$, otherwise, play action $a_{r}$. Repeat action $a_{r}$ until getting the signal $s_{c}$. Then, play $a_{u}$ until getting a signal $s_{l}$ or $s_{r}$. When this happens, play $a_{l}$ or $a_{r}$ accordingly until getting signal $s_{c}$. And so, repeat the cycle.

This optimal strategy is ergodic for $p_{1}$ and the corresponding value partition is given by $(\mathcal{K}_{1}=\{k_{u}\},\mathcal{K}_{2}=\{k_{d}\})$ because, if $K_{1}=k_{u}$, the long-run reward is [math] and, if $K_{1}=k_{d}$, the long-run reward is $7/8$. Contrary to the previous example, the super-support does not describe completely the belief $P_{m}$. Indeed, consider the initial belief $p_{1}$, which is supported on the extreme states, and that the decision-maker gets either signals $s_{l}$ or $s_{r}$. Then, the new belief is supported in all the transitional states and the super-support is the same under any of these two histories, and equal to: $B=(B^{1}=\{k_{l_{1}},k_{r_{1}}\},B^{2}=\{k_{l_{2}},k_{r_{2}}\})$. Based on this super-support one can not reconstruct the current belief, but one knows more than only the support: we can differentiate the origin ($k_{u}$ or $k_{d}$) of the current belief distribution.

Notice that using the super-support alone is not enough to get $\varepsilon$-optimal strategies. Indeed, in transitional states, the decision-maker needs to know whether the state is more likely to be in a left state or a right state in order to play well, and the super-support does not contain such information. That is why, in the proof of Lemma 5.7, we consider a more sophisticated class of strategies, that combine super-support and bounded recall. For the moment, let us describe such a strategy for this example. Choose $n_{0}$ very large, and for each $\ell\geq 1$, play the following strategy in the time block $[\ell n_{0}+1\,..\,(\ell+1)n_{0}]$:

Case 1: the super-support at stage $\ell n_{0}+1$ is $(\{k_{u}\},\{k_{d}\})$. Play the previous 0-optimal strategy, that is: play action $a_{d}$ until getting a signal $s_{l}$ or $s_{r}$. If the decision-maker gets signal $s_{l}$, then play action $a_{l}$, otherwise, play action $a_{r}$. Repeat action $a_{r}$ until getting the signal $s_{c}$. Then, play $a_{u}$ until getting a signal $s_{l}$ or $s_{r}$. When this happens, play $a_{l}$ or $a_{r}$ accordingly until getting signal $s_{c}$. And so, repeat the cycle.

• -

Case 2: the super-support at stage $\ell n_{0}+1$ is $(\{k_{d}\},\{k_{u}\})$. Play the same strategy as in Case 1, except that the roles of $a_{u}$ and $a_{d}$ are switched.

• -

Case 3: the super-support at stage $\ell n_{0}+1$ is $(\{k_{l_{1}},k_{r_{1}}\},\{k_{l_{2}},k_{r_{2}}\})$. Play $a_{l}$ (or $a_{r}$) until getting the signal $s_{c}$. At this point, the super-support is $(\{k_{d}\},\{k_{u}\})$. Then, play as in Case 2.

• -

Case 4: the super-support at stage $\ell n_{0}+1$ is $(\{k_{l_{2}},k_{r_{2}}\},\{k_{l_{1}},k_{r_{1}}\})$. Play $a_{l}$ (or $a_{r}$) until getting the signal $s_{c}$. At this point, the super-support is $(\{k_{u}\},\{k_{d}\})$. Then, play as in Case 1.

This strategy is sub-optimal during the first phase of Case 3 and Case 4, until the decision-maker receives signal $s_{c}$. As $n_{0}$ grows larger and larger, this part becomes negligible. Thus, for any $\varepsilon>0$, there exists $n_{0}$ such that this strategy is $\varepsilon$-optimal (but not optimal).

6.3 Properties

Now we can state properties of super-supports when the strategy $\sigma$ is ergodic for $p^{*}$ and explain how rich is the structure of the random sequence of beliefs $(P_{m})_{m\geq 1}$. By definition of ergodic strategies, the map $k\mapsto\gamma^{k}_{\infty}(\sigma)$ is constant on $\mathcal{K}_{i}$, and we denote its value by $\gamma^{i}_{\infty}$.

Lemma 6.5** (Continuation value).**

For all $m\geq 1$ and $i\in[1\,..\,I]$ it holds $\mathbb{P}^{p^{*}}_{\sigma}$-a.s. that

[TABLE]

Consequently, $B^{1}_{m},\ldots,B^{I}_{m}$ are disjoint $\mathbb{P}^{p^{*}}_{\sigma}$-a.s.

Proof.

Let $i\in[1\,..\,I]$. Considering the law given by $\mathbb{P}^{p^{*}}_{\sigma}$, fix a realization $K_{m}=k\in B^{i}_{m}$. By definition of super-support, there exists $k^{\prime}\in\mathcal{K}_{i}\subseteq\operatorname{supp}(p^{*})$ such that $k$ can be reached from $k^{\prime}$ in $m$ steps. Recall that, since $\sigma$ is ergodic for $p^{*}$,

[TABLE]

In particular, the convergence holds when $K_{m}=k$. Then,

[TABLE]

and therefore $\gamma^{k}_{\infty}(\sigma_{m})=\gamma^{i}_{\infty}$. ∎

Another property of the super-support is concerned with consecutive conditioning and is fairly intuitive. We formally state it in the following lemma and show the proof for completeness.

Lemma 6.6** (Continuation super-support).**

Let $i\in[1\,..\,I]$, $m,m^{\prime}\geq 0$. For all realizations $H_{m+m^{\prime}}=h=(h_{m},h_{m^{\prime}})\in\mathcal{H}_{m+m^{\prime}}$, denoting $\mathcal{C}=\mathcal{B}^{i}_{m+m^{\prime}}(h_{m+m^{\prime}})$, we have that, for all $k\in\mathcal{B}^{i}_{m}(h_{m})$,

[TABLE]

In other words, the super-support that arises at stage $m+m^{\prime}$, $B_{m+m^{\prime}}$, coincides with the super-support that would arise from a two-step procedure: first, advancing $m$ stages; and then, applying the continuation of the strategy, $\sigma_{m}$, for $m^{\prime}$ more stages.

Proof.

Fix a realization $H_{m+m^{\prime}}=h=(h_{m},h_{m^{\prime}})$ and let $k\in\mathcal{B}^{i}_{m}(h_{m})$. Recall that, by definition of super-support,

[TABLE]

Therefore, there exists $\bar{k}_{1}\in\mathcal{K}_{i}$ such that $k\in\operatorname{supp}(\mathbb{P}^{\bar{k}_{1}}_{\sigma}(K_{m}=\cdot\mid H_{m}=h_{m}))$. In particular, we have that $\mathbb{P}^{\bar{k}_{1}}_{\sigma}(K_{m}=k)>0$.

Consider $k^{\prime}$ such that $\mathbb{P}^{k}_{\sigma[h_{m}]}(K_{m^{\prime}}=k^{\prime}\mid H_{m^{\prime}}=h_{m^{\prime}})>0$. By a semi-group property, we deduce that

[TABLE]

which implies that $k^{\prime}\in\mathcal{B}^{i}_{m}(h)=\mathcal{C}$, and thus the lemma is proved. ∎

Remark 6.7*.*

This property does not depend on the fact that $\sigma$ is ergodic for $p^{*}$.

6.4 Proof of Lemma 5.7

Fix $p^{*}\in\Delta(\mathcal{K})$ such that $\sigma$ is ergodic for $p^{*}$. Note that, for all $m\geq 1$, $B_{m}\in\{(\mathcal{C}_{1},\ldots,\mathcal{C}_{I}):\mathcal{C}_{1},\ldots,\mathcal{C}_{I}\subseteq\mathcal{K}\}$, which is a finite set. Denote all different super-supports that can occur with positive probability by $\mathcal{D}^{1},\mathcal{D}^{2},\ldots,\mathcal{D}^{J}$, i.e.

[TABLE]

Moreover, since $\mathcal{D}^{j}$ corresponds to a super-support that occurs at some stage and under some history, there exists $h^{j}$ and $m_{j}$ such that $h^{j}\in\mathcal{H}_{m_{j}}$ and $\mathcal{D}^{j}=\mathcal{B}^{i}_{m}(h^{j})$. In other words, $\mathcal{D}^{j}$ is the realization of the super-support at stage $m_{j}$ under history $h^{j}$ and $\left\{\mathcal{D}^{1},\mathcal{D}^{2},\ldots,\mathcal{D}^{J}\right\}$ contains all super-supports that can occur.

Definition of the strategy $\sigma^{\prime}$.

Let $\varepsilon>0$. By Lemma 6.5, there exists $n_{0}\in\mathbb{N}^{*}$ such that for all $i\in[1\,..\,I]$, $j\in[1\,..\,J]$ and $k\in\mathcal{D}^{j}_{i}$,

[TABLE]

Define the strategy $\sigma^{\prime}$ by blocks, and characterize each block by induction. For each $\ell\geq 0$, the block number $\ell$ consists in the stages $m$ such that $\ell n_{0}+1\leq m\leq(\ell+1)n_{0}$. We characterize the behavior in block $\ell$ by a variable $J_{\ell}\in[1\,..\,J]$ in the following way. For stage $m$ inside block $\ell$, the strategy $\sigma^{\prime}$ plays according to $J_{\ell}$ and the history between stages $\ell n_{0}+1$ and $m$. Each block is characterized by induction because the variable $J_{\ell}$ is computed at stage $\ell n_{0}+1$ according to $J_{(\ell-1)}$ and the history in the last $n_{0}$ stages. Thus, $\sigma^{\prime}$ can be seen as mapping from $\cup_{m=1}^{n_{0}}\mathcal{H}_{m}\times[1\,..\,J]$ to $\mathcal{A}$.

Consider $\ell=0$, i.e. the first block. The strategy $\sigma^{\prime}$ is defined on the first $n_{0}$ stages as follows. Consider the value partition $\{\mathcal{K}_{1},\ldots,\mathcal{K}_{I}\}$ given by $p^{*}$ and $\sigma$. By definition of $\mathcal{D}^{1},\mathcal{D}^{2},\ldots,\mathcal{D}^{J}$, there exists $j\in[1\,..\,J]$ such that $B_{1}=\mathcal{D}^{j}$. Set $J_{0}=j$, and define $\sigma^{\prime}(h,J_{0})\coloneqq\sigma(h^{J_{0}},h)$ for all $h\in\mathcal{H}_{m}$ and $m\leq n_{0}$.

Let us proceed to the induction step. Consider $\ell\geq 1$ and assume that we have defined $J_{(\ell-1)}$ and $\sigma^{\prime}$ up to stage $\ell n_{0}$. Denote the history between stages $(\ell-1)n_{0}+1$ and $\ell n_{0}+1$ by $h\in\mathcal{H}_{n_{0}+1}$ and define $J_{\ell}$ such that, for all $i\in[1\,..\,I]$,

[TABLE]

Then, extend $\sigma^{\prime}$ for $n_{0}$ additional stages as before: $\sigma^{\prime}(h,J_{\ell})\coloneqq\sigma(h^{J_{\ell}},h)$ for all $h\in\mathcal{H}_{m}$ and $m\leq n_{0}$. Thus, we have defined $J_{\ell}$ and extended $\sigma^{\prime}$ up to stage $(\ell+1)n_{0}$.

To summarize our construction in words, during stages $\ell n_{0}+1,\ell n_{0}+2,\ldots,(\ell+1)n_{0}$, the decision-maker plays as if he was playing $\sigma$ from history $h^{J_{\ell}}$. Notice that the indexes $J_{0},J_{1},\ldots$ depend on the history, and therefore are random. Now we will connect the strategy $\sigma^{\prime}$ with the super-support given by $p^{*}$ and $\sigma$.

Lemma 6.8**.**

For all $i\in[1\,..\,I]$, $k\in\mathcal{K}_{i}$ and $\ell\geq 0$, we have that

[TABLE]

Consequently, $\mathcal{D}^{J_{\ell}}=(\mathcal{D}^{J_{\ell}}_{1},\mathcal{D}^{J_{\ell}}_{2},\ldots,\mathcal{D}^{J_{\ell}}_{I})$ is a partition of the support of $P_{\ell n_{0}+1}$. Moreover,

[TABLE]

Proof.

Fix $i\in[1\,..\,I]$ and $k\in\mathcal{K}_{i}$. We will prove the result by induction on $\ell\geq 0$. For $\ell=0$, $\mathcal{D}^{J_{0}}=(\mathcal{K}_{1},\dots,\mathcal{K}_{I})$ and $\mathbb{P}^{k}_{\sigma^{\prime}}(K_{1}\in\mathcal{D}^{J_{0}}_{i}=\mathcal{K}_{i})=1$. Thus, the result holds.

Assume $\ell\geq 1$. Note that $H_{(\ell-1)n_{0}+1}$ determines the value of $J_{0},\ldots,J_{(\ell-1)}$. Therefore, $\mathcal{D}^{J_{(\ell-1)}}_{i}$ is also determined by $H_{(\ell-1)n_{0}+1}$. By induction hypothesis,

[TABLE]

We must prove that, under these circumstances, $K_{\ell n_{0}+1}\in\mathcal{D}^{J_{\ell}}_{i}$.

Indeed, index $J_{\ell-1}$ defines strategy $\sigma^{\prime}$ for stages $\ell n_{0}+1,\ell n_{0}+2,\ldots,(\ell+1)n_{0}$: the decision-maker will play according to $\sigma[h^{J_{(\ell-1)}}]$. By playing $\sigma^{\prime}$ during this block, a history $h\in\mathcal{H}_{n_{0}+1}$ will be collected. Let $m\coloneqq J_{(\ell-1)}$ and $m^{\prime}\coloneqq n_{0}+1$. By Lemma 6.6, we have that, starting from $K_{(\ell-1)n_{0}+1}\in\mathcal{D}^{J_{(\ell-1)}}_{i}$, playing $\sigma[h^{J_{(\ell-1)}}]$ during $n_{0}$ stages and collecting history $h\in\mathcal{H}_{n_{0}+1}$ leads to a state that, by definition of $J_{\ell}$, is in $\mathcal{D}^{J_{\ell}}_{i}$. Therefore,

[TABLE]

which proves the first result.

Now we know that the union of $\mathcal{D}^{J_{\ell}}_{1},\mathcal{D}^{J_{\ell}}_{2},\ldots,\mathcal{D}^{J_{\ell}}_{I}$ covers the support of $\mathbb{P}^{p^{*}}_{\sigma^{\prime}}(K_{\ell n_{0}+1}=\cdot)$. Moreover, by Lemma 6.5, $\mathcal{D}^{J_{\ell}}_{1},\mathcal{D}^{J_{\ell}}_{2},\ldots,\mathcal{D}^{J_{\ell}}_{I}$ are disjoint. Since $(\mathcal{K}_{1},\dots,\mathcal{K}_{I})$ partitions the support of $p^{*}$, we have that, for all $i\in[1\,..\,I]$,

[TABLE]

which proves the second property. ∎

To finish the proof of Lemma 5.7, we must show that the finite-memory strategy $\sigma^{\prime}$ guarantees the reward obtained by $\sigma$ up to $\varepsilon$. The idea is that in each block we are playing some shift of $\sigma$ for $n_{0}$ stages. The shift is chosen so that information about the initial belief is correctly updated, while the number $n_{0}$ is chosen so that the expected average reward of the whole block is close to the expected limit average reward. Then, since all blocks have the same approximation error, the average considering all blocks yields approximately $\gamma^{p^{*}}_{\infty}(\sigma)$. This is the intuition behind the following lemma.

Lemma 6.9**.**

Let $L\in\mathbb{N}^{*}$. The following inequality holds:

[TABLE]

Proof.

We have, for all $\ell\geq 0$,

[TABLE]

It follows that

[TABLE]

∎

To conclude, since $\sigma^{\prime}$ has finite memory, we have

[TABLE]

and the above lemma implies that $\gamma_{\infty}^{p^{*}}(\sigma^{\prime})\geq\gamma_{\infty}^{p^{*}}(\sigma)-\varepsilon$, which proves Lemma 5.7: for each ergodic strategy $\sigma$ and $\varepsilon>0$, one can construct a finite-memory strategy $\sigma^{\prime}$ that guarantees the reward obtained by $\sigma$ up to $\varepsilon$.

Appendix A Proof of Lemma 5.3

A.1 Notation

Recall that Lemma 5.3 is a consequence of [27, Lemma 33]. Thus, we start by introducing some of the terms used in [27], namely: $n$-stage game, invariant measure, occupation measure and the Kantorovich-Rubinstein distance.

Definition A.1** ($n$-stage game).**

Given a POMDP $\Gamma=(\mathcal{K},\mathcal{A},\mathcal{S},q,g)$, we denote $\Gamma_{n}$ the $n$-stage game with a value defined by

[TABLE]

where $\gamma^{p}_{n}(\sigma)\coloneqq n^{-1}\mathbb{E}^{p}_{\sigma}\left(\sum_{m=1}^{n}G_{m}\right)$.

Remark A.2*.*

As the notation suggests, it was proven in [27] that for any finite POMDP $(v_{n})\xrightarrow[n\to\infty]{}v_{\infty}$ uniformly. The fact that $(v_{n})_{n\geq 1}$ converges was proven in [23].

The set $\Delta(\mathcal{K})$ is equipped with its Borelian $\sigma$-algebra $\mathcal{B}(\Delta(\mathcal{K}))$, and $\mathcal{C}(\Delta(\mathcal{K}),[0,1])$ denotes the set of continuous functions from $\Delta(\mathcal{K})$ to $[0,1]$.

Definition A.3** (Invariant measure).**

Given a POMDP $\Gamma=(\mathcal{K},\mathcal{A},\mathcal{S},q,g)$, $\mu\in\Delta(\Delta(\mathcal{K}))$ and $\sigma\colon\Delta(\mathcal{K})\to\Delta(\mathcal{A})$ measurable, we say that $\mu$ is $\sigma$-invariant if $\forall f\in\mathcal{C}(\Delta(\mathcal{K}),[0,1])$ we have that

[TABLE]

where $\tilde{q}\colon\Delta(\mathcal{K})\times\mathcal{A}\rightarrow\Delta(\Delta(\mathcal{K}))$ is the natural transition in $\Delta(\mathcal{K})$ from one belief to another, given by Bayes rule.

The above definition can be intuitively understood in the following way: if the initial belief is distributed according to $\mu$, and the decision-maker plays the stationary strategy $\sigma$ at stage 1, then the belief at stage 2 is distributed according to $\mu$ too.

Remark A.4*.*

Since $v_{\infty}\colon\Delta(\mathcal{K})\to[0,1]$ is a continuous function, one can replace $f$ by $v_{\infty}$ in the previous definition. Moreover, interpreting $\sigma$ as a (mixed) stationary strategy, we would have that the sequence $(\mathbb{E}^{\mu}_{\sigma}[v_{\infty}(P_{m})])_{m\geq 1}$ is constant.

Definition A.5** ($m$-stage occupation measure).**

Given a POMDP $\Gamma=(\mathcal{K},\mathcal{A},\mathcal{S},q,g)$, a measure $\mu\in\Delta(\Delta(\mathcal{K}))$ and a strategy $\sigma$, consider the following dynamic over $\Delta(\mathcal{K})$. First, $P_{1}$ is drawn according to $\mu$. Then, $(P_{n})_{n\geq 1}$ is obtained by playing according to $\sigma$. This way, for each $m\geq 1$, we have that $\Gamma$, $\mu$ and $\sigma$ induce a probability over $\Delta(\mathcal{K})$: for each measurable set $\mathcal{A}\subseteq\Delta(\mathcal{K})$, we can define $\mathbb{P}^{\mu}_{\sigma}(P_{m}\in\mathcal{A})$. Therefore, the $m$-stage belief, $P_{m}$, is a random belief.

We denote the $m$-stage occupation measure $z_{m}[\mu,\sigma]\in\Delta(\Delta(\mathcal{K}))$ by the law of $P_{m}$ over $\Delta(\mathcal{K})$. Formally, $z_{m}[\mu,\sigma]\colon\mathcal{B}(\Delta(\mathcal{K}))\to[0,1]$ is given by, for all $\mathcal{C}\in\mathcal{B}(\Delta(\mathcal{K}))$,

[TABLE]

For sake of notation, we identify $\Delta(\mathcal{K})$ with the extreme points of $\Delta(\Delta(\mathcal{K}))$.

Definition A.6** (Kantorovich-Rubinstein distance).**

For all $z,z^{\prime}\in\Delta(\Delta(\mathcal{K}))$, define

[TABLE]

where $\mathcal{E}_{1}$ is the set of 1-Lipschitz functions from $\Delta(\mathcal{K})$ to $[0,1]$.

Remark A.7*.*

The set $\Delta(\Delta(\mathcal{K}))$ equipped with distance $d_{KR}$ is a compact metric space.

A.2 Proof

Now we can state [27, Lemma 33].

Lemma A.8**.**

Consider a POMDP $\Gamma$ and let $p_{1}\in\Delta(\mathcal{K})$. There exists $\mu^{*}\in\Delta(\Delta(\mathcal{K}))$ and a (mixed) stationary strategy $\sigma^{*}\colon\Delta(\mathcal{K})\to\Delta(\mathcal{A})$ such that

$\mu^{*}$* is $\sigma^{*}$-invariant.* 2. 2.

For all $\varepsilon>0$ and $N\geq 1$, there exists $n_{\varepsilon}\geq N$ and $\sigma^{\varepsilon}$ a pure strategy in $\Gamma$ such that $\sigma^{\varepsilon}$ is 0-optimal in the $n_{\varepsilon}$-stage game $\Gamma_{n_{\varepsilon}}(p_{1})$ and

[TABLE] 3. 3.

[TABLE]

Remark A.9*.*

Lemma A.8 works with elements in $\Delta(\Delta(\mathcal{K}))$ and a mixed stationary strategy, while Lemma 5.3 deals with (random) elements in $\Delta(\mathcal{K})$ and a pure strategy. In this sense, we would like to “go down a level”: moving from $\mu^{*}$ to a random $P^{*}$, from $z_{m}$ to $P_{m}$ and still preserve a relationship between $v_{\infty}(p_{1})$ and $\mathbb{E}_{\sigma^{\varepsilon}}^{p_{1}}(v_{\infty}(P^{*}))$. The ergodic property 2 of Lemma 5.3 follows from the first and third property of Lemma A.8.

Proof of Lemma 5.3.

Consider $p_{1}\in\Delta(\mathcal{K})$ and $\varepsilon>0$ fixed. Since $(v_{n})_{n\geq 1}$ converges uniformly to $v_{\infty}$, consider $N\geq 1$ such that $\varepsilon\geq 1/N$ and $\forall n\geq N\quad||v_{n}-v_{\infty}||_{\infty}\leq\varepsilon$. Now, using Lemma A.8, there exists $\mu^{*}$ and $\sigma^{*}$ such that $\mu^{*}$ is $\sigma^{*}$-invariant and, considering $\varepsilon^{4}$, $\exists n_{\varepsilon}\geq N$ such that

[TABLE]

with $\sigma^{\varepsilon}\in\Gamma_{n_{\varepsilon}}(p_{1})$ an optimal pure strategy for the $n_{\varepsilon}$-stage game starting in $p_{1}$.

We claim that $\exists m_{\varepsilon}\leq\lceil\varepsilon n_{\varepsilon}\rceil$ such that

[TABLE]

Proceeding by contradiction, assume that $\forall m\leq\lceil\varepsilon n_{\varepsilon}\rceil$, we have $\mathbb{P}^{p_{1}}_{\sigma^{\varepsilon}}(P_{m}\in\operatorname{supp}(\mu^{*})+B(0,\varepsilon))\leq 1-\varepsilon$. Define the function $f\colon\Delta(\Delta(\mathcal{K}))\to[0,1]$ by $f(p)=d_{\infty}(p,\operatorname{supp}(\mu^{*}))$, the supremum distance from $\operatorname{supp}(\mu^{*})$. Clearly, $f\in\mathcal{E}_{1}$. Moreover,

[TABLE]

which is a contradiction for $\varepsilon<1$. Thus, we have proven (1).

Take $P^{*}\in\operatorname{argmin}_{p\in\operatorname{supp}(\mu^{*})}\left\|p-P_{m_{\varepsilon}}\right\|_{1}$. By equation (1), the first property of Lemma 5.3 is satisfied.

For the second property of Lemma 5.3, note that, with probability higher than $1-\varepsilon$,

[TABLE]

On the other hand, taking expectation we get that

[TABLE]

Therefore, we conclude that

[TABLE]

To complete the proof of Lemma 5.3, given $P^{*}$, we need a (pure) strategy $\sigma$ such that

[TABLE]

and such that

[TABLE]

Consider the random process $(Y_{m})_{m\geq 1}$ on $\mathcal{Y}\coloneqq\mathcal{K}\times\mathcal{A}\times\Delta(\mathcal{K})$ defined by $Y_{m}\coloneqq(K_{m},A_{m},P_{m})$. We claim that, under $\sigma^{*}$, the process $(Y_{m})_{m\geq 1}$ is a Markov chain. Indeed, given $m\geq 1$ and $(Y_{1},\ldots,Y_{m})\in\mathcal{Y}^{m}$, $Y_{m+1}$ is generated by the following procedure:

Draw a pair $(K_{m+1},S_{m})$ according to $q(K_{m},A_{m})$, 2. 2.

Compute $P_{m+1}$ using Bayes rule according to $P_{m}$ and $S_{m}$, 3. 3.

Draw the next action $A_{m+1}$ according to $\sigma^{*}(P_{m+1})$.

By construction, the law of $Y_{m+1}$ depends only on $Y_{m}$ and therefore $(Y_{m})_{m\geq 1}$ is a Markov chain.

Define $\nu^{*}\in\Delta(\mathcal{Y})$ by fixing the third marginal to $\mu^{*}$ and for all $p\in\Delta(\mathcal{K})$, $\nu^{*}(\cdot\mid p)\in\Delta(\mathcal{K}\times\mathcal{A})$ is $p\otimes\sigma^{*}(p)$. We claim that $\nu^{*}$ is an invariant measure for $(Y_{m})_{m\geq 1}$. Indeed, fixing $\sigma^{*}$ as the strategy for the player, if $P_{1}$ is drawn according to $\mu^{*}$, then, since $\mu^{*}$ is $\sigma^{*}$-invariant, the third marginal of $Y_{m}$ follows $\mu^{*}$, for all $m\geq 1$. Moreover, conditional on $P_{m}$, the random variables $K_{m}$ and $A_{m}$ are independent: the conditional distribution of $K_{m}$ is $P_{m}$ and the one of $A_{m}$ is $\sigma^{*}(P_{m})$. Thus, $\nu^{*}$ is an invariant measure of $(Y_{m})_{m\geq 1}$.

The strategy $\sigma^{*}\colon\Delta(\mathcal{K})\to\Delta(\mathcal{A})$ is a (stationary) mixed strategy, and we are looking for a deterministic strategy $\sigma\in\Sigma$. To derandomize this strategy, note that $\sigma^{*}$ starting from any $p\in\Delta(\mathcal{K})$ is strategically equivalent to a $p$-dependent element of $\Delta(\Sigma)$, that is, a distribution over pure (not necessarily stationary) strategies (Kuhn’s theorem, see [11]). To simplify notations, we still denote this equivalent strategy $\sigma^{*}$, and omit its dependence in $p$.

Define $f\colon\mathcal{Y}\to[0,1]$ by $f(k,a,p)\coloneqq g(k,a)$, a measurable function. Applying an ergodic theorem in Hernández-Lerma and Lasserre [15, Theorem 2.5.1, page 37], we know that $\exists f^{*}$ integrable with respect to $\mu^{*}$ such that, for all $p\in\operatorname{supp}(\mu^{*})$ and $\sigma\in\operatorname{supp}(\sigma^{*})$,

[TABLE]

where $f^{*}$ satisfies that $\int_{\Delta(\mathcal{K})}f^{*}(y)\nu^{*}(dy)=\int_{\Delta(\mathcal{K})}f(y)\nu^{*}(dy)$.

We claim that, for all $p\in\operatorname{supp}(\mu^{*})$ and $\sigma\in\operatorname{supp}(\sigma^{*})$, we have that, for all $k\in\operatorname{supp}(p)$,

[TABLE]

where $a_{1}$ is the first action according to $\sigma$ (formally, $a_{1}=\sigma(\emptyset)=\sigma^{*}(p)$).

Indeed, take $p\in\operatorname{supp}(\mu^{*})$, $\sigma\in\operatorname{supp}(\sigma^{*})$ and $k\in\operatorname{supp}(p)$, then

[TABLE]

Since $\operatorname{supp}(P^{*})\subseteq\{k\in\mathcal{K}:\exists p\in\operatorname{supp}(\mu^{*})\text{ s.t. }k\in\operatorname{supp}(p)\}$, property (2) is satisfied.

Let us now turn to property (3). We claim that, $\mu^{*}-a.s.$ and $\sigma^{*}-a.s.$,

[TABLE]

Indeed, note that

[TABLE]

By definition of $v_{\infty}$, for all $\sigma\in\Sigma$, $\gamma^{\cdot}_{\infty}(\sigma)\leq v_{\infty}(\cdot)$. Therefore, by positivity, we can conclude that, $\mu^{*}-a.s.$ and $\sigma^{*}-a.s.$, $\gamma^{p}_{\infty}(\sigma)=v_{\infty}(p)$. Since the support of $P^{*}$ is included in $\operatorname{supp}(\mu^{*})$, we conclude that

[TABLE]

and property (3) is satisfied. ∎