Partial and Conditional Expectations in Markov Decision Processes with Integer Weights

TL;DR

This paper studies two variants of stochastic shortest path problems in Markov decision processes with integer weights, providing polynomial algorithms for finiteness checks, existence of optimal deterministic schedulers, and approximation methods.

Contribution

It introduces polynomial-time algorithms for finiteness checks and shows the existence of optimal deterministic schedulers in MDPs with arbitrary integer weights.

Findings

Polynomial-time algorithms for finiteness of expectations

Existence of optimal deterministic schedulers with infinite memory

Approximation of optimal values within epsilon in exponential time

Abstract

The paper addresses two variants of the stochastic shortest path problem ('optimize the accumulated weight until reaching a goal state') in Markov decision processes (MDPs) with integer weights. The first variant optimizes partial expected accumulated weights, where paths not leading to a goal state are assigned weight 0, while the second variant considers conditional expected accumulated weights, where the probability mass is redistributed to paths reaching the goal. Both variants constitute useful approaches to the analysis of systems without guarantees on the occurrence of an event of interest (reaching a goal state), but have only been studied in structures with non-negative weights. Our main results are as follows. There are polynomial-time algorithms to check the finiteness of the supremum of the partial or conditional expectations in MDPs with arbitrary integer weights. If finite, then optimal weight-based deterministic schedulers exist. In contrast to the setting of non-negative weights, optimal schedulers can need infinite memory and their value can be irrational. However, the optimal value can be approximated up to an absolute error of $\epsilon$ in time exponential in the size of the MDP and polynomial in $\log(1/\epsilon)$.

Equations

$${\oplus }^{b}goal(\zeta )=\{\begin{array}{ll}wgt(\zeta )+b& \text{if }\zeta \models ◊goal\text{,}\\ 0& \text{if }\zeta /\models ◊goal\text{.}\end{array}$$

$$x_{goal}=x_{fail}=0,$$

$$x_s=wgt(s,\alpha )\cdot p_s+\sum _{t}P(s,\alpha ,t)\cdot x_t\text{ for }s\in S\setminus \{goal,fail\}.$$

$${\mathfrak{p}}_R:=\sup \left\{\frac{{PE}_{s,\alpha }^{\mathfrak{Max}}-{PE}_s^{\mathfrak{Max}}}{p_s^{\max }-p_{s,\alpha }^{\max }}|s\in S,\alpha \in Act(s)\setminus Ac{t}^{\max }(s)\right\}-R$$

$$\text{For }r\ge {\mathfrak{p}}_R:x_{s,r}=p_s^{\max }\cdot (r+R)+E_s^{\mathfrak{Max}},$$

$$\text{for }r<{\mathfrak{p}}_R\text{ and }\alpha \in Act(s):$$

$$x_{s,r}\ge P(s,\alpha ,goal)\cdot (r+R+wgt(s,\alpha ))+\sum _{t\in S^{\prime }}P(s,\alpha ,t)\cdot x_{t,r+wgt(s,\alpha ).}$$

$$\frac{1}{2^{k+1}}\left(r-k\right)+\sum _{i=1}^k\frac{1}{2^i}(r-i)=\frac{1}{2^{k+1}}+\sum _{i=1}^{k+1}\frac{1}{2^i}(r-i)=\frac{k-r+4}{2^{k+1}}+r-2.$$

$$\mathfrak{T}(\pi )=\{\begin{array}{ll}\mathfrak{S}(\pi )& \text{if any prefix }{\pi }^{\prime }\text{ of }\pi \text{ satisfies }{R}_{ϵ}^{-}\le wgt({\pi }^{\prime })\le R_{ϵ}^{+},\\ \mathfrak{Max}(\pi )& \text{if the shortest prefix }{\pi }^{\prime }\text{ of }\pi \text{ with }wgt({\pi }^{\prime })\notin [R_{ϵ}^{-},R_{ϵ}^{+}]\\ & \text{satisfies }wgt({\pi }^{\prime })>R_{ϵ}^{+},\\ \mathfrak{Min}(\pi )& \text{otherwise,}\end{array}$$

$$\underset{s}{\max }{\mathrm{Pr}}_{MEC(\mathcal{M}),s}^{\max }(◊wgt>M\cdot |S|)\le 1-{\delta }^{|S|}.$$

$$\mathfrak{Q}\uparrow \pi (\rho ):=\mathfrak{Q}(\pi ;\rho )$$

$$\frac{wgt(\pi )+{PE}_{last(\pi )}^{\mathfrak{S}\uparrow \pi }}{{\mathrm{Pr}}_{last(\pi )}^{\mathfrak{T}\uparrow \pi }(◊goal)}\le c.$$

$${CE}_{s_{init}}^{\mathfrak{S}}=\frac{{\sum }_{\pi \in \Pi }{\mathrm{Pr}}_{s_{init}}^{\mathfrak{S}}(\pi )\cdot (wgt(\pi )+{PE}_{last(\pi )}^{\mathfrak{S}\uparrow \pi })}{{\sum }_{\pi \in \Pi }{\mathrm{Pr}}_{s_{init}}^{\mathfrak{S}}(\pi )\cdot {\mathrm{Pr}}_{last(\pi )}^{\mathfrak{S}\uparrow \pi }(◊goal)}\le c$$

$$F:=\underset{s\in S}{\max }{PE}_s^{\sup }$$

$$E:={\min }_{s\in S}{PE}_s^{\mathfrak{Max}}.$$

$$\delta :={\min }_{s\in S,\alpha \in Act(s)\setminus Ac{t}^{\max }(s)}{p}_s^{\max }-p_{s,\alpha }^{\max },$$

$$\mathfrak{q}:=\frac{F-E}{\delta }-R$$

$$\mathfrak{S}{⊲}_x\mathfrak{T}(\pi ):=\{\begin{array}{ll}\mathfrak{S}(\pi )& \text{if }wgt(\pi )\le x,\\ \mathfrak{T}(\pi )& \text{otherwise.}\end{array}$$

$$\mathfrak{S}\uparrow \pi (\rho ):=\mathfrak{S}(\pi ;\rho )$$

$${PE}_{last(\pi )}^{\mathfrak{S}\uparrow \pi }[R]+{\mathrm{Pr}}_{last(\pi )}^{\mathfrak{S}\uparrow \pi }(◊goal)\cdot wgt(\pi )>{PE}_{last(\pi )}^{\mathfrak{Max}}[R]+p_{last(\pi )}^{\max }\cdot wgt(\pi ).$$

$${PE}_{last(\pi )}^{\mathfrak{S}\uparrow \pi }[R]+{\mathrm{Pr}}_{last(\pi )}^{\mathfrak{S}\uparrow \pi }(◊goal)\cdot wgt(\pi )$$

$$p_{s,\alpha }^{\max }:=\sum _{t\in S}\mathrm{Pr}(s,\alpha ,t)\cdot p_t^{\max },$$

$${PE}_{s,\alpha }^{\mathfrak{Max}}:=p_{s,\alpha }^{\max }\cdot wgt(s,\alpha )+\sum _{t\in S}\mathrm{Pr}(s,\alpha ,t)\cdot {PE}_t^{\mathfrak{Max}}.$$

$${\mathfrak{p}}_R:=\sup \left\{\frac{{PE}_{s,\alpha }^{\mathfrak{Max}}-E_s^{\mathfrak{Max}}}{p_s^{\max }-p_{s,\alpha }^{\max }}|s\in S,\alpha \in Act(s)\setminus Ac{t}^{\max }(s)\right\}-R$$

$${\mathbb{E}}_s^{\mathfrak{Max}}({\oplus }^{{\mathfrak{p}}_R+R}goal)\ge {\mathbb{E}}_s^{\mathfrak{S}}({\oplus }^{{\mathfrak{p}}_R+R}goal)$$

$${\mathbb{E}}_s^{\mathfrak{Max}}({\oplus }^{{\mathfrak{p}}_R+R}goal)<{\mathbb{E}}_s^{\mathfrak{S}}({\oplus }^{{\mathfrak{p}}_R+R}goal)$$

$$D_s:={\sup }_{\mathfrak{T}}{\mathbb{E}}_s^{\mathfrak{T}}({\oplus }^{{\mathfrak{p}}_R+R}goal)-{\mathbb{E}}_s^{\mathfrak{Max}}({\oplus }^{{\mathfrak{p}}_R+R}goal)$$

$${\mathbb{E}}_t^{\mathfrak{S}}({\oplus }^{{\mathfrak{p}}_R+R}goal)$$

$$0$$

$$\text{For }r\ge {\mathfrak{p}}_R:x_{s,r}=p_s^{\max }\cdot (r+R)+E_s^{\mathfrak{Max}},$$

$$\text{for }r<{\mathfrak{p}}_R\text{ and }\alpha \in Act(s):$$

$$x_{s,r}\ge P(s,\alpha ,goal)\cdot (r+R+wgt(s,\alpha ))+\sum _{t\in S^{\prime }}P(s,\alpha ,t)\cdot x_{t,r+wgt(s,\alpha ).}$$

$$T^{R}_{s,r}(x):=\begin{cases}\max\limits_{\alpha\in Act(s)}\Big{(}\sum_{t\in S^{\prime}}P(s,\alpha,t)\cdot x_{t,r+wgt(s,\alpha)}&\\ \hskip 42.67912pt+P(s,\alpha,\mathit{goal})\cdot(r+R+wgt(s,\alpha))\Big{)}&\text{if }r<\mathfrak{p}_{R},\\ p_{s}^{\max}\cdot(r+R)+E^{\mathfrak{Max}}_{s}&\text{otherwise.}\end{cases}$$

Full text

1 Introduction

Stochastic shortest path (SSP) problems generalize the shortest path problem on graphs with weighted edges. The SSP problem is formalized using finite state Markov decision processes (MDPs), which are a prominent model combining probabilistic and nondeterministic choices. In each state of an MDP, one is allowed to choose nondeterministically from a set of actions, each of them is augmented with probability distributions over the successor states and a weight (cost or reward). The SSP problem asks for a policy to choose actions (here called a scheduler) maximizing or minimizing the expected accumulated weight until reaching a goal state. In the classical setting, one seeks an optimal proper scheduler where proper means that a goal state is reached almost surely. Polynomial-time solutions exist exploiting the fact that optimal memoryless deterministic schedulers exist (provided the optimal value is finite) and can be computed using linear programming techniques, possibly in combination with model transformations (see [5, 10, 1]). The restriction to proper schedulers, however, is often too restrictive. First, there are models that have no proper scheduler. Second, even if proper schedulers exist, the expectation of the accumulated weight of schedulers missing the goal with a positive probability should be taken into account as well. Important such applications include the semantics of probabilistic programs (see e.g. [12, 14, 4, 7, 16]) where no guarantee for almost sure termination can be given and the analysis of program properties at termination time gives rise to stochastic shortest (longest) path problems in which the goal (halting configuration) is not reached almost surely. Other examples are the fault-tolerance analysis (e.g., expected costs of repair mechanisms) in selected error scenarios that can appear with some positive, but small probability or the trade-off analysis with conjunctions of utility and cost constraints that are achievable with positive probability, but not almost surely (see e.g. [2]).

This motivates the switch to variants of classical SSP problems where the restriction to proper schedulers is relaxed. One option (e.g., considered in [8]) is to seek a scheduler optimizing the expectation of the random variable that assigns weight [math] to all paths not reaching the goal and the accumulated weight of the shortest prefix reaching the goal to all other paths. We refer to this expectation as partial expectation. Second, we consider the conditional expectation of the accumulated weight until reaching the goal under the condition that the goal is reached. In general, partial expectations describe situations in which some reward (positive and negative) is accumulated but only retrieved if a certain goal is met. In particular, partial expectations can be an appropriate replacement for the classical expected weight before reaching the goal if we want to include schedulers which miss the goal with some (possibly very small) probability. In contrast to conditional expectations, the resulting scheduler still has an incentive to reach the goal with a high probability, while schedulers maximizing the conditional expectation might reach the goal with a very small positive probability.

Previous work on partial or conditional expected accumulated weights was restricted to the case of non-negative weights. More precisely, partial expectations have been studied in the setting of stochastic multiplayer games with non-negative weights [8]. Conditional expectations in MDPs with non-negative weights have been addressed in [3]. In both cases, optimal values are achieved by weight-based deterministic schedulers that depend on the current state and the weight that has been accumulated so far, while memoryless schedulers are not sufficient. Both [8] and [3] prove the existence of a saturation point for the accumulated weight from which on optimal schedulers behave memoryless and maximize the probability to reach a goal state. This yields exponential-time algorithms for computing optimal schedulers using an iterative linear programming approach. Moreover, [3] proves that the threshold problem for conditional expectations (“does there exist a scheduler $\mathfrak{S}$ such that the conditional expectation under $\mathfrak{S}$ exceeds a given threshold?”) is PSPACE-hard even for acyclic MDPs.

The purpose of the paper is to study partial and conditional expected accumulated weights for MDPs with integer weights. The switch from non-negative to integer weights indeed causes several additional difficulties. We start with the following observation. While optimal partial or conditional expectations in non-negative MDPs are rational, they can be irrational in the general setting:

Example 1**.**

Consider the MDP $\mathcal{M}$ depicted on the left in Figure 1. In the initial state $s_{\mathit{init}}$, two actions are enabled. Action $\tau$ leads to $\mathit{goal}$ with probability $1$ and weight [math]. Action $\sigma$ leads to the states $s$ and $t$ with probability $1/2$ from where we will return to $s_{\mathit{init}}$ with weight $-2$ or $+1$, respectively. The scheduler choosing $\tau$ immediately leads to an expected weight of [math] and is optimal among schedulers reaching the goal almost surely. As long as we choose $\sigma$ in $s_{\mathit{init}}$, the accumulated weight follows an asymmetric random walk increasing by $1$ or decreasing by $2$ with probability $1/2$ before we return to $s_{\mathit{init}}$. It is well known that the probability to ever reach accumulated weight $+1$ in this random walk is $1/\Phi$ where $\Phi=\frac{1+\sqrt{5}}{2}$ is the golden ratio. Likewise, ever reaching accumulated weight $n$ has probability $1/\Phi^{n}$ for all $n\in\mathbb{N}$. Consider the scheduler $\mathfrak{S}_{k}$ choosing $\tau$ as soon as the accumulated weight reaches $k$ in $s_{\mathit{init}}$. Its partial expectation is $k/\Phi^{k}$ as the paths which never reach weight $k$ are assigned weight [math]. The maximum is reached at $k=2$. In Section 4, we prove that there are optimal schedulers whose decisions only depend on the current state and the weight accumulated so far. With this result we can conclude that the maximal partial expectation is indeed $2/\Phi^{2}$, an irrational number.

The conditional expectation of $\mathfrak{S}_{k}$ in $\mathcal{M}$ is $k$ as $\mathfrak{S}_{k}$ reaches the goal with accumulated weight $k$ if it reaches the goal. So, the conditional expectation is not bounded. If we add a new initial state making sure that the goal is reached with positive probability as in the MDP $\mathcal{N}$, we can obtain an irrational maximal conditional expectation as well: The scheduler $\mathfrak{T}_{k}$ choosing $\tau$ in $c$ as soon as the weight reaches $k$ has conditional expectation $\frac{k/2\Phi^{k}}{1/2+1/2\Phi^{k}}$. The maximum is obtained for $k=3$; the maximal conditional expectation is $\frac{3/\Phi^{3}}{1+1/\Phi^{3}}=\frac{3}{3+\sqrt{5}}$.

Moreover, while the proposed algorithms of [8, 3] crucially rely on the monotonicity of the accumulated weights along the prefixes of paths, the accumulated weights of prefixes of path can oscillate when there are positive and negative weights. As we will see later, this implies that the existence of saturation points is no longer ensured and optimal schedulers might require infinite memory (more precisely, a counter for the accumulated weight). These observations provide evidence why linear-programming techniques as used in the case of non-negative MDPs [8, 3] cannot be expected to be applicable for the general setting.

Contributions.

We study the problem of maximizing the partial and conditional expected accumulated weight in MDPs with integer weights. Our first result is that the finiteness of the supremum of partial and conditional expectations in MDPs with integer weights can be checked in polynomial time (Section 3). For both variants we show that there are optimal weight-based deterministic schedulers if the supremum is finite (Section 4). Although the suprema might be irrational and optimal schedulers might need infinite memory, the suprema can be $\epsilon$-approximated in time exponential in the size of the MDP and polynomial in $\log(1/\epsilon)$ (Section 5). By duality of maximal and minimal expectations, analogous results hold for the problem of minimizing the partial or conditional expected accumulated weight. (Note that we can multiply all weights by ${-}1$ and then apply the results for maximal partial resp. conditional expectations.)

Related work.

Closest to our contribution is the above mentioned work on partial expected accumulated weights in stochastic multiplayer games with non-negative weights in [8] and on computation schemes for maximal conditional expected accumulated weights in non-negative MDPs [3]. Conditional expected termination time in probabilistic push-down automata has been studied in [11], which can be seen as analogous considerations for a class of infinite-state Markov chains with non-negative weights. The recent work on notions of conditional value at risk in MDPs [15] also studies conditional expectations, but the considered random variables are limit averages and a notion of (non-accumulated) weight-bounded reachability.

2 Preliminaries

We give basic definitions and present our notation. More details can be found in textbooks, e.g. [17].

Notations for Markov decision processes.

A Markov decision process (MDP) is a tuple $\mathcal{M}=(S,\mathit{Act},P,s_{\mathit{\scriptscriptstyle init}},\mathit{wgt})$ where $S$ is a finite set of states, $\mathit{Act}$ a finite set of actions, $s_{\mathit{\scriptscriptstyle init}}\in S$ the initial state, $P:S\times\mathit{Act}\times S\to[0,1]\cap\mathbb{Q}$ is the transition probability function and $\mathit{wgt}:S\times\mathit{Act}\to\mathbb{Z}$ the weight function. We require that $\sum_{t\in S}P(s,\alpha,t)\in\{0,1\}$ for all $(s,\alpha)\in S\times\mathit{Act}$. We write $\mathit{Act}(s)$ for the set of actions that are enabled in $s$, i.e., $\alpha\in\mathit{Act}(s)$ iff $\sum_{t\in S}P(s,\alpha,t)=1$. We assume that $\mathit{Act}(s)$ is non-empty for all $s$ and that all states are reachable from $s_{\mathit{init}}$. We call a state absorbing if the only enabled action leads to the state itself with probability $1$ and weight [math]. The paths of $\mathcal{M}$ are finite or infinite sequences $s_{0}\,\alpha_{0}\,s_{1}\,\alpha_{1}\,s_{2}\,\alpha_{2}\ldots$ where states and actions alternate such that $P(s_{i},\alpha_{i},s_{i+1})>0$ for all $i\geq 0$. If $\pi=s_{0}\,\alpha_{0}\,s_{1}\,\alpha_{1}\,\ldots\alpha_{k-1}\,s_{k}$ is finite, then $\mathit{wgt}(\pi)=\mathit{wgt}(s_{0},\alpha_{0})+\ldots+\mathit{wgt}(s_{k-1},\alpha_{k-1})$ denotes the accumulated weight of $\pi$, $P(\pi)=P(s_{0},\alpha_{0},s_{1})\cdot\ldots\cdot P(s_{k-1},\alpha_{k-1},s_{k})$ its probability, and $\mathit{last}(\pi)=s_{k}$ its last state. The size of $\mathcal{M}$, denoted $\mathit{size}(\mathcal{M})$, is the sum of the number of states plus the total sum of the logarithmic lengths of the non-zero probability values $P(s,\alpha,s^{\prime})$ as fractions of co-prime integers and the weight values $\mathit{wgt}(s,\alpha)$.

Scheduler.

A (history-dependent, randomized) scheduler for $\mathcal{M}$ is a function $\mathfrak{S}$ that assigns to each finite path $\pi$ a probability distribution over $\mathit{Act}(\mathit{last}(\pi))$. $\mathfrak{S}$ is called memoryless if $\mathfrak{S}(\pi)=\mathfrak{S}(\pi^{\prime})$ for all finite paths $\pi$, $\pi^{\prime}$ with $\mathit{last}(\pi)=\mathit{last}(\pi^{\prime})$, in which case $\mathfrak{S}$ can be viewed as a function that assigns to each state $s$ a distribution over $\mathit{Act}(s)$. $\mathfrak{S}$ is called deterministic if $\mathfrak{S}(\pi)$ is a Dirac distribution for each path $\pi$, in which case $\mathfrak{S}$ can be viewed as a function that assigns an action to each finite path $\pi$. Scheduler $\mathfrak{S}$ is said to be weight-based if $\mathfrak{S}(\pi)=\mathfrak{S}(\pi^{\prime})$ for all finite paths $\pi$, $\pi^{\prime}$ with $\mathit{wgt}(\pi)=\mathit{wgt}(\pi^{\prime})$ and $\mathit{last}(\pi)=\mathit{last}(\pi^{\prime})$. Thus, deterministic weight-based schedulers can be viewed as functions that assign actions to state-weight-pairs. By $\mathit{HR}^{\mathcal{M}}$ we denote the class of all schedulers, by $\mathit{WR}^{\mathcal{M}}$ the class of weight-based schedulers, by $\mathit{WD}^{\mathcal{M}}$ the class of weight-based, deterministic schedulers, and by $\mathit{MD}^{\mathcal{M}}$ the class of memoryless deterministic schedulers. Given a scheduler $\mathfrak{S}$, $\varsigma\,=\,s_{0}\,\alpha_{0}\,s_{1}\,\alpha_{1}\ldots$ is a $\mathfrak{S}$-path iff $\varsigma$ is a path and $\mathfrak{S}(s_{0}\,\alpha_{0}\,s_{1}\,\alpha_{1}\ldots\alpha_{k-1}\,s_{k})(\alpha_{k})>0$ for all $k\geq 0$.

Probability measure.

We write $\mathrm{Pr}^{\mathfrak{S}}_{\mathcal{M},s}$ or briefly $\mathrm{Pr}^{\mathfrak{S}}_{s}$ to denote the probability measure induced by $\mathfrak{S}$ and $s$. For details, see [17]. We will use LTL-like formulas to denote measurable sets of paths and also write $\Diamond(wgt\bowtie x)$ to describe the set of infinite paths having a prefix $\pi$ with $wgt(\pi)\bowtie x$ for $x\in\mathbb{Z}$ and $\bowtie\ \in\{<,\leq,=,\geq,>\}$. Given a measurable set $\psi$ of infinite paths, we define $\mathrm{Pr}^{\min}_{\mathcal{M},s}(\psi)=\inf_{\mathfrak{S}}\mathrm{Pr}^{\mathfrak{S}}_{\mathcal{M},s}(\psi)$ and $\mathrm{Pr}^{\max}_{\mathcal{M},s}(\psi)=\sup_{\mathfrak{S}}\mathrm{Pr}^{\mathfrak{S}}_{\mathcal{M},s}(\psi)$ where $\mathfrak{S}$ ranges over all schedulers for $\mathcal{M}$. Throughout the paper, we suppose that the given MDP has a designated state $\mathit{goal}$. Then, $p_{s}^{\max}$ and $p_{s}^{\min}$ denote the maximal resp. minimal probability of reaching $\mathit{goal}$ from $s$. That is, $p_{s}^{\max}=\mathrm{sup}_{\mathfrak{S}}\mathrm{Pr}^{\mathfrak{S}}_{s}(\Diamond\mathit{goal})$ and $p_{s}^{\min}=\inf_{\mathfrak{S}}\mathrm{Pr}^{\mathfrak{S}}_{s}(\Diamond\mathit{goal})$. Let $Act^{\max}(s)=\{\alpha\in Act(s)|\sum_{t\in S}P(s,\alpha,t)\cdot p_{t}^{\max}=p_{s}^{\max}\}$, and $Act^{\min}(s)=\{\alpha\in Act(s)|\sum_{t\in S}$ $P(s,\alpha,t)\cdot p_{t}^{\min}=p_{s}^{\min}\}$.

Mean payoff. A well-known measure for the long-run behavior of a scheduler $\mathfrak{S}$ in an MDP $\mathcal{M}$ is the mean payoff. Intuitively, the mean payoff is the amount of weight accumulated per step on average in the long run. Formally, we define the mean payoff as the following random variable on infinite paths $\zeta=s_{0}\alpha_{0}s_{1}\alpha_{1}\dots$: $\mathit{MP}(\zeta):=\liminf\limits_{k\to\infty}\frac{\sum_{i=0}^{k}wgt(s_{i},\alpha_{i})}{k+1}$. The mean payoff of the scheduler $\mathfrak{S}$ starting in $s_{\mathit{init}}$ is then defined as the expected value $\mathbb{E}^{\mathfrak{S}}_{s_{\mathit{init}}}(\mathit{MP})$. The maximal mean payoff is the supremum over all schedulers which is equal to the maximum over all $MD$-schedulers: $\mathbb{E}^{\max}_{s_{\mathit{init}}}(\mathit{MP})=\max_{\mathfrak{S}\in\mathit{MD}}\mathbb{E}^{\mathfrak{S}}_{s_{\mathit{init}}}(\mathit{MP})$. In strongly connected MDPs, the maximal mean payoff does not depend on the initial state.

End components, MEC-quotient.

An end component of $\mathcal{M}$ is a strongly connected sub-MDP. End components can be formalized as pairs $\mathcal{E}=(E,\mathfrak{A})$ where $E$ is a nonempty subset of $S$ and $\mathfrak{A}$ a function that assigns to each state $s\in E$ a nonempty subset of $\mathit{Act}(s)$ such that the graph induced by $\mathcal{E}$ is strongly connected. $\mathcal{E}$ is called maximal if there is no end component $\mathcal{E}^{\prime}=(E^{\prime},\mathfrak{A}^{\prime})$ with $\mathcal{E}\not=\mathcal{E}^{\prime}$, $E\subseteq E^{\prime}$ and $\mathfrak{A}(s)\subseteq\mathfrak{A}^{\prime}(s)$ for all $s\in E$. The MEC-quotient of an MDP $\mathcal{M}$ is the MDP $\mathit{MEC}(\mathcal{M})$ arising from $\mathcal{M}$ by collapsing all states that belong to the same maximal end component $\mathcal{E}$ to a state $s_{\mathcal{E}}$. All actions enabled in some state in $\mathcal{E}$ not belonging to $\mathcal{E}$ are enabled in $s_{\mathcal{E}}$. Details and the formal construction can be found in [9]. We call an end component $\mathcal{E}$ positively weight-divergent if there is a scheduler $\mathfrak{S}$ for $\mathcal{E}$ such that $\mathrm{Pr}^{\mathfrak{S}}_{\mathcal{E},s}(\Diamond(\mathit{wgt}\geq n))=1$ for all $s\in\mathcal{E}$ and $n\in\mathbb{N}$. In [1], it is shown that the existence of positively weight-divergent end components can be decided in polynomial time.

3 Partial and Conditional Expectations in MDPs

We define partial and conditional expectations in MDPs. We extend the definition of [8] by introducing partial expectations with bias which are closely related to conditional expectations. Afterwards, we sketch the computation of maximal partial expectations in MDPs with non-negative weights and in Markov chains.

Partial and conditional expectation. In the sequel, let $\mathcal{M}$ be an MDP with a designated absorbing goal state $\mathit{goal}$. Furthermore, we collapse all states from which $\mathit{goal}$ is not reachable to one absorbing state $\mathit{fail}$. Let $b\in\mathbb{R}$. We define the random variable $\oplus^{b}\mathit{goal}$ on infinite paths $\zeta$ by

[TABLE]

We call the expectation of this random variable under a scheduler $\mathfrak{S}$ the partial expectation with bias $b$ of $\mathfrak{S}$ and write $\mathit{PE}^{\mathfrak{S}}_{\mathcal{M},s_{\mathit{init}}}[b]:=\mathbb{E}^{\mathfrak{S}}_{\mathcal{M},s_{\mathit{init}}}(\oplus^{b}\mathit{goal})$ as well as $\mathit{PE}^{\sup}_{\mathcal{M},s_{\mathit{init}}}[b]:=\sup_{\mathfrak{S}\in\mathit{HR}^{\mathcal{M}}}\mathit{PE}^{\mathfrak{S}}_{\mathcal{M},s_{\mathit{init}}}[b]$. If $b=0$, we sometimes drop the argument $b$; if $\mathcal{M}$ is clear from the context, we drop the subscript. In order to maximize the partial expectation, intuitively one has to find the right balance between reaching $\mathit{goal}$ with high probability and accumulating a high positive amount of weight before reaching $\mathit{goal}$. The bias can be used to shift this balance by additionally rewarding or penalizing a high probability to reach $\mathit{goal}$.

The conditional expectation of $\mathfrak{S}$ is defined as the expectation of $\oplus^{0}\mathit{goal}$ under the condition that $\mathit{goal}$ is reached. It is defined if $\mathrm{Pr}^{\mathfrak{S}}_{\mathcal{M},s_{\mathit{init}}}(\Diamond\mathit{goal})>0$. We write $\mathit{CE}^{\mathfrak{S}}_{\mathcal{M},s_{\mathit{init}}}:=\mathbb{E}^{\mathfrak{S}}_{\mathcal{M},s_{\mathit{init}}}(\oplus^{0}\mathit{goal}|\Diamond\mathit{goal})$ and $\mathit{CE}^{\sup}_{\mathcal{M},s_{\mathit{init}}}=\sup_{\mathfrak{S}}\mathit{CE}^{\mathfrak{S}}_{\mathcal{M},s_{\mathit{init}}}$ where the supremum is taken over all schedulers $\mathfrak{S}$ with $\mathrm{Pr}^{\mathfrak{S}}_{\mathcal{M},s_{\mathit{init}}}(\Diamond\mathit{goal})>0$. We can express the conditional expectation as $\mathit{CE}^{\mathfrak{S}}_{\mathcal{M},s_{\mathit{init}}}={\mathit{PE}^{\mathfrak{S}}_{\mathcal{M},s_{\mathit{init}}}}/{\mathrm{Pr}^{\mathfrak{S}}_{\mathcal{M},s_{\mathit{init}}}(\Diamond\mathit{goal})}$. The following proposition establishes a close connection between conditional expectations and partial expectations with bias.

Proposition 2**.**

Let $\mathcal{M}$ be an MDP, $\mathfrak{S}$ a scheduler with $\mathrm{Pr}^{\mathfrak{S}}_{s_{\mathit{\scriptscriptstyle init}}}(\Diamond\mathit{goal})>0$, $\theta\in\mathbb{Q}$, and $\bowtie\in\{<,\leq,\geq,>\}$. Then we have $\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}[-\theta]\bowtie 0\text{ iff }\mathit{CE}^{\mathfrak{S}}_{s_{\mathit{init}}}\bowtie\theta$. Further, if $\mathrm{Pr}^{\min}_{s_{\mathit{init}}}(\Diamond\mathit{goal})>0$, then $\mathit{PE}^{\sup}_{s_{\mathit{init}}}[-\theta]\bowtie 0\text{ iff }\mathit{CE}^{\sup}_{s_{\mathit{init}}}\bowtie\theta$.

Proof.

The first claim follows from $\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}[-\theta]=\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}[0]-\mathrm{Pr}^{\mathfrak{S}}_{s_{\mathit{init}}}(\Diamond\mathit{goal})\cdot\theta$. The second claim follows by quantification over all schedulers. ∎

In [3], it is shown that deciding whether $\mathit{CE}^{\sup}_{s_{\mathit{init}}}\bowtie\theta$ for $\bowtie\in\{<,\leq,\geq,>\}$ and $\theta\in\mathbb{Q}$ is PSPACE-hard even for acyclic MDPs. We conclude:

Corollary 3**.**

Given an MDP $\mathcal{M}$, $\bowtie\in\{<,\leq,\geq,>\}$, and $\theta\in\mathbb{Q}$, deciding whether $\mathit{PE}^{\sup}_{\mathcal{M},s_{\mathit{init}}}\bowtie\theta$ is PSPACE-hard.

Finiteness.

We present criteria for the finiteness of $\mathit{PE}^{\sup}_{s_{\mathit{init}}}[b]$ and $\mathit{CE}^{\sup}_{s_{\mathit{init}}}$. Detailed proofs can be found in Appendix 0.A.1. By slightly modifying the construction from [1] which removes end components only containing [math]-weight cycles, we obtain the following result.

Proposition 4**.**

Let $\mathcal{M}$ be an MDP which does not contain positively weight-divergent end components and let $b\in\mathbb{Q}$. Then there is a polynomial time transformation to an MDP $\mathcal{N}$ containing all states from $\mathcal{M}$ and possibly an additional absorbing state $\mathit{fail}$ such that

• •

all end components of $\mathcal{N}$ have negative maximal expected mean payoff,

• •

for any scheduler $\mathfrak{S}$ for $\mathcal{M}$ there is a scheduler $\mathfrak{S}^{\prime}$ for $\mathcal{N}$ with $\mathrm{Pr}^{\mathfrak{S}}_{\mathcal{M},s}(\Diamond\mathit{goal})=\mathrm{Pr}^{\mathfrak{S}^{\prime}}_{\mathcal{N},s}(\Diamond\mathit{goal})$ and $\mathit{PE}^{\mathfrak{S}}_{\mathcal{M},s}[b]=\mathit{PE}^{\mathfrak{S}^{\prime}}_{\mathcal{N},s}[b]$ for any state $s$ in $\mathcal{M}$, and vice versa.

Hence, we can restrict ourselves to MDPs in which all end components have negative maximal expected mean payoff if there are no positively weight divergent end components. The following result is now analogous to the result in [1] for the classical SSP problem.

Proposition 5**.**

Let $\mathcal{M}$ be an MDP and $b\in\mathbb{R}$ arbitrary. The optimal partial expectation $\mathit{PE}^{\sup}_{s_{\mathit{init}}}[b]$ is finite if and only if there are no positively weight-divergent end components in $\mathcal{M}$.

To obtain an analogous result for conditional expectations, we observe that the finiteness of the maximal partial expectation is necessary for the finiteness of the maximal conditional expectation. However, this is not sufficient. In [3], a critical scheduler is defined as a scheduler $\mathfrak{S}$ for which there is a path containing a positive cycle and for which $\mathrm{Pr}^{\mathfrak{S}}_{s_{\mathit{init}}}(\Diamond\mathit{goal})=0$. Given a critical scheduler, it is easy to construct a sequence of schedulers with unbounded conditional expectation (see Appendix 0.A.1 and [3]). On the other hand, if $\mathrm{Pr}^{\min}_{\mathcal{M},s_{\mathit{init}}}(\Diamond\mathit{goal})>0$, then $\mathit{CE}^{\sup}_{s_{\mathit{init}}}$ is finite if and only if $\mathit{PE}^{\sup}_{s_{\mathit{init}}}$ is finite. We will show how we can restrict ourselves to this case if there are no critical schedulers:

So, let $\mathcal{M}$ be an MDP with $\mathrm{Pr}^{\min}_{\mathcal{M},s_{\mathit{init}}}(\Diamond\mathit{goal})=0$ and suppose there are no critical schedulers for $\mathcal{M}$. Let $S_{0}$ be the set of all states reachable from $s_{\mathit{init}}$ while only choosing actions in $Act^{\min}$. As there are no critical schedulers, $(S_{0},Act^{\min})$ does not contain positive cycles. So, there is a finite maximal weight $w_{s}$ among paths leading from $s_{\mathit{init}}$ to $s$ in $S_{0}$. Consider the following MDP $\mathcal{N}$: It contains the MDP $\mathcal{M}$ and a new initial state $t_{\mathit{init}}$. For each $s\in S_{0}$ and each $\alpha\in Act(s)\setminus Act^{\min}(s)$, $\mathcal{N}$ also contains a new state $t_{s,\alpha}$ which is reachable from $t_{\mathit{init}}$ via an action $\beta_{s,\alpha}$ with weight $w_{s}$ and probability $1$. In $t_{s,\alpha}$, only action $\alpha$ with the same probability distribution over successors and the same weight as in $s$ is enabled. So in $\mathcal{N}$, one has to decide immediately in which state to leave $S_{0}$ and one accumulates the maximal weight which can be accumulated in $\mathcal{M}$ to reach this state in $S_{0}$. In this way, we ensure that $\mathrm{Pr}^{\min}_{\mathcal{N},t_{\mathit{init}}}(\Diamond\mathit{goal})>0$.

Proposition 6**.**

The constructed MDP $\mathcal{N}$ satisfies $\mathit{CE}^{\sup}_{\mathcal{N},t_{\mathit{init}}}=\mathit{CE}^{\sup}_{\mathcal{M},s_{\mathit{init}}}$.

We can rely on this reduction to an MDP in which $\mathit{goal}$ is reached with positive probability for $\epsilon$-approximations and the exact computation of the optimal conditional expectation. In particular, the values $w_{s}$ for $s\in S_{0}$ are easy to compute by classical shortest path algorithms on weighted graphs. Furthermore, we can now decide the finiteness of the maximal conditional expectation.

Proposition 7**.**

For an arbitrary MDP $\mathcal{M}$, $\mathit{CE}^{\sup}_{\mathcal{M},s_{\mathit{init}}}$ is finite if and only if there are no positively weight-divergent end components and no critical schedulers.

Partial and conditional expectations in Markov chains.

Markov chains with integer weights can be seen as MDPs with only one action $\alpha$ enabled in every state. Consequently, there is only one scheduler for a Markov chain. Hence, we drop the superscripts in $p^{\max}$ and $\mathit{PE}^{\sup}$.

Proposition 8**.**

The partial and conditional expectation in a Markov chain $\mathcal{C}$ are computable in polynomial time.

Proof.

Let $\alpha$ be the only action available in $\mathcal{C}$. Assume that all states from which $\mathit{goal}$ is not reachable have been collapsed to an absorbing state $\mathit{fail}$. Then $\mathit{PE}_{\mathcal{C},s_{\mathit{\scriptscriptstyle init}}}$ is the value of $x_{s_{\mathit{\scriptscriptstyle init}}}$ in the unique solution to the following system of linear equations with one variable $x_{s}$ for each state $s$:

[TABLE]

The existence of a unique solution follows from the fact that $\{\mathit{goal}\}$ and $\{\mathit{fail}\}$ are the only end components (see [17]). It is straight-forward to check that $(\mathit{PE}_{\mathcal{C},s})_{s\in S}$ is this unique solution. The conditional expectation is obtained from the partial expectation by dividing by the probability $p_{s_{\mathit{init}}}$ to reach the goal. ∎

This result can be seen as a special case of the following result. Restricting ourselves to schedulers which reach the goal with maximal or minimal probability in an MDP without positively weight-divergent end components, linear programming allows us to compute the following two memoryless deterministic schedulers (see [8, 3]).

Proposition 9**.**

Let $\mathcal{M}$ be an MDP without positively weight-divergent end components. There is a scheduler $\mathfrak{Max}\in\mathit{MD}^{\mathcal{M}}$ such that for each $s\in S$ we have $\mathrm{Pr}^{\mathfrak{Max}}_{s}(\Diamond\mathit{goal})=p_{s}^{{\max}}$ and $\mathit{PE}^{\mathfrak{Max}}_{s}=\mathrm{sup}_{\mathfrak{S}}\mathit{PE}^{\mathfrak{S}}_{s}$ where the supremum is taken over all schedulers $\mathfrak{S}$ with $\mathrm{Pr}^{\mathfrak{S}}_{s}(\Diamond\mathit{goal})=p_{s}^{{\max}}$. Similarly, there is a scheduler $\mathfrak{Min}\in\mathit{MD}^{\mathcal{M}}$ maximizing the partial expectation among all schedulers reaching the goal with minimal probability. Both these schedulers and their partial expectations are computable in polynomial time.

These schedulers will play a crucial role for the approximation of the maximal partial expectation and the exact computation of maximal partial expectations in MDPs with non-negative weights.

Partial expectations in MDPs with non-negative weights.

In [8], the computation of maximal partial expectations in stochastic multiplayer games with non-negative weights is presented. We adapt this approach to MDPs with non-negative weights. A key result is the existence of a saturation point, a bound on the accumulated weight above which optimal schedulers do not need memory.

In the sequel, let $R\in\mathbb{Q}$ be arbitrary, let $\mathcal{M}$ be an MDP with non-negative weights, $\mathit{PE}^{\sup}_{s_{\mathit{init}}}<\infty$, and assume that end components have negative maximal mean payoff (see Proposition 4). A saturation point for bias $R$ is a natural number $\mathfrak{p}$ such that there is a scheduler $\mathfrak{S}$ with $\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}[R]=\mathit{PE}^{{\sup}}_{s_{\mathit{init}}}[R]$ which is memoryless and deterministic as soon as the accumulated weight reaches $\mathfrak{p}$. I.e. for any two paths $\pi$ and $\pi^{\prime}$, with $last(\pi)=last(\pi^{\prime})$ and $wgt(\pi),wgt(\pi^{\prime})>\mathfrak{p}$, $\mathfrak{S}(\pi)=\mathfrak{S}(\pi^{\prime})$.

Transferring the idea behind the saturation point for conditional expectations given in [3], we provide the following saturation point which can be considerably smaller than the saturation point given in [8] in stochastic multiplayer games. Detailed proofs to this section are given in Appendix 0.A.2.

Proposition 10**.**

We define $p_{s,\alpha}^{\max}:=\sum_{t\in S}P(s,\alpha,t)\cdot p_{t}^{\max}$ and $\mathit{PE}^{\mathfrak{Max}}_{s,\alpha}:=p_{s,\alpha}^{\max}\cdot wgt(s,\alpha)+\sum_{t\in S}P(s,\alpha,t)\cdot\mathit{PE}^{\mathfrak{Max}}_{t}$. Then,

[TABLE]

is an upper saturation point for bias $R$ in $\mathcal{M}$.

The saturation point $\mathfrak{p}_{R}$ is chosen such that, as soon as the accumulated weight exceeds $\mathfrak{p}_{R}$, the scheduler $\mathfrak{Max}$ is better than any scheduler deviating from $\mathfrak{Max}$ for only one step. So, the proposition states that $\mathfrak{Max}$ is then also better than any other scheduler.

As all values involved in the computation can be determined by linear programming, the saturation point $\mathfrak{p}_{R}$ is computable in polynomial time. This also means that the logarithmic length of $\mathfrak{p}_{R}$ is polynomial in the size of $\mathcal{M}$ and hence $\mathfrak{p}_{R}$ itself is at most exponential in the size of $\mathcal{M}$.

Proposition 11**.**

Let $R\in\mathbb{Q}$ and let $B_{R}$ be the least integer greater or equal to $\mathfrak{p}_{R}+\max_{s\in S,\alpha\in Act(s)}wgt(s,\alpha)$ and let $S^{\prime}:=S\setminus\{\mathit{goal},fail\}$. The values $(\mathit{PE}^{\sup}_{s_{\mathit{init}}}[r{+}R])_{s\in S^{\prime},0\leq r\leq B_{R}}$ form the unique solution to the following linear program in the variables $(x_{s,r})_{s\in S^{\prime},0\leq r\leq B_{R}}$ (r ranges over integers):

Minimize $\sum_{s\in S^{\prime},0\leq r\leq B_{R}}x_{s,r}$ under the following constraints:

[TABLE]

From a solution $x$ to the linear program, we can easily extract an optimal weight-based deterministic scheduler. This scheduler only needs finite memory because the accumulated weight increases monotonically along paths and as soon as the saturation point is reached $\mathfrak{Max}$ provides the optimal decisions. As $B_{R}$ is exponential in the size of $\mathcal{M}$, the computation of the optimal partial expectation via this linear program runs in time exponential in the size of $\mathcal{M}$.

4 Existence of Optimal Schedulers

We prove that there are optimal weight-based deterministic schedulers for partial and conditional expectations. After showing that, if finite, $\mathit{PE}^{\sup}_{s_{\mathit{init}}}$ is equal to $\sup_{\mathfrak{S}\in\mathit{WD}^{\mathcal{M}}}\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}$, we take an analytic approach to show that there is indeed a weight-based deterministic scheduler maximizing the partial expectation. We define a metric on $\mathit{WD}^{\mathcal{M}}$ turning it into a compact space. Then, we prove that the function assigning the partial expectation to schedulers is upper semi-continuous. We conclude that there is a weight-based deterministic scheduler obtaining the maximum. Proofs to this section can be found in Appendix 0.B.

Proposition 12**.**

Let $\mathcal{M}$ be an MDP with $\mathit{PE}^{\sup}_{s_{\mathit{init}}}<\infty$. Then we have $\mathit{PE}^{\sup}_{s_{\mathit{init}}}=\sup_{\mathfrak{S}\in\mathit{WD}^{\mathcal{M}}}\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}$.

Proof sketch.

We can assume that all end components have negative maximal expected mean payoff (see Proposition 4). Given a scheduler $\mathfrak{S}\in\mathit{HR}^{\mathcal{M}}$, we take the expected number of times $\theta_{s,w}$ that $s$ is visited with accumulated weight $w$ under $\mathfrak{S}$ for each state-weight pair $(s,w)$, and the expected number of times $\theta_{s,w,\alpha}$ that $\mathfrak{S}$ then chooses $\alpha$. These values are finite due to the negative maximal mean payoff in end components. We define the scheduler $\mathfrak{T}\in\mathit{WR}^{\mathcal{M}}$ choosing $\alpha$ in $s$ with probability $\theta_{s,w,\alpha}/\theta_{s,w}$ when weight $w$ has been accumulated. Then, we show by standard arguments that we can replace all probability distributions that $\mathfrak{T}$ chooses by Dirac distributions to obtain a scheduler $\mathfrak{T}^{\prime}\in\mathit{WD}^{\mathcal{M}}$ such that $\mathit{PE}^{\mathfrak{T}^{\prime}}_{s_{\mathit{init}}}\geq\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}$. ∎

It remains to show that the supremum is obtained by a weight-based deterministic scheduler. Given an MDP $\mathcal{M}$ with arbitrary integer weights, we define the following metric $d^{\mathcal{M}}$ on the set of weight-based deterministic schedulers, i.e. on the set of functions from $S\times\mathbb{Z}\to Act$: For two such schedulers $\mathfrak{S}$ and $\mathfrak{T}$, we let $d^{\mathcal{M}}(\mathfrak{S},\mathfrak{T}):=2^{-R}$ where $R$ is the greatest natural number such that $\mathfrak{S}\upharpoonright S\times\{-(R{-}1),\dots,R{-}1\}=\mathfrak{T}~{}\upharpoonright~{}S\times\{-(R{-}1),\dots,R{-}1\}$ or $\infty$ if there is no greatest such natural number.

Lemma 13**.**

The metric space $(Act^{S\times\mathbb{Z}},d^{\mathcal{M}})$ is compact.

Having defined this compact space of schedulers, we can rely on the analytic notion of upper semi-continuity.

Lemma 14** (Upper Semi-Continuity of Partial Expectations).**

If $\mathit{PE}^{\sup}_{s_{\mathit{init}}}$ is finite in $\mathcal{M}$, then the function $\mathit{PE}:(\mathit{WD},d^{\mathit{WD}})\to(\mathbb{R}_{\infty},d^{euclid})$ assigning $\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}$ to a weight-based deterministic scheduler $\mathfrak{S}$ is upper semi-continuous.

The technical proof of this lemma can be found in Appendix 0.B. We arrive at the main result of this section.

Theorem 15** (Existence of Optimal Schedulers for Partial Expectations).**

If $\mathit{PE}^{\sup}_{s_{\mathit{init}}}$ is finite in an MDP $\mathcal{M}$, then there is a weight-based deterministic scheduler $\mathfrak{S}$ with $\mathit{PE}^{\sup}_{s_{\mathit{init}}}=\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}$.

Proof.

If $\mathit{PE}^{\sup}_{s_{\mathit{init}}}$ is finite, then the map $\mathit{PE}:(\mathit{WD},d^{\mathit{WD}})\to(\mathbb{R}_{\infty},d^{euclid})$ is upper semi-continuous. So, this map has a maximum because $(\mathit{WD},d^{\mathit{WD}})$ is a compact metric space. ∎

Corollary 16** (Existence of Optimal Schedulers for Conditional Expectations).**

If $\mathit{CE}^{\sup}_{s_{\mathit{init}}}$ is finite in an MDP $\mathcal{M}$, then there is a weight-based deterministic scheduler $\mathfrak{S}$ with $\mathit{CE}^{\sup}_{s_{\mathit{init}}}=\mathit{CE}^{\mathfrak{S}}_{s_{\mathit{init}}}$.

Proof.

By Proposition 6, we can assume that $\mathrm{Pr}^{\min}_{s_{\mathit{init}}}(\Diamond\mathit{goal})>0$. We know that $\mathit{PE}^{\sup}_{s_{\mathit{init}}}[-\mathit{CE}^{\sup}_{s_{\mathit{init}}}]=0$ and that there is a weight-based deterministic scheduler $\mathfrak{S}$ with $\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}[-\mathit{CE}^{\sup}_{s_{\mathit{init}}}]=0$. By Proposition 2, $\mathfrak{S}$ maximizes the conditional expectation as it reaches $\mathit{goal}$ with positive probability. ∎

In MDPs with non-negative weights, the optimal decision in a state $s$ only depends on $s$ as soon as the accumulated weight exceeds a saturation point. In MDPs with arbitrary integer weights, it is possible that the optimal choice of action does not become stable for increasing values of accumulated weight as we see in the following example.

Example 17**.**

Let us first consider the MDP $\mathcal{N}$ depicted in Figure 2. Let $\pi$ be a path reaching $t$ for the first time with accumulated weight $r$. Consider a scheduler which chooses $\beta$ for the first $k$ times and then $\alpha$. In this situation, the partial expectation from this point on is:

[TABLE]

For $r\geq 2$, this partial expectation has its unique maximum for the choice $k=r{-}2$. This already shows that an optimal scheduler needs infinite memory. No matter how much weight $r$ has been accumulated when reaching $t$, the optimal scheduler has to count the $r{-}2$ times it chooses $\beta$.

Furthermore, we can transfer the optimal scheduler for the MDP $\mathcal{N}$ to the MDP $\mathcal{M}$. In state $t$, we have to make a nondeterministic choice between two action leading to the states $q_{0}$ and $q_{1}$, respectively. In both of these states, action $\beta$ is enabled which behaves like the same action in the MDP $\mathcal{N}$ except that it moves between the two states if $\mathit{goal}$ is not reached. So, the action $\alpha$ is only enabled every other step. As in $\mathcal{N}$, we want to choose $\alpha$ after choosing $\beta$ $r{-}2$ times if we arrived in $t$ with accumulated weight $r\geq 2$. So, the choice in $t$ depends on the parity of $r$: For $r=1$ or $r$ even, we choose $\delta$. For odd $r\geq 3$, we choose $\gamma$. This shows that the optimal scheduler in the MDP $\mathcal{M}$ needs specific information about the accumulated weight, in this case the parity, no matter how much weight has been accumulated.

In the example, the optimal scheduler has a periodic behavior when fixing a state and looking at optimal decisions for increasing values of accumulated weight. The question whether an optimal scheduler always has such a periodic behavior remains open.

5 Approximation

As the optimal values for partial and conditional expectation can be irrational, there is no hope to compute these values by linear programming as in the case of non-negative weights. In this section, we show how we can nevertheless approximate the values. The main result is the following.

Theorem 18**.**

Let $\mathcal{M}$ be an MDP with $\mathit{PE}^{\sup}_{\mathcal{M},s_{\mathit{init}}}<\infty$ and $\epsilon>0$. The maximal partial expectation $\mathit{PE}^{\sup}_{\mathcal{M},s_{\mathit{init}}}$ can be approximated up to an absolute error of $\epsilon$ in time exponential in the size of $\mathcal{M}$ and polynomial in $\log(1/\epsilon)$. If further, $\mathit{CE}^{\sup}_{\mathcal{M},s_{\mathit{init}}}<\infty$, also $\mathit{CE}^{\sup}_{\mathcal{M},s_{\mathit{init}}}$ can be approximated up to an absolute error of $\epsilon$ in time exponential in the size of $\mathcal{M}$ and polynomial in $\log(1/\epsilon)$.

We first prove that upper bounds for $\mathit{PE}^{\sup}_{\mathcal{M},s_{\mathit{init}}}$ and $\mathit{CE}^{\sup}_{\mathcal{M},s_{\mathit{init}}}$ can be computed in polynomial time. Then, we show that there are $\epsilon$-optimal schedulers for the partial expectation which become memoryless as soon as the accumulated weight leaves a sufficiently large weight window around [math]. We compute the optimal partial expectation of such a scheduler by linear programming. The result can then be extended to conditional expectations.

Upper Bounds. Let $\mathcal{M}$ be an MDP in which all end components have negative maximal mean payoff. Let $\delta$ be the minimal non-zero transition probability in $\mathcal{M}$ and $W:=\max_{s\in S,\alpha\in Act(s)}|\mathit{wgt}(s,\alpha)|$. Moving through the MEC-quotient, the probability to reach an accumulated weight of $|S|\cdot W$ is bounded by $1-\delta^{|S|}$ as $\mathit{goal}$ or $\mathit{fail}$ is reached within $S$ steps with probability at least $1-\delta^{|S|}$. It remains to show similar bounds inside an end component.

We will use the characterization of the maximal mean payoff in terms of super-harmonic vectors due to Hordijk and Kallenberg [13] to define a supermartingale controlling the growth of the accumulated weight in an end component under any scheduler. As the value vector for the maximal mean payoff in an end component is constant and negative in our case, the results of [13] yield:

Proposition 19** (Hordijk, Kallenberg).**

Let $\mathcal{E}=(S,Act)$ be an end component with maximal mean payoff $-t$ for some $t>0$. Then there is a vector $(u_{s})_{s\in S}$ such that $-t+u_{s}\geq wgt(s,\alpha)+\sum_{s^{\prime}\in S}P(s,\alpha,s^{\prime})\cdot u_{s^{\prime}}$.

Furthermore, let $v$ be the vector (-t,…,-t) in $\mathbb{R}^{S}$. Then, $(v,u)$ is the solution to a linear program with $2|S|$ variables, $2|S||Act|$ inequalities, and coefficients formed from the transition probabilities and weights in $\mathcal{E}$.

We will call the vector $u$ a super-potential because the expected accumulated weight after $i$ steps is at most $u_{s}-\min_{t\in S}u_{t}-i\cdot t$ when starting in state $s$. Let $\mathfrak{S}$ be a scheduler for $\mathcal{E}$ starting in some state $s$. We define the following random variables on $\mathfrak{S}$-runs in $\mathcal{E}$: let $s(i)\in S$ be the state after $i$ steps, let $\alpha(i)$ be the action chosen after $i$ steps, let $w(i)$ be the accumulated weight after $i$ steps, and let $\pi(i)$ be the history, i.e. the finite path after $i$ steps.

Lemma 20**.**

The sequence $m(i):=w(i)+u_{s(i)}$ satisfies $\mathbb{E}(m(i+1)|\pi(0),\dots,\pi(i))$ $\leq m(i)-t$ for all $i$.111This means that $m(i)+i\cdot t$ is a super-martingale with respect to the history $\pi(i)$.

Proof.

By Proposition 19, $\mathbb{E}(m(i+1)|\pi(0),\dots,\pi(i))-m(i)=wgt(s(i),\mathfrak{S}(\pi(i)))+\sum_{s^{\prime}\in S}P(s(i),\mathfrak{S}(\pi(i)),s^{\prime})\cdot u_{s^{\prime}}-u_{s(i)}\leq-t$. ∎

We are going to apply the following theorem by Blackwell [6].

Theorem 21** (Blackwell [6]).**

Let $X_{1},X_{2},\dots$ be random variables, and let $S_{n}:=\sum_{k=1}^{n}X_{k}$. Assume that $|X_{i}|\leq 1$ for all $i$ and that there is a $u>0$ such that $\mathbb{E}(X_{n+1}|X_{1},\dots,X_{n})\leq-u$. Then, $\mathrm{Pr}(\sup_{n\in\mathbb{N}}S_{n}\geq t)\leq\left(\frac{1-u}{1+u}\right)^{t}$.

We denote $\max_{s^{\prime}\in S}u_{s^{\prime}}-\min_{s^{\prime}\in S}u_{s^{\prime}}$ by $\|u\|$. Observe that $|m(i+1)-m(i)|\leq\|u\|+W=:c_{\mathcal{E}}$. We can rescale the sequence $m(i)$ by defining $m^{\prime}(i):=(m(i)-m(0))/c_{\mathcal{E}}$. This ensures that $m^{\prime}(0)=0$, $|m^{\prime}(i+1)-m^{\prime}(i)|\leq 1$ and $\mathbb{E}(m^{\prime}(i+1)|m^{\prime}(0),\dots,m^{\prime}(i))\leq-t/c_{\mathcal{E}}$ for all $i$. In this way, we arrive at the following conclusion, putting $\lambda_{\mathcal{E}}:=\frac{1-t/c_{\mathcal{E}}}{1+t/c_{\mathcal{E}}}$.

Corollary 22**.**

For any scheduler $\mathfrak{S}$ and any starting state $s$ in $\mathcal{E}$, we have $\mathrm{Pr}^{\mathfrak{S}}_{s}(\Diamond wgt\geq(k{+}1)\cdot c_{\mathcal{E}})\leq\lambda_{\mathcal{E}}^{k}$.

Proof.

By Theorem 21, $\mathrm{Pr}^{\mathfrak{S}}_{s}(\Diamond wgt\geq(k+1)\cdot c_{\mathcal{E}})\leq\mathrm{Pr}^{\mathfrak{S}}_{s}(\Diamond wgt\geq\|u\|+k\cdot c_{\mathcal{E}})\leq\mathrm{Pr}^{\mathfrak{S}}_{s}(\exists i:m(i)-m(0)\geq k\cdot c_{\mathcal{E}})=\mathrm{Pr}^{\mathfrak{S}}_{s}(\sup_{i\in\mathbb{N}}m^{\prime}(i)\geq k)\leq\left(\frac{1-t/c_{\mathcal{E}}}{1+t/c_{\mathcal{E}}}\right)^{k}.$ ∎

Let $\mathit{MEC}$ be the set of maximal end components in $\mathcal{M}$. For each $\mathcal{E}\in\mathit{MEC}$, let $\lambda_{\mathcal{E}}$ and $c_{\mathcal{E}}$ be as in Corollary 22. Define $\lambda_{\mathcal{M}}:=1-(\delta^{|S|}\cdot\prod_{\mathcal{E}\in\mathit{MEC}}(1-\lambda_{\mathcal{E}}))$, and $c_{\mathcal{M}}:=|S|\cdot W+\sum_{\mathcal{E}\in\mathit{MEC}}c_{\mathcal{E}}$. Then an accumulated weight of $c_{\mathcal{M}}$ cannot be reached with a probability greater than $\lambda_{\mathcal{M}}$ because reaching accumulated weight $c_{\mathcal{M}}$ would require reaching weight $c_{\mathcal{E}}$ in some end component $\mathcal{E}$ or reaching weight $|S|\cdot W$ in the MEC-quotient and $1-\lambda_{\mathcal{M}}$ is a lower bound on the probability that none of this happens (under any scheduler).

Proposition 23**.**

Let $\mathcal{M}$ be an MDP with $\mathit{PE}^{\sup}_{s_{\mathit{init}}}<\infty$. There is an upper bound $\mathit{PE}^{ub}$ for the partial expectation in $\mathcal{M}$ computable in polynomial time.

Proof.

In any end component $\mathcal{E}$, the maximal mean payoff $-t$ and the super-potential $u$ are computable in polynomial time. Hence, $c_{\mathcal{E}}$ and $\lambda_{\mathcal{E}}$, and in turn also $c_{\mathcal{M}}$ and $\lambda_{\mathcal{M}}$ are also computable in polynomial time. When we reach accumulated weight $c_{\mathcal{M}}$ for the first time, the actual accumulated weight is at most $c_{\mathcal{M}}+W$. So, we conclude that $\mathrm{Pr}^{\max}_{s}(\Diamond wgt\geq k\cdot(c_{\mathcal{M}}+W))\leq\lambda_{\mathcal{M}}^{k}$. The partial expectation can now be bounded by $\sum_{k=0}^{\infty}(k+1)\cdot(c_{\mathcal{M}}+W)\cdot\lambda_{\mathcal{M}}^{k}=\frac{c_{\mathcal{M}}+W}{(1-\lambda_{\mathcal{M}})^{2}}$. ∎

Corollary 24**.**

Let $\mathcal{M}$ be an MDP with $\mathit{CE}^{\sup}_{\mathcal{M},s_{\mathit{init}}}<\infty$. There is an upper bound $\mathit{CE}^{ub}$ for the conditional expectation in $\mathcal{M}$ computable in polynomial time.

Proof.

By Proposition 6, we can construct an MDP $\mathcal{N}$ in which $\mathit{goal}$ is reached with probability $q>0$ in polynomial time with $\mathit{CE}^{\sup}_{\mathcal{M},s_{\mathit{init}}}=\mathit{CE}^{\sup}_{\mathcal{N},s_{\mathit{init}}}$. Now, $\mathit{CE}^{ub}:=\mathit{PE}^{ub}/q$ is an upper bound for the conditional expectation in $\mathcal{M}$. ∎

Approximating optimal partial expectations.

The idea for the approximation is to assume that the partial expectation is $\mathit{PE}^{\mathfrak{Max}}_{s_{\mathit{init}}}+w\cdot p^{\max}_{s}$ if a high weight $w$ has been accumulated in state $s$. Similarly, for small weights $w^{\prime}$, we use the value $\mathit{PE}^{\mathfrak{Min}}_{s_{\mathit{init}}}+w\cdot p^{\min}_{s}$. We will first provide a lower “saturation point” making sure that only actions minimizing the probability to reach the goal are used by an optimal scheduler as soon as the accumulated weight drops below this saturation point. Proofs to this section can be found in Appendix 0.C.1

Proposition 25**.**

Let $\mathcal{M}$ be an MDP with $\mathit{PE}^{\sup}_{s_{\mathit{init}}}<\infty$. Let $s\in S$ and let $\mathfrak{q}_{s}:=\frac{\mathit{PE}^{ub}-\mathit{PE}^{\mathfrak{Min}}_{s}}{p_{s}^{\min}-\min\limits_{\alpha\not\in Act^{\min}(s)}p_{s,\alpha}^{\min}}$. Then any weight-based deterministic scheduler $\mathfrak{S}$ maximizing the partial expectation in $\mathcal{M}$ satisfies $\mathfrak{S}(s,w)\in Act^{\min}(s)$ if $w\leq\mathfrak{q}_{s}$.

Let $\mathfrak{q}:=\min_{s\in S}\mathfrak{q}_{s}$ and let $D:=\mathit{PE}^{ub}-\min\{\mathit{PE}^{\mathfrak{Max}}_{s},\mathit{PE}^{\mathfrak{Min}}_{s}|s\in S\}$. Given $\epsilon>0$, we define $R^{+}_{\epsilon}:=(c_{\mathcal{M}}+W)\cdot\left\lceil\frac{\log(2D)+\log(1/\epsilon)}{\log(1/\lambda_{\mathcal{M}})}\right\rceil$ and $R^{-}_{\epsilon}:=\mathfrak{q}-R^{+}_{\epsilon}$.

Theorem 26**.**

There is a weight-based deterministic scheduler $\mathfrak{S}$ such that the scheduler $\mathfrak{T}$ defined by

[TABLE]

satisfies $\mathit{PE}^{\mathfrak{T}}_{s_{\mathit{init}}}\geq\mathit{PE}^{\sup}_{s_{\mathit{init}}}-\epsilon$.

This result now allows us to compute an $\epsilon$-approximation and an $\epsilon$-optimal scheduler with finite memory by linear programming, similar to the case of non-negative weights, in a linear program with $R^{+}_{\epsilon}+R^{-}_{\epsilon}$ many variables and $|Act|$-times as many inequalities.

Corollary 27**.**

$\mathit{PE}^{\sup}_{s_{\mathit{init}}}$* can be approximated up to an absolute error of $\epsilon$ in time exponential in the size of $\mathcal{M}$ and polynomial in $\log(1/\epsilon)$.*

If the logarithmic length of $\theta\in\mathbb{Q}$ is polynomial in the size of $\mathcal{M}$, we can also approximate $\mathit{PE}^{\sup}_{s_{\mathit{init}}}[\theta]$ up to an absolute error of $\epsilon$ in time exponential in the size of $\mathcal{M}$ and polynomial in $\log(1/\epsilon)$: We can add a new initial state $s$ with a transition to $s_{\mathit{init}}$ with weight $\theta$ and approximate $\mathit{PE}^{\sup}_{s}$ in the new MDP.

Transfer to conditional expectations.

Let $\mathcal{M}$ be an MDP with $\mathit{CE}^{sup}_{s_{\mathit{init}}}<\infty$ and $\epsilon>0$. By Proposition 6, we can assume that $\mathrm{Pr}^{\min}_{\mathcal{M},s_{\mathit{init}}}(\Diamond\mathit{goal})=:p$ is positive. Clearly, $\mathit{CE}^{sup}_{s_{\mathit{init}}}\in[\mathit{CE}^{\mathfrak{Max}}_{s_{\mathit{init}}},\mathit{CE}^{ub}]$. We perform a binary search to approximate $\mathit{CE}^{sup}_{s_{\mathit{init}}}$: We put $A_{0}:=\mathit{CE}^{\mathfrak{Max}}_{s_{\mathit{init}}}$ and $B_{0}:=\mathit{CE}^{ub}$. Given $A_{i}$ and $B_{i}$, let $\theta_{i}:=(A_{i}+B_{i})/2$. Then, we approximate $\mathit{PE}^{\sup}_{s_{\mathit{init}}}[-\theta_{i}]$ up to an absolute error of $p\cdot\epsilon$. Let $E_{i}$ be the value of this approximation. If $E_{i}\in[-2p\cdot\epsilon,2p\cdot\epsilon]$, terminate and return $\theta_{i}$ as the approximation for $\mathit{CE}^{\sup}_{s_{\mathit{init}}}$. If $E_{i}<-2p\cdot\epsilon$, put $A_{i+1}:=A_{i}$ and $B_{i+1}:=\theta_{i}$, and repeat. If $E_{i}>2p\cdot\epsilon$, put $A_{i+1}:=\theta_{i}$ and $B_{i+1}:=B_{i}$, and repeat.

Proposition 28**.**

The procedure terminates after at most $\lceil\log((A_{0}-B_{0})/(p\cdot\epsilon))\rceil$ iterations and returns an $3\epsilon$-approximation of $\mathit{CE}^{\sup}_{s_{\mathit{init}}}$ in time exponential in the size of $\mathcal{M}$ and polynomial in $\log(1/\epsilon)$.

The proof can be found in Appendix 0.C.2. This finishes the proof of Theorem 18.

6 Conclusion

Compared to the setting of non-negative weights, the optimization of partial and conditional expectations faces substantial new difficulties in the setting of integer weights. The optimal values can be irrational showing that the linear programming approaches from the setting of non-negative weights cannot be applied for the computation of optimal values. We showed that this approach can nevertheless be adapted for approximation algorithms. Further, we were able to show that there are optimal weight-based deterministic schedulers. These schedulers, however, can require infinite memory and it remains open whether we can further restrict the class of schedulers necessary for the optimization. In examples, we have seen that optimal schedulers can switch periodically between actions they choose for increasing values of accumulated weight. Further insights on the behavior of optimal schedulers would be helpful to address threshold problems (“Is $\mathit{PE}^{\sup}_{s_{\mathit{init}}}\geq\theta$?”).

Appendix

Appendix 0.A Partial and Conditional Expectations in Markov Decision Processes

In this section, we give proofs to the claims of Section 3.

0.A.1 Finiteness and Preprocessing

The finiteness of maximal partial expectations depends on the existence of positively weight-divergent end components. Using the construction from [1] which removes end components only containing [math]-weight cycles, we can show the following: See 4

Proof.

In an end component which has non-negative maximal expected mean payoff and which is not positively weight-divergent, all cycles have weight [math] (see [1]). We will use the so-called spider construction from [1] with a small modification to remove such end components: So, let $\mathcal{M}$ be an MDP and let $\mathcal{E}=(E,\mathfrak{A})$ be an end component of $\mathcal{M}$ in which all cycles have weight [math]. The spider construction successively flattens sub-end components which contain exactly one action per state: So, let $E^{\prime}\subseteq E$ and for each $s\in E^{\prime}$, let $\alpha_{s}\in\mathfrak{A}(s)$ such that $\mathcal{E}^{\prime}=\{(s,\alpha_{s})|s\in E^{\prime}\}$ is an end component. We pick a state $s_{0}\in E^{\prime}$. As all cycles in $\mathcal{E}^{\prime}$ have weight [math], there is a unique weight $w_{s}$ for each $s\in E^{\prime}$ such that all paths from $s$ to $s_{0}$ in $\mathcal{E}^{\prime}$ have weight $w_{s}$. The spider construction now does the following:

Disable the action $\alpha_{s}$ in $s$ for all $s\in E^{\prime}$. 2. 2.

Enable a new action $\tau$ in $s$ with $P(s,\tau,s_{0})=1$ and $\mathit{wgt}(s,\tau)=w_{s}$ for all $s\not=s_{0}$ in $E^{\prime}$. 3. 3.

For each state $s\not=s_{0}$ in $E^{\prime}$ and each action $\beta\in\mathit{Act}_{\mathcal{M}}(s)\setminus\{\alpha_{s}\}$, disable $\beta$ in $s$ and instead enable a new action $\beta_{s}$ in $s_{0}$ with $P(s_{0},\beta_{s},t)=P(s,\beta,t)$ for all states $t$ in $\mathcal{M}$ and $\mathit{wgt}(s_{0},\beta_{s})=\mathit{wgt}(s,\beta)-w_{s}$.

We extend the construction by adding an absorbing state $\mathit{fail}$ and additionally enabling a new action $\tau$ in $s_{0}$ with $P(s_{0},\tau,\mathit{fail})=1$ and $\mathit{wgt}(s_{0},\tau)=0$. We call the resulting MDP after one application of the construction $\mathcal{N}^{\prime}$. In [1], it is shown that polynomially many applications (in polynomial time in total) of the construction result in an MDP $\mathcal{N}$ satisfying the first requirement in the statement.

Hence, it is sufficient to show the correspondence between schedulers claimed in the second requirement for the MDPs $\mathcal{M}$ and $\mathcal{N}^{\prime}$. Given a scheduler $\mathfrak{S}$ for $\mathcal{M}$, we construct the following scheduler $\mathfrak{S}^{\prime}$ for $\mathcal{N}^{\prime}$: Whenever a run in $\mathcal{M}$ under $\mathfrak{S}$ reaches $\mathcal{E}^{\prime}$, let $p_{\mathit{fail}}$ be the probability that $\mathfrak{S}$ will never leave $\mathcal{E}^{\prime}$ again. Further, for each state $s$ in $\mathcal{E}^{\prime}$ and each action $\beta\in Act(s)$ not belonging to $\mathcal{E}$, let $p_{s,\beta}$ be the probability that $\mathfrak{S}$ leaves $\mathcal{E}^{\prime}$ from $s$ via $\beta$. This behavior can now be mimicked in $\mathcal{N}^{\prime}$: $\mathfrak{S}^{\prime}$ goes to $\mathit{fail}$ with probability $p_{\mathit{fail}}$ and takes the action $\beta_{s}$ in $s_{0}$ with probability $p_{s,\beta}$. It is straightforward to check that this does not affect the partial expectation or the probability to reach $\mathit{goal}$.

Conversely, a scheduler $\mathfrak{S}^{\prime}$ for $\mathcal{N}^{\prime}$ can easily be transformed to a scheduler $\mathfrak{S}$ for $\mathcal{M}$: Whenever $\mathfrak{S}^{\prime}$ moves to $\mathit{fail}$ from $s_{0}$, the scheduler $\mathfrak{S}$ stays in $\mathcal{E}^{\prime}$ forever. If $\mathfrak{S}^{\prime}$ chooses $\beta_{s}$ in $s_{0}$, $\mathfrak{S}$ moves through $\mathcal{E}^{\prime}$ until it reaches $s$. This happens almost surely. Then, $\mathfrak{S}$ chooses $\beta$. Again, it is easy to check that the partial expectation and the probability to reach $\mathit{goal}$ are preserved. ∎

See 5

Proof.

Suppose there is a positively weight-divergent end component $\mathcal{E}$. Since $\mathcal{E}$ is reachable and we can accumulated arbitrarily high weights inside $\mathcal{E}$ with probability $1$, we can easily construct a sequence of schedulers whose partial expectation diverges to $+\infty$ by letting the schedulers stay in a positively weight divergent end component until an arbitrarily high weight has been accumulated, before they try to reach the goal.

Now, suppose that there are no positively weight-divergent end components. We can assume that all end components have negative maximal mean payoff (see Proposition 4). We claim that there is a natural number $W$ such that $\max_{s}\mathrm{Pr}^{\max}_{s}(\Diamond wgt\geq W):=p<1$. Let $M:=\max_{s,\alpha}|wgt(s,\alpha)|$. Then, the claim follows as follows: For all $n\in\mathbb{N}$ we get that $\max_{s\in S}\mathrm{Pr}^{\max}_{\mathcal{M},s}(\Diamond wgt\geq n\cdot W+M)\leq p^{n}$. Then the partial expectation of any scheduler is bounded by $\sum_{n=0}^{\infty}(n+1)\cdot W\cdot p^{n}=\frac{W}{(1-p)^{2}}$.

For each end component $\mathcal{E}$, there is a number $W_{\mathcal{E}}$ and a probability $p_{\mathcal{E}}$ such that in $\mathcal{E}$ we have $\max_{s\in\mathcal{E}}\mathrm{Pr}^{\max}_{\mathcal{E},s}(\Diamond wgt\geq W_{\mathcal{E}}):=p_{\mathcal{E}}<1$. On the other hand, in the MEC-quotient of $\mathcal{M}$ the probability to reach $\mathit{goal}$ or $fail$ in $|S|$ steps is at least $\delta^{|S|}$ where $\delta$ is the minimal transition probability. Then we can conclude that

[TABLE]

All in all, it is impossible for a scheduler to almost surely reach an accumulated weight above $M\cdot|S|+\sum_{\mathcal{E}\text{ is an end component}}W_{\mathcal{E}}$. ∎

Recall that we define a critical scheduler to be a scheduler $\mathfrak{S}$, for which there is a path containing a positive cycle, and for which $\mathrm{Pr}^{\mathfrak{S}}_{s_{\mathit{init}}}(\Diamond\mathit{goal})=0$

Proposition 29**.**

Let $\mathcal{M}$ be an MDP. The optimal conditional expectation $\mathit{CE}^{\sup}_{s_{\mathit{init}}}=\infty$ if $\mathit{PE}^{\sup}_{s_{\mathit{init}}}=\infty$ or if there is a critical scheduler $\mathfrak{S}$.

Proof.

If $\mathit{PE}^{\sup}_{s_{\mathit{init}}}=\infty$ clearly also $\mathit{CE}^{\sup}_{s_{\mathit{init}}}=\infty$. So, let $\mathfrak{S}$ be a scheduler which can reach a positive cycle but almost surely does not reach $\mathit{goal}$. Then, for any $n$, we can construct the following scheduler $\mathfrak{S}_{n}$. The scheduler $\mathfrak{S}_{n}$ attempts to reach the positive cycle directly, i.e. without visiting a state twice before. Then, it attempts to take the cycle $n$ times in a row. Only if $\mathfrak{S}_{n}$ succeeds to do so, it maximizes the probability to reach the goal from then on. Otherwise, it avoids the $\mathit{goal}$. The scheduler $\mathfrak{S}_{n}$ reaches the goal with positive probability and $\mathit{CE}^{\mathfrak{S}_{n}}_{s_{\mathit{init}}}\to\infty$ for $n\to\infty$. ∎

In Section 3, we gave the following construction: Let $\mathcal{M}$ be an MDP with $\mathrm{Pr}^{\min}_{\mathcal{M},s_{\mathit{init}}}(\Diamond\mathit{goal})=0$ and $\mathit{CE}^{\sup}_{\mathcal{M},s_{\mathit{init}}}<\infty$. In particular, this means that there are no critical schedulers for $\mathcal{M}$. Let $S_{0}$ be the set of all states reachable from $s_{\mathit{init}}$ while only choosing actions in $Act^{\min}$. As there are no critical schedulers, $(S_{0},Act^{\min})$ does not contain positive cycles. So, there is a unique maximal weight $w_{s}$ of paths leading from $s_{\mathit{init}}$ to $s$ in $S_{0}$. Consider the following MDP $\mathcal{N}$: It contains the MDP $\mathcal{M}$ and a new initial state $t_{\mathit{init}}$. For each $s\in S_{0}$ and each $\alpha\in Act(s)\setminus Act^{\min}(s)$, $\mathcal{N}$ also contains a new state $t_{s,\alpha}$ which is reachable from $t_{\mathit{init}}$ via an action $\beta_{s,\alpha}$ with weight $w_{s}$ and probability $1$. In $t_{s,\alpha}$, only action $\alpha$ with the same probability distribution over successors and the same weight as in $s$ is enabled. In this way, we ensure that $\mathrm{Pr}^{\min}_{\mathcal{N},t_{\mathit{init}}}(\Diamond\mathit{goal})>0$.

See 6

Proof.

For each pair $(s,\alpha)$ with $s\in S_{0}$ and $\alpha\in Act(s)\setminus Act^{\min}(s)$, let $c_{s,\alpha}:=\sup_{\mathfrak{S}}\mathit{CE}^{\mathfrak{S}}_{\mathcal{N},t_{\mathit{init}}}$ where the supremum is taken over all schedulers $\mathfrak{S}$ for $\mathcal{N}$ which assign probability $1$ to the action $\beta_{s,\alpha}$ in $t_{\mathit{init}}$. Then, $\mathit{CE}^{\sup}_{\mathcal{N},t_{\mathit{init}}}=\max_{s,\alpha}c_{s,\alpha}=:c$.

A scheduler reaching the $\mathit{goal}$ with positive probability has to choose an action not in $Act^{\min}$ after at least one path. Let $s\in S_{0}$ and $\alpha\in Act(t)\setminus Act^{\min}(s)$ be such that $c=c_{s,\alpha}$. For any scheduler $\mathfrak{T}$ for $\mathcal{N}$ starting with $\beta_{s,\alpha}$, we define the following scheduler $\mathfrak{T}^{\prime}$: $\mathfrak{T}^{\prime}$ starts by following a path with maximal accumulated weight from $s_{\mathit{init}}$ to $s$. If it reaches $s$ with accumulated weight $w_{s}$ it chooses $\alpha$ and follows the choices of $\mathfrak{T}$ from then on. If it does not reach $s$ with accumulated weight $w_{s}$, $\mathfrak{T}^{\prime}$ just picks actions in $Act^{\min}$, in this way making sure that the $\mathit{goal}$ will not be reached. In this way, $\mathit{CE}^{\mathfrak{T}^{\prime}}_{\mathcal{M},s_{\mathit{init}}}=\mathit{CE}^{\mathfrak{T}}_{\mathcal{N},t_{\mathit{init}}}$. So, $\mathit{CE}^{\sup}_{\mathcal{M},s_{\mathit{init}}}\geq c$.

Before we show the other direction, we define, given a finite path $\pi$, a finite path $\rho$ starting in $last(\pi)$, and a scheduler $\mathfrak{Q}$, the scheduler $\mathfrak{Q}\uparrow\pi$ by

[TABLE]

where $\pi;\rho$ denotes the concatenation of the paths $\pi$ and $\rho$.

To show that for any scheduler $\mathfrak{S}$ for $\mathcal{M}$ with $\mathrm{Pr}^{\mathfrak{S}}_{s_{\mathit{init}}}>0$ we have $\mathit{CE}^{\mathfrak{S}}_{\mathcal{M},s_{\mathit{init}}}\leq c$, let $\mathfrak{S}$ be such a scheduler and consider the set $\Pi$ of finite $\mathfrak{S}$-paths $\pi$ in $\mathcal{M}^{\min}$ such that $\mathfrak{S}(\pi)\in Act(last(\pi))\setminus Act^{\min}$. We know that for each $\pi\in\Pi$,

[TABLE]

We conclude that also

[TABLE]

as all denominators are positive. ∎

See 7

Proof.

We have seen that $\mathit{CE}^{\sup}_{s_{\mathit{init}}}=\infty$ if there is a positively weight-divergent end component or a critical scheduler. On the other hand, we can rely on the reduction from the previous proposition if there are no critical scheduler. In $\mathcal{N}$, the maximal partial expectation is finite as there are no positively weight-divergent end components. As the minimal probability to reach $\mathit{goal}$ is furthermore positive, the maximal conditional expectation is finite as well. Hence, $\mathit{CE}^{\sup}_{\mathcal{M},s_{\mathit{init}}}=\mathit{CE}^{\sup}_{\mathcal{N},t_{\mathit{init}}}<\infty$. ∎

0.A.2 Partial Expectations in MDPs with Non-Negative Weights

Let $R\in\mathbb{Q}$ be arbitrary. In this section, we consider an MDP $\mathcal{M}$ in which all weights are non-negative, and we assume:

$\mathit{PE}^{\sup}_{s_{init}}<\infty$, 2. 2.

the only end components are the two distinct absorbing states $\mathit{goal}$ and $fail$, 3. 3.

$\mathit{goal}$ can be reached from any state $s\in S\setminus\{fail\}$.

Assumption 2. is justified as all weight are non-negative and hence the maximal expected mean payoff of an end component cannot be negative.

Saturation point.

Recall that a saturation point for bias $R$ is a natural number $\mathfrak{p}$ such that there is a scheduler $\mathfrak{S}$ with $\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}[R]=\mathit{PE}^{{\sup}}_{s_{\mathit{init}}}[R]$ which is memoryless and deterministic as soon as the accumulated weight reaches $\mathfrak{p}$. I.e. for any two paths $\pi$ and $\pi^{\prime}$, with $last(\pi)=last(\pi^{\prime})$ and $wgt(\pi),wgt(\pi^{\prime})>\mathfrak{p}$, $\mathfrak{S}(\pi)=\mathfrak{S}(\pi^{\prime})$. We first provide the following saturation point which we need in the proof of the smaller saturation point given in Section 3.

Proposition 30**.**

Let

[TABLE]

and

[TABLE]

Further, let

[TABLE]

where $p_{s,\alpha}^{\max}:=\sum_{t\in S}P(s,\alpha,t)\cdot p_{t}^{\max}$. Then,

[TABLE]

is a saturation point for bias $R$.

(If the minimum in the definition of $\delta$ is taken over an empty set, $\mathfrak{Max}$ is already the optimal scheduler and hence any value is an upper saturation point.)

Proof.

Given two schedulers $\mathfrak{S}$ and $\mathfrak{T}$ and some $x\in\mathbb{R}$, we define the scheduler $\mathfrak{S}\lhd_{x}\mathfrak{T}$ via:

[TABLE]

Given a finite path $\pi$ and a path $\rho$ starting in $last(\pi)$, we further define the scheduler $\mathfrak{S}\uparrow\pi$ by

[TABLE]

where $\pi;\rho$ denotes the concatenation of the paths $\pi$ and $\rho$.

Suppose, there is a scheduler $\mathfrak{S}$ such that $\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}[R]>\mathit{PE}^{\mathfrak{S}\lhd_{\mathfrak{q}}\mathfrak{Max}}_{s_{\mathit{init}}}[R]$. By Proposition 9, this means that there must be a path $\pi$ with $wgt(\pi)\geq\mathfrak{q}$ such that $\mathfrak{S}(\pi)\not\in Act^{\max}(last(\pi))$ and

[TABLE]

But on the other hand, we have

[TABLE]

contradicting the supposition. For the third line, note that $wgt(\pi)+R>0$. ∎

In the setting of non-negative weights, it has been shown in [8] that there is a weight-based deterministic scheduler maximizing the partial expectation. Of course, this also follows from our results in Section 4 for MDPs with arbitrary weights. This allows us to conclude the following.

Corollary 31**.**

The supremum in $\mathit{PE}^{{\max},R}_{s}:=\mathrm{sup}_{\mathfrak{S}}\mathit{PE}^{\mathfrak{S},R}_{s}$ can also be taken over weight-based schedulers which behave memoryless as soon as the accumulated weight reaches $\mathfrak{q}$. As there are only finitely many such schedulers, the supremum is furthermore in fact a maximum.

Proposition 32**.**

Let $\mathcal{M}$ be an MDP with non-negative weights satisfying the assumptions 1–3. Furthermore, let

[TABLE]

and

[TABLE]

Then,

[TABLE]

is an upper saturation point for $\mathcal{M}$.

Proof.

It is enough to show that

[TABLE]

for all states $s$ and all schedulers $\mathfrak{S}$. Since $\mathfrak{Max}$ maximizes the probability of reaching the goal, this implies that $\mathbb{E}^{\mathfrak{Max}}_{s}(\oplus^{w+R}\mathit{goal})\geq\mathbb{E}^{\mathfrak{S}}_{s}(\oplus^{w+R}\mathit{goal})$ for all $w\geq\mathfrak{p}_{R}$.

So, we show (1). Suppose that

[TABLE]

for some state $s$ and some scheduler $\mathfrak{S}$.

Now, let

[TABLE]

for each state $s$. As we know that there are only finitely many relevant schedulers for the supremum we can actually choose a (deterministic weight-based) scheduler $\mathfrak{S}$ such that $\mathrm{sup}_{\mathfrak{T}}\mathbb{E}^{\mathfrak{T}}_{s}(\oplus^{\mathfrak{p}_{R}+R}\mathit{goal})=\mathbb{E}^{\mathfrak{S}}_{s}(\oplus^{\mathfrak{p}_{R}+R}\mathit{goal})$ for all states $s$. Now, let $t$ be a state such that $D_{t}>0$ is maximal, and such that the first action $\alpha$ that $\mathfrak{S}$ chooses starting in $t$ leads to a state $r$ with $D_{r}<D_{t}$ with positive probability. As $D_{\mathit{goal}}=0$ such a state exists. Then,

[TABLE]

Subtracting $\mathbb{E}^{\mathfrak{S}}_{t}(\oplus^{\mathfrak{p}_{R}+R}\mathit{goal})$, we get

[TABLE]

But, the right hand side evaluates to [math] by the definition of $\mathfrak{p}_{R}$ leading to a contradiction. ∎

Computation of the Partial Expectation.

Proposition 33**.**

Let $R\in\mathbb{Q}$ and let $B_{R}$ be the least integer greater or equal to $\mathfrak{p}_{R}+\max_{s\in S,\alpha\in Act(s)}wgt(s,\alpha)$ and let $S^{\prime}:=S\setminus\{\mathit{goal},fail\}$. Consider the following linear program in the variables $(x_{s,r})_{s\in S^{\prime},0\leq r\leq B_{R}}$ (r ranges over integers): Minimize $\sum_{s\in S^{\prime},0\leq r\leq B_{R}}x_{s,r}$ under the following constraints:

[TABLE]

The values $(\mathit{PE}^{\sup}_{s_{\mathit{init}}}[r+R])_{s\in S,0\leq r\leq B_{R}}$ form the unique solution to this linear program.

We prove the unique solvability of this linear program in detail. Linear programs claimed to be uniquely solvable below can be treated analogously.

Proof.

Following a standard approach by Veinott [19], we want to show that the linear program is uniquely solvable by defining a contraction mapping with respect to a weighted maximum norm whose fixed point is the optimal solution. The definition of the weights we use is made explicit by Tseng [18].

We define a function $T^{R}:\mathbb{R}^{S^{\prime}\times\{0,\dots,B_{R}\}}\to\mathbb{R}^{S^{\prime}\times\{0,\dots,B_{R}\}}$. For $s\in S$ and $r\leq B_{R}$, let

[TABLE]

In order to define a suitable weighted maximum norm, we begin by recursively defining the following partition $S_{0},\dots,S_{k}$ of $S$:

[TABLE]

If at some point $S_{0},\dots,S_{i}$ is not yet a partition of $S$, then $S_{i+1}$ is non-empty. If it was empty, then $T:=S\setminus(S_{0}\cup\dots\cup S_{i})$ would contain an end component, as for each $t\in T$ there would be an action $\alpha$ such that $P(t,\alpha,T)=1$. So, the recursive definition produces a partition $S=S_{0}\cup\dots\cup S_{k}$ in finitely many steps.

Now, for each $s\in S_{i}$ let $w_{s}:=1-\delta^{2i}$ where

[TABLE]

These $w_{s}$ will serve as weights for our weighted supremum norm.

We use the following fact [18, Lemma 3]: Let $\gamma:=\frac{1-\delta^{2k-1}}{1-\delta^{2k}}<1$. For all $s\not\in\{gaol,fail\}$ and all $\alpha\in Act(s)$, we have

[TABLE]

We define the following norm on $\mathbb{R}^{S^{\prime}\times\{0,\dots,B_{R}\}}$:

[TABLE]

We show that $T^{R}$ is a contraction with respect to this norm: For $x,y\in\mathbb{R}^{S^{\prime}\times\{0,\dots,B_{R}\}}$, we claim

[TABLE]

Let $s\in S^{\prime}$ and $r<\mathfrak{p}_{R}$.

[TABLE]

By symmetry, we obtain the same inequality for $T^{R}_{s,r}(y)-T^{R}_{s,r}(x)$. Dividing the inequality by $w_{s}$ concludes the proof that $T^{R}$ is a contraction. By the Banach fixed-point theorem, $T^{R}$ hence has a unique fixed point.

That this fixed point is the unique solution of the linear program is now easy to see. The map $T^{R}$ is defined such that any $z$ satisfying the constraints of the linear program satisfies $z\geq T^{R}(z)$. But, if there are coordinates $s,r$ such that $z_{s,r}>T^{R}_{s,r}(z)$ then replacing $z_{s,r}$ by $T^{R}_{s,r}(z)$ leads to $z^{\prime}$ still satisfying the constraints and resulting in a smaller value of $\sum_{s,r}z^{\prime}_{s,r}$. So, the unique fixed point of $T^{R}$ is the unique optimal solution of the linear program.

Finally, we can easily check that $(\mathbb{E}^{\max}_{s}(\oplus^{r+R}\mathit{goal}))_{s\in S,0\leq r\leq B_{R}}$ is indeed a fixed point of $T^{R}$ as we know that $\mathfrak{p}_{R}$ is a saturation point. ∎

Appendix 0.B Existence of Optimal Schedulers

We provide the proofs to Section 4 here.

Recall that we consider an MDP $\mathcal{M}$ with finite maximal partial expectation. In particular, we assume that all states are reachable from $s_{\mathit{init}}$ and that $\mathit{goal}$ is reachable from all states except $fail$. Furthermore, there are no positively weight-divergent end components and so we can assume that all end components have negative maximal expected mean payoff (see Proposition 4).

We split the proof of Proposition 12 into the following two propositions:

Proposition 34**.**

Let $\mathcal{M}$ be an MDP with $\mathit{PE}^{\sup}_{s_{\mathit{init}}}<\infty$. For each scheduler $\mathfrak{S}\in HR^{\mathcal{M}}$, there is a scheduler $\mathfrak{T}\in WR^{\mathcal{M}}$ such that $\mathit{PE}^{\mathfrak{S}}=\mathit{PE}^{\mathfrak{T}}$ and $\mathrm{Pr}^{\mathfrak{S}}_{s_{\mathit{init}}}(\Diamond\mathit{goal})=\mathrm{Pr}^{\mathfrak{T}}_{s_{\mathit{\scriptscriptstyle init}}}(\Diamond\mathit{goal})$.

Proof.

Let $\mathfrak{S}\in HR$. For each state-weight pair $(s,w)$ with $s\in S\setminus\{\mathit{goal},fail\}$ and $w\in\mathbb{Z}$, we let $\theta^{\mathfrak{S}}_{s,w}$ be the expected number of times that $s$ is reached with accumulated weight $w$ under $\mathfrak{S}$, and we let $\theta^{\mathfrak{S}}_{s,w,\alpha}$ be the expected number of times that $\alpha$ is chosen in this situation by $\mathfrak{S}$. We have that

[TABLE]

Note that $\theta^{\mathfrak{S}}_{s,w}$ is finite for all $s\in S\setminus\{\mathit{goal},fail\}$, $w\in\mathbb{Z}$ as all end components have negative maximal expected mean payoff.

Now, we define a weight-based deterministic scheduler $\mathfrak{T}$ by

[TABLE]

Clearly, only state-weight-pairs $(s,w)$ which are reachable under $\mathfrak{S}$ are reachable under $\mathfrak{T}$. Further, $\mathfrak{T}$ is well-defined as $\sum_{\alpha\in Act(s)}\mathfrak{T}(s,w)(\alpha)=1$ for all reachable $(s,w)$.

For each state-weight-pair $(s,w)$ reachable under $\mathfrak{S}$ and hence under $\mathfrak{T}$ let $\theta^{\mathfrak{T}}_{s,w}$ be the expected number of times that $(s,w)$ is reached under $\mathfrak{T}$. Then, the collection of all $\theta^{\mathfrak{T}}_{s,w}$ is uniquely determined by the following set of equations: For all $s,w$,

[TABLE]

where $\delta_{s,w}=1$ iff $s=s_{\mathit{init}}$ and $w=0$, and $\delta_{s,w}=0$ otherwise. By spelling out the last steps of the paths in the definition of $\theta_{s,w}^{\mathfrak{S}}$, one can see that $\theta_{s,w}^{\mathfrak{S}}$ provides the solution to this set of equations and hence $\theta_{s,w}^{\mathfrak{S}}=\theta_{s,w}^{\mathfrak{T}}$ for all $(s,w)$. By the definition of $\mathfrak{T}$, the expected number of times action $\alpha$ is chosen in $(s,w)$ under $\mathfrak{T}$ is hence $\theta_{s,w,\alpha}^{\mathfrak{S}}$ as well and the claim follows. ∎

Proposition 35**.**

Let $\mathcal{M}$ be an MDP with $\mathit{PE}^{\sup}_{s_{\mathit{init}}}<\infty$. Then, we have

[TABLE]

Proof.

Let $\mathfrak{S}$ be a weight-based randomized scheduler and let $(s,w)$ be a state-weight-pair such that $\mathfrak{S}(s,w)$ is not a Dirac distribution. We define $\mathfrak{S}\uparrow w$ by

[TABLE]

Now,

[TABLE]

But then there is an action $\beta\in Act(last(\pi))$ such that

[TABLE]

because $\mathfrak{S}(s,w)$ is a probability distribution. We conclude that the scheduler $\mathfrak{S}^{\prime}$ which agrees with $\mathfrak{S}$ on all state-weight-pairs except $(s,w)$ and assigns probability $1$ to $\beta$ for $(s,w)$ satisfies $\mathit{PE}^{\mathfrak{S}^{\prime}}_{s_{\mathit{init}}}\geq\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}$. In this way, we can replace all probability distributions that $\mathfrak{S}$ chooses by Dirac distributions and generate a sequence of schedulers with non-decreasing partial expectations. Ultimately, we obtain a weight-based deterministic scheduler $\mathfrak{T}$ with $\mathit{PE}^{\mathfrak{T}}_{s_{\mathit{init}}}\geq\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}$. ∎

Definition 36** (Metric on weight-based deterministic schedulers).**

Given an MDP $\mathcal{M}$ with arbitrary integer weights, we define the following metric $d^{\mathcal{M}}$ on the set of weight-based deterministic schedulers, i.e. on the set of functions from $S\times\mathbb{Z}\to Act$: For two such schedulers $\mathfrak{S}$ and $\mathfrak{T}$, we let

[TABLE]

where $R$ is the greatest natural number such that

[TABLE]

or $\infty$ if there is no greatest such natural number.

Lemma 37**.**

The metric space $(Act^{S\times\mathbb{Z}},d^{\mathcal{M}})$ is compact.

Proof.

We can identify $Act^{S\times\mathbb{Z}}$ with $(Act^{S\times\{+,-\}})^{\mathbb{N}}$. Then it is easy to see that the metric $d^{\mathcal{M}}$ induces the usual tree topology on this finitely branching tree of height $\omega$. Therefore, the space is homeomorphic to the Cantor space $2^{\omega}$ and hence compact. ∎

Lemma 38** (Upper Semi-Continuity of Partial Expectations).**

Assume that $\mathit{PE}^{\sup}_{s_{\mathit{init}}}$ is finite in $\mathcal{M}$. Then, the function

[TABLE]

assigning $\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}$ to a weight-based deterministic scheduler $\mathfrak{S}$ is upper semi-continuous.

Proof.

Let $\mathfrak{S}$ be a $WD$-scheduler with $\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}>-\infty$. Given $\epsilon>0$, we will define a natural number $R$ such that any $WD$-scheduler $\mathfrak{T}$ with $\mathfrak{T}\upharpoonright S\times[-R,R]=\mathfrak{S}\upharpoonright S\times[-R,R]$ satisfies $\mathit{PE}^{\mathfrak{T}}_{s_{\mathit{init}}}<\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}+4\epsilon$.

First, we observe that for each state $s$ there is a natural number $W_{s}$ and a probability $p_{s}<1$ such that $\mathrm{Pr}^{\max}_{s}(\Diamond wgt>W_{s})\leq p_{s}$ because $\mathit{PE}^{\sup}_{s}<\infty$. Let $W:=\max_{s}W_{s}+\max_{s,\alpha}|wgt(s,\alpha)|$ and $p:=\max_{s}p_{s}$. Then, for each state $s$ and each natural number $n$, we have that $\mathrm{Pr}^{\max}_{s}(\Diamond wgt>n\cdot W)\leq p^{n}$.

As $\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}>-\infty$, we know that

[TABLE]

We define $l_{\epsilon}^{\mathfrak{S}}$ to be the smallest natural number such that

[TABLE]

Let $k$ be the smallest natural number such that $p^{k}\cdot(l^{\mathfrak{S}}_{\epsilon}+W/(1-p)^{2})<\epsilon$. Further define $R^{-}:=l^{\mathfrak{S}}_{\epsilon}+k\cdot W$. Let

[TABLE]

and let $n$ be the least natural number such that

[TABLE]

Finally, define $R^{+}:=n\cdot W$.

We claim that $R:=\max\{R^{-},R^{+}\}$ does the job. So let $\mathfrak{T}$ be a scheduler with $\mathfrak{T}\upharpoonright S\times[-R,R]=\mathfrak{S}\upharpoonright S\times[-R,R]$. Let

[TABLE]

Recall the following definition. Given a finite path $\pi$ and a path $\rho$ starting in $last(\pi)$ and a scheduler $\mathfrak{Q}$, we further define the scheduler $\mathfrak{Q}\uparrow\pi$ by

[TABLE]

where $\pi;\rho$ denotes the concatenation of the paths $\pi$ and $\rho$.

The schedulers $\mathfrak{S}$ and $\mathfrak{T}$ agree on all paths not having a prefix in one of these two sets. So,

[TABLE]

Split up into two sums, we get the following inequalities:

[TABLE]

[TABLE]

This finishes the case where $\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}>-\infty$. If $\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}=-\infty$, we have to show for each $b\in\mathbb{R}$ that there is an $R$ such that all schedulers agreeing with $\mathfrak{S}$ on the weight-window $[-R,R]$ have a partial expectation below $b$. But as we can make

[TABLE]

arbitrarily small, this is easy to see. ∎

Theorem 39** (Existence of Optimal Schedulers for Partial Expectations).**

If $\mathit{PE}^{\sup}_{s_{\mathit{init}}}$ is finite in an MDP $\mathcal{M}$, then there is a weight-based deterministic scheduler $\mathfrak{S}$ with

[TABLE]

Proof.

If $\mathit{PE}^{\sup}_{s_{\mathit{init}}}$ is finite, then the map $\mathit{PE}:(WD,d^{WD})\to(\mathbb{R}_{\infty},d^{euclid})$ is upper semi-continuous as we have just shown. But then, this map has a maximum because $(WD,d^{WD})$ is a compact metric space. ∎

Appendix 0.C Approximation

0.C.1 Approximating Optimal Partial Expectations

Proposition 40**.**

Let $s\in S$ and let

[TABLE]

Then any $WD$-scheduler $\mathfrak{S}$ maximizing the partial expectation in $\mathcal{M}$ satisfies $\mathfrak{S}(s,w)\in Act^{\min}(s)$ if $w\leq\mathfrak{q}_{s}$.

Proof.

The proof works analogously to the proof of Proposition 30. ∎

Let $\mathfrak{q}:=\min_{s\in S}\mathfrak{q}_{s}$ and let $D:=\mathit{PE}^{ub}-\min\{\mathit{PE}^{\mathfrak{Max}}_{s},\mathit{PE}^{\mathfrak{Min}}_{s}|s\in S\}$. Given $\epsilon>0$, we define $R^{+}_{\epsilon}:=(c_{\mathcal{M}}+W)\cdot\left\lceil\frac{\log(2D)+\log(1/\epsilon)}{\log(1/\lambda_{\mathcal{M}})}\right\rceil$ and $R^{-}_{\epsilon}:=\mathfrak{q}-R^{+}_{\epsilon}$.

Theorem 41**.**

There is a weight-based deterministic scheduler $\mathfrak{S}$ such that the scheduler $\mathfrak{T}$ defined by

[TABLE]

satisfies $\mathit{PE}^{\mathfrak{T}}_{s_{\mathit{init}}}\geq\mathit{PE}^{\sup}_{s_{\mathit{init}}}-\epsilon$.

Proof.

Let $\mathfrak{S}$ be a weight-based deterministic scheduler with $\mathit{PE}^{\mathfrak{S}}_{s_{\mathit{init}}}=\mathit{PE}^{\sup}_{s_{\mathit{init}}}$. Define

[TABLE]

We give an estimation for the difference $\mathit{PE}^{\sup}_{s_{\mathit{init}}}-\mathit{PE}^{\mathfrak{T}}_{s_{\mathit{init}}}$. In order to do so, we define the following two sets:

[TABLE]

The schedulers $\mathfrak{S}$ and $\mathfrak{T}$ agree on all paths not in $\Pi^{+}_{\epsilon}$ or $\Pi^{-}_{\epsilon}$. Hence,

[TABLE]

For the first sum, we have the following estimation:

[TABLE]

For the second sum, consider the following scheduler. On extensions of paths in $\Pi^{-}_{\epsilon}$, let $\mathfrak{S}^{\prime}$ be the scheduler which behaves like $\mathfrak{S}$ until the accumulated weight is at least $\mathfrak{q}$ again and then switches to the choices of $\mathfrak{Min}$. We know that $\mathfrak{S}$ only chooses actions in $Act^{\min}(s)$ when in a state $s$ with accumulated weight below $\mathfrak{q}$. On the other hand, $\mathfrak{Min}$ is optimal among these schedulers. So, $\mathfrak{Min}$ is at least as good as $\mathfrak{S}^{\prime}$ on extensions of paths in $\Pi^{-}_{\epsilon}$ with respect to maximizing the partial expectation. Further, starting at a path in $\Pi^{-}_{\epsilon}$ we reach an accumulated weight of at least $\mathfrak{q}$ only if we accumulate a weight of at least $R^{+}$. Afterwards, we can bound the advantage of $\mathfrak{S}$ over $\mathfrak{Min}$ by $D$. So, we get the following estimation:

[TABLE]

So, $\mathit{PE}^{\sup}_{s_{\mathit{init}}}-\mathit{PE}^{\mathfrak{T}}_{s_{\mathit{init}}}\leq\epsilon$. ∎

This result now allows us to compute an $\epsilon$-approximation and an $\epsilon$-optimal scheduler with finite memory by linear programming, similar to the case of non-negative weights. The linear program has $R^{+}_{\epsilon}+R^{-}_{\epsilon}$ many variables and $|Act|$-times as many inequalities.

Corollary 42**.**

The maximal partial expectation $\mathit{PE}^{\sup}_{s_{\mathit{init}}}$ can be approximated up to an absolute error of $\epsilon$ in time exponential in the size of $\mathcal{M}$ and polynomial in $\log(1/\epsilon)$.

Proof.

Consider the following linear program with one variable $x_{s,w}$ for each $s\in S$ and $R^{-}-W\leq w\leq R^{+}+W$:

Minimize $\sum_{s,w}x_{s,w}$ under the following constraints:

[TABLE]

for $w\geq R^{+}$ and $s\in S\setminus\{\mathit{goal},fail\}$,

[TABLE]

for $w\leq R^{-}$ and $s\in S\setminus\{\mathit{goal},fail\}$,

[TABLE]

and for $R^{-}<w<R^{+}$, $s\in S\setminus\{\mathit{goal},fail\}$, and $\alpha\in Act(s)$,

[TABLE]

The unique solvability can be shown as in Proposition 33 using that all end components have negative mean payoff: We can interpret the linear program on an MDP with state space $S\times\{R^{-}-W,\dots,R^{+}\}$ and the transitions induced by $\mathcal{M}$. This MDP now has no end components. ∎

0.C.2 Transfer to Conditional Expectations

We restate the algorithm given in Section 5. Let $\mathcal{M}$ be an MDP with $\mathit{CE}^{sup}_{s_{\mathit{init}}}<\infty$ and let $\epsilon>0$. By Proposition 6, we can assume that $\mathrm{Pr}^{\min}_{\mathcal{M},s_{\mathit{init}}}(\Diamond\mathit{goal})=:p$ is positive. We know that $\mathit{CE}^{sup}_{s_{\mathit{init}}}\in[\mathit{CE}^{\mathfrak{Max}}_{s_{\mathit{init}}},\mathit{CE}^{ub}]$. We perform a binary search to approximate $\mathit{CE}^{sup}_{s_{\mathit{init}}}$:

We put $A_{0}:=\mathit{CE}^{\mathfrak{Max}}_{s_{\mathit{init}}}$ and $B_{0}:=\mathit{CE}^{ub}$. Given $A_{i}$ and $B_{i}$, let $\theta_{i}:=(A_{i}+B_{i})/2$. Then, we approximate $\mathit{PE}^{\sup}_{s_{\mathit{init}}}[-\theta_{i}]$ up to an absolute error of $p\cdot\epsilon$. Let $E_{i}$ be the value of this approximation. If $E_{i}\in[-2p\cdot\epsilon,2p\cdot\epsilon]$, terminate and return $\theta_{i}$ as the approximation for $\mathit{CE}^{\sup}_{s_{\mathit{init}}}$. If $E_{i}<-2p\cdot\epsilon$, put $A_{i+1}:=A_{i}$ and $B_{i+1}:=\theta_{i}$, and repeat. If $E_{i}>2p\cdot\epsilon$, put $A_{i+1}:=\theta_{i}$ and $B_{i+1}:=B_{i}$, and repeat.

Proposition 43**.**

The procedure terminates after at most $\lceil\log((A_{0}-B_{0})/(p\cdot\epsilon))\rceil$ iterations and returns an $3\epsilon$-approximation of $\mathit{CE}^{\sup}_{s_{\mathit{init}}}$ in time exponential in the size of $\mathcal{M}$ and polynomial in $\log(1/\epsilon)$.

Proof.

We begin by showing that the algorithm terminates after at most $\lceil\log((A_{0}-B_{0})/(\epsilon\cdot p))\rceil$ many iterations, i.e. when $|A_{i}-B_{i}|\leq\epsilon\cdot p$. We know that $\mathit{PE}^{\sup}_{s_{\mathit{init}}}[-\theta_{i}]<0$ if $E_{i}<-2p\epsilon$ and $\mathit{PE}^{\sup}_{s_{\mathit{init}}}[-\theta_{i}]>0$ if $E_{i}>2p\epsilon$. By Proposition 2, we conclude that $\mathit{CE}^{\sup}_{s_{\mathit{init}}}\in[A_{i+1},B_{i+1}]$ at any time. So, after at most $log((A_{0}-B_{0})/(\epsilon\cdot p))$ many iteration, we have that $|A_{i}-B_{i}|\leq\epsilon\cdot p$ and hence $\mathit{CE}^{\sup}_{s_{\mathit{init}}}-\epsilon\cdot p\leq\theta_{i}\leq\mathit{CE}^{\sup}_{s_{\mathit{init}}}+\epsilon\cdot p$. We claim that then $E_{i}\in[-2p\epsilon,2p\epsilon]$. Suppose $E_{i}<-2p\epsilon$. Then $\mathit{PE}^{\sup}_{s_{\mathit{init}}}[-\theta_{i}]<-p\epsilon$. But we have

[TABLE]

contradicting the supposition. Analogously, we show that $E_{i}$ cannot be greater than $2p\epsilon$.

Next, we show that the algorithm returns an $3\epsilon$-approximation of $\mathit{CE}^{\sup}_{s_{\mathit{init}}}$. As soon as the algorithm terminates, we have that $E_{i}\in[-2p\epsilon,2p\epsilon]$. So, $\mathit{PE}^{\sup}_{s_{\mathit{init}}}[-\theta_{i}]\in[-3p\epsilon,3p\epsilon]$. So there is a scheduler $\mathfrak{S}$ with

[TABLE]

As $\mathrm{Pr}^{\mathfrak{S}}_{s_{\mathit{init}}}(\Diamond\mathit{goal})\geq p$, this implies $\mathit{CE}^{\sup}_{s_{\mathit{init}}}\geq\mathit{CE}^{\mathfrak{S}}_{s_{\mathit{init}}}\geq\theta_{i}-3\epsilon$. On the other hand, suppose that $\mathit{CE}^{\sup}_{s_{\mathit{init}}}>\theta_{i}+3\epsilon$. Then there is a scheduler $\mathfrak{T}$ with $\mathit{CE}^{\mathfrak{T}}_{s_{\mathit{init}}}>\theta_{i}+3\epsilon$. For this scheduler, we have

[TABLE]

This contradicts $\mathit{PE}^{\sup}_{s_{\mathit{init}}}[-\theta_{i}]\leq 3p\epsilon$. Therefore, the algorithm indeed returns a $3\epsilon$-approximation of $\mathit{CE}^{\sup}_{s_{\mathit{init}}}$.

Finally, we show that the claimed running time is correct: The algorithm stops after at most $\lceil\log((A_{0}-B_{0})/(\epsilon\cdot p))\rceil$ iterations. As all values involved can be computed in polynomial time, this is polynomial in the size of $\mathcal{M}$ and linear in $\log(1/\epsilon)$. In each iteration, we have to approximate the maximal partial expectation $\mathit{PE}^{\sup}_{s_{\mathit{init}}}[-\theta_{i}]$ up to an absolute error of $p\cdot\epsilon$. As the logarithmic length of $\theta_{i}$ is polynomial in the size of $\mathcal{M}$ as well, this can be done in time exponential in the size of $\mathcal{M}$ and polynomial in $\log(1/\epsilon)$. ∎