Efficient Computation of Updated Lower Expectations for Imprecise Continuous-Time Hidden Markov Chains

TL;DR

This paper introduces a polynomial-time algorithm for efficiently computing lower expectations in imprecise continuous-time hidden Markov chains, accommodating various output variable types and model uncertainties.

Contribution

It presents the first polynomial runtime algorithm for inference in imprecise continuous-time hidden Markov chains with flexible output variables.

Findings

Algorithm achieves polynomial runtime for lower expectation computation.

Supports both discrete and continuous output variables.

Handles various types of model uncertainty using imprecise probabilities.

Abstract

We consider the problem of performing inference with imprecise continuous-time hidden Markov chains, that is, imprecise continuous-time Markov chains that are augmented with random output variables whose distribution depends on the hidden state of the chain. The prefix `imprecise' refers to the fact that we do not consider a classical continuous-time Markov chain, but replace it with a robust extension that allows us to represent various types of model uncertainty, using the theory of imprecise probabilities. The inference problem amounts to computing lower expectations of functions on the state-space of the chain, given observations of the output variables. We develop and investigate this problem with very few assumptions on the output variables; in particular, they can be chosen to be either discrete or continuous random variables. Our main result is a polynomial runtime algorithm to compute the lower expectation of functions on the state-space at any given time-point, given a collection of observations of the output variables.

Equations

$$P(X_{t+\Delta}\,|\,X_{t},X_{u}=x_{u})\approx\bigl{[}I+\Delta Q_{t,x_{u}}\bigr{]}(X_{t},X_{t+\Delta})\,,$$

$${\underline{\mathbb{E}}}_{\mathcal{Q},\mathcal{M}}[\cdot |\cdot ]:=\inf \left\{{\mathbb{E}}_P[\cdot |\cdot ]:P\in {\mathbb{P}}_{\mathcal{Q},\mathcal{M}}\right\},$$

$$\left[\underline{Q}f\right](x):=\inf \left\{\sum _{x^{\prime }\in \mathcal{X}}Q(x,x^{\prime })f(x^{\prime }):Q\in \mathcal{Q}\right\}\text{leavevmode}\text{leavevmode}\text{for all x∈X}.$$

$${\underline{\mathbb{E}}}_{\mathcal{Q},\mathcal{M}}[f(X_s)|X_t]={\underline{\mathbb{E}}}_{\mathcal{Q}}[f(X_s)|X_t]:=\underset{n\to +\infty }{\lim }{\left[I+\frac{(s-t)}{n}\underline{Q}\right]}^{n}f,$$

$$\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[f(X_{s})\,|\,X_{t}]=\underline{\mathbb{E}}_{\mathcal{Q}}[f(X_{s})\,|\,X_{t}]\approx\bigl{[}I+\Delta\underline{Q}\,\bigr{]}^{n}f\,.$$

$${\underline{\mathbb{E}}}_{\mathcal{Q},\mathcal{M}}[f(X_0)]={\underline{\mathbb{E}}}_{\mathcal{M}}[f(X_0)]:=\inf \left\{\sum _{x\in \mathcal{X}}p(x)f(x):p\in \mathcal{M}\right\}.$$

$$\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[f(X_{t})]=\underline{\mathbb{E}}_{\mathcal{M}}\bigl{[}\underline{\mathbb{E}}_{\mathcal{Q}}[f(X_{t})\,|\,X_{0}]\bigr{]}\,,$$

$$P(Y_t\in O_t,X_t=x_t,X_v=x_v):=P_{\mathcal{Y}|\mathcal{X}}(O_t|x_t)P_{\mathcal{X}}(X_t=x_t,X_v=x_v).\text{vspace}-2pt$$

$$P(Y_u\in O_u,X_u=x_u,X_v=x_v):=P_{\mathcal{Y}|\mathcal{X}}(O_u|x_u)P_{\mathcal{X}}(X_u=x_u,X_v=x_v).\text{vspace}-2pt$$

$${\mathbb{E}}_P[f(X_v)|Y_u\in O_u]:=\sum _{x_v\in {\mathcal{X}}_v}f(x_v)\frac{P(X_v=x_v,Y_u\in O_u)}{P(Y_u\in O_u)}=\frac{{\mathbb{E}}_{P_{\mathcal{X}}}[f(X_v)P_{\mathcal{Y}|\mathcal{X}}(O_u|X_u)]}{{\mathbb{E}}_{P_{\mathcal{X}}}[P_{\mathcal{Y}|\mathcal{X}}(O_u|X_u)]},$$

$$\underline{\mathbb{E}}_{\mathcal{Z}}\bigl{[}f(X_{v})\,|\,Y_{u}\in O_{u}\bigr{]}:\!=\inf\bigl{\{}\mathbb{E}_{P}[f(X_{v})\,|\,Y_{u}\in O_{u}]\,:\,P\in\mathcal{Z},\,P(Y_{u}\in O_{u})>0\bigr{\}}\,,$$

$$\underline{\mathbb{E}}_{\mathcal{Z}}\bigl{[}f(X_{v})\,|\,Y_{u}\in O_{u}\bigr{]}=\max\left\{\mu\in\mathbb{R}\,:\,\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\bigl{[}P_{\mathcal{Y}|\mathcal{X}}(O_{u}|X_{u})\bigl{(}f(X_{v})-\mu\bigr{)}\bigr{]}\geq 0\right\}\,.$$

$${\mathbb{E}}_P[f(X_v)|Y_u=y_u]:=\underset{i\to +\infty }{\lim }{\mathbb{E}}_P[f(X_v)|Y_u\in O_u^i].$$

$${\varphi }_u(y_u|x_u):=\underset{i\to +\infty }{\lim }\frac{P_{\mathcal{Y}|\mathcal{X}}(O_u^i|x_u)}{{\lambda }_i}$$

$${\mathbb{E}}_P[f(X_v)|Y_u=y_u]:=\underset{i\to +\infty }{\lim }{\mathbb{E}}_P[f(X_v)|Y_u\in O_u^i]=\frac{{\mathbb{E}}_{P_{\mathcal{X}}}[f(X_v){\varphi }_u(y_u|X_u)]}{{\mathbb{E}}_{P_{\mathcal{X}}}[{\varphi }_u(y_u|X_u)]}.$$

$${\varphi }_t(y_t|x_t):=\underset{i\to +\infty }{\lim }\frac{P_{\mathcal{Y}|\mathcal{X}}(O_t^i|x_t)}{{\lambda }_{t,i}}$$

$$P_{\mathcal{Y}|\mathcal{X}}(O|x):={\int }_O\psi (y|x)\mathrm{d}y,$$

$${\varphi }_t(y_t|x_t)=\underset{i\to +\infty }{\lim }\frac{P_{\mathcal{Y}|\mathcal{X}}(O_t^i|x_t)}{\lambda (O_t^i)}=\psi (y_t|x_t),$$

$$\underline{\mathbb{E}}_{\mathcal{Z}}\bigl{[}f(X_{v})\,|\,Y_{u}=y_{u}\bigr{]}:\!=\inf\{\mathbb{E}_{P}[f(X_{v})|Y_{u}=y_{u}]\,:\,P\in\mathcal{Z}\}\,,$$

$$\underline{\mathbb{E}}_{\mathcal{Z}}\bigl{[}f(X_{v})\,|\,Y_{u}=y_{u}\bigr{]}=\max\left\{\mu\in\mathbb{R}\,:\,\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\bigl{[}\phi_{u}(y_{u}|X_{u})\bigl{(}f(X_{v})-\mu\bigr{)}\bigr{]}\geq 0\right\}\,.$$

$${\underline{\mathbb{E}}}_{\mathcal{Z}}^{\mathrm{R}}[f(X_v)|Y_u=y_u]:=\inf \left\{{\mathbb{E}}_P[f(X_v)|Y_u=y_u]:P\in \mathcal{Z},{\mathbb{E}}_{P_{\mathcal{X}}}[{\varphi }_u(y_u|X_u)]>0\right\},$$

$${\underline{\mathbb{E}}}_{\mathcal{Z}}^{\mathrm{L}}[f(X_v)|Y_u=y_u]:=\inf \left\{{\mathbb{E}}_P[f(X_v)|Y_u=y_u]:P\in \mathcal{Z},\text{leavevmode}\text{EP[f(Xv)∣Yu=yu] exists}\right\},$$

$$P_{\mathcal{Y}|\mathcal{X}}(O_u|X_u)=\prod _{t\in u}{P}_{\mathcal{Y}|\mathcal{X}}(O_t|X_t)\text{and}{\varphi }_u(y_u|X_u)=\prod _{t\in u}{\varphi }_t(y_t|X_t).\text{vspace}-5pt$$

$$\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\left[\left(\prod_{t\in u}g_{t}(X_{t})\right)\bigl{(}f(X_{s})-\mu\bigr{)}\right]=\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\left[\prod_{t\in u^{\prime}}g_{t}(X_{t})\right]\,.$$

$$g_{t_{i}}^{+}(x_{t_{i}}):\!=\left\{\begin{array}[]{ll}g_{t_{i}}(x_{t_{i}})\underline{\mathbb{E}}_{\mathcal{Q}}[g_{t_{i+1}}^{+}(X_{t_{i+1}})\,|\,X_{t_{i}}=x_{t_{i}}]&\text{if $g_{t_{i}}(x_{t_{i}})\geq 0$,}\\ g_{t_{i}}(x_{t_{i}})\overline{\mathbb{E}}_{\mathcal{Q}}[g_{t_{i+1}}^{-}(X_{t_{i+1}})\,|\,X_{t_{i}}=x_{t_{i}}]&\text{if $g_{t_{i}}(x_{t_{i}})<0$}\end{array}\right.\vspace{-5pt}$$

$$g_{t_{i}}^{-}(x_{t_{i}}):\!=\left\{\begin{array}[]{ll}g_{t_{i}}(x_{t_{i}})\overline{\mathbb{E}}_{\mathcal{Q}}[g_{t_{i+1}}^{-}(X_{t_{i+1}})\,|\,X_{t_{i}}=x_{t_{i}}]&\text{if $g_{t_{i}}(x_{t_{i}})\geq 0$,}\\ g_{t_{i}}(x_{t_{i}})\underline{\mathbb{E}}_{\mathcal{Q}}[g_{t_{i+1}}^{+}(X_{t_{i+1}})\,|\,X_{t_{i}}=x_{t_{i}}]&\text{if $g_{t_{i}}(x_{t_{i}})<0$.}\end{array}\right.\vspace{5pt}$$

$$\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[f(X_{u},X_{v},X_{w})\,|\,X_{u}]=\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\bigl{[}\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[f(X_{u},X_{v},X_{w})\,|\,X_{u},X_{v}]\,|\,X_{u}\bigr{]}\,.$$

$$\mathbb{E}_{P_{\mathcal{X}}}\bigl{[}g(X_{u})\bigl{(}f(X_{v})-\mu\bigr{)}\bigr{]}=0\,.$$

$$P_{\mathcal{X}}(X_u=x_u)=0,$$

$$\displaystyle\mathbb{E}_{P_{\mathcal{X}}}\bigl{[}g(X_{u})\bigl{(}f(X_{v})-\mu\bigr{)}\bigr{]}$$

Full text

1 Introduction

A continuous-time Markov chain (CTMC) is a stochastic model that describes the evolution of a dynamical system under uncertainty. Specifically, it provides a probabilistic description of how such a system might move through a finite state-space, as time elapses in a continuous fashion. There are various ways in which this model class can be extended.

One such extension are continuous-time hidden Markov chains (CTHMC’s) (Wei et al., 2002). Such a CTHMC is a stochastic model that contains a continuous-time Markov chain as a latent variable—that is, the actual realised behaviour of the system cannot be directly observed. This model furthermore incorporates random output variables, which depend probabilistically on the current state of the system, and it is rather realisations of these variables that one observes. Through this stochastic dependency between the output variables and the states in which the system might be, one can perform inferences about quantities of interest that depend on these states—even though they have not been, or cannot be, observed directly.

Another extension of CTMC’s, arising from the theory of imprecise probabilities (Walley, 1991), are imprecise continuous-time Markov chains (ICTMC’s) (Škulj, 2015; Krak et al., 2016). This extension can be used to robustify against uncertain numerical parameter assessments, as well as the simplifying assumptions of time-homogeneity and that the model should satisfy the Markov property. Simply put, an ICTMC is a set of continuous-time stochastic processes, some of which are “traditional” time-homogeneous CTMC’s. However, this set also contains more complicated processes, which are non-homogeneous and do not satisfy the Markov property.

In this current work, we combine these two extensions by considering imprecise continuous-time hidden Markov chains—a stochastic model analogous to a CTHMC, but where the latent CTMC is replaced by an ICTMC. We will focus in particular on practical aspects of the corresponding inference problem. That is, we provide results on how to efficiently compute lower expectations of functions on the state-space, given observed realisations of the output variables.

The proofs of all results are gathered in an appendix, where they are largely ordered by their chronological appearance in the main text.

1.1 Related Work

As should be clear from the description of CTHMC’s in Section 1, this model class extends the well-known (discrete-time) hidden Markov models (HMM’s) to a continuous-time setting. In the same sense, the present subject of ICTHMC’s can be seen to extend previous work on imprecise hidden Markov models (iHMM’s) (de Cooman et al., 2010) to a continuous-time setting. Hence, the model under consideration should hopefully be intuitively clear to readers familiar with (i)HMM’s.

The main novelty of this present work is therefore not the (somewhat obvious) extension of iHMM’s to ICTHMC’s, but rather the application of recent results on ICTMC’s (Krak et al., 2016) to derive an efficient solution to the continuous-time analogue of inference in iHMM’s. The algorithm that we present is largely based on combining these results with the ideas behind the MePiCTIr algorithm (de Cooman et al., 2010) for inference in credal trees under epistemic irrelevance.

A second novelty of the present paper is that, contrary to most of the work in the literature on iHMM’s, we allow the output variables of the ICTHMC to be either discrete or continuous. This allows the model to be applied to a much broader range of problems. At the same time, it turns out that this does not negatively influence the efficiency of the inference algorithm.

2 Preliminaries

We denote the reals as $\mathbb{R}$, the non-negative reals as $\mathbb{R}_{\geq 0}$, and the positive reals as $\mathbb{R}_{>0}$. The natural numbers are denoted by $\mathbb{N}$, and we define $\mathbb{N}_{0}:\!=\mathbb{N}\cup\{0\}$.

Since we are working in a continuous-time setting, a time-point is an element of $\mathbb{R}_{\geq 0}$, and these are typically denoted by $t$ or $s$. We also make extensive use of non-empty, finite sequences of time points $u\subset\mathbb{R}_{\geq 0}$. These are taken to be ordered, so that they may be written $u=t_{0},\ldots,t_{n}$, for some $n\in\mathbb{N}_{0}$, and such that then $t_{i}<t_{j}$ for all $i,j\in\{0,\ldots,n\}$ for which $i<j$. Such sequences are usually denoted by $u$ or $v$, and we let $\mathcal{U}$ be the entire set of them.

Throughout, we consider some fixed, finite state space $\mathcal{X}$. A generic element of $\mathcal{X}$ will be denoted by $x$. When considering the state-space at a specific time $t$, we write $\mathcal{X}_{t}:\!=\mathcal{X}$, and $x_{t}$ denotes a generic state-assignment at this time. When considering multiple time-points $u$ simultaneously, we define the joint state-space as $\mathcal{X}_{u}:\!=\prod_{t_{i}\in u}\mathcal{X}_{t_{i}}$, of which $x_{u}=(x_{t_{0}},\ldots,x_{t_{n}})$ is a generic element.

For any $u\in\mathcal{U}$, we let $\mathcal{L}(\mathcal{X}_{u})$ be the set of all real-valued functions on $\mathcal{X}_{u}$.

2.1 Imprecise Continuous-Time Markov Chains

We here briefly recall the most important properties of imprecise continuous-time Markov chains (ICTMC’s), following the definitions and results of Krak et al. (2016). For reasons of brevity, we provide these definitions in a largely intuitive, non-rigorous manner, and refer the interested reader to this earlier work for an in-depth treatise on the subject.

An ICTMC will be defined below as a specific set of continuous-time stochastic processes. Simply put, a continuous-time stochastic process is a joint probability distribution over random variables $X_{t}$, for each time $t\in\mathbb{R}_{\geq 0}$, where each random variable $X_{t}$ takes values in $\mathcal{X}$.

It will be convenient to have a way to numerically parameterise such a stochastic process $P$. For this, we require two different kinds of parameters. First, we need the specification of the initial distribution $P(X_{0})$ over the state at time zero; this simply requires the specification of some probability mass function on $\mathcal{X}_{0}$. Second, we need to parameterise the dynamic behaviour of the model.

In order to describe this dynamic behaviour, we require the concept of a rate matrix. Such a rate matrix $Q$ is a real-valued $\lvert\mathcal{X}\rvert\times\lvert\mathcal{X}\rvert$ matrix, whose off-diagonal elements are non-negative, and whose every row sums to zero—thus, the diagonal elements are non-positive. Such a rate matrix may be interpreted as describing the “rate of change” of the conditional probability $P(X_{s}\,|\,X_{t},X_{u}=x_{u})$, when $s$ is close to $t$. In this conditional probability, it is assumed that $u<t$, whence the state assignment $x_{u}$ is called the history. For small enough $\Delta\in\mathbb{R}_{>0}$, we may now write that

[TABLE]

for some rate matrix $Q_{t,x_{u}}$, where $I$ denotes the $\lvert\mathcal{X}\rvert\times\lvert\mathcal{X}\rvert$ identity matrix, and where the quantity $[I+\Delta Q_{t,x_{u}}](X_{t},X_{t+\Delta})$ denotes the element at the $X_{t}$-row and $X_{t+\Delta}$-column of the matrix $I+\Delta Q_{t,x_{u}}$. Note that in general, this rate matrix $Q_{t,x_{u}}$ may depend on the specific time $t$ and history $x_{u}$ at which this relationship is stated.

If these rate matrices only depend on the time $t$ and not on the history $x_{u}$, i.e. if $Q_{t,x_{u}}=Q_{t}$ for all $t$ and all $x_{u}$, then it can be shown that $P$ satisfies the Markov property: $P(X_{s}\,|\,X_{t},X_{u})=P(X_{s}\,|\,X_{t})$. In this case, $P$ is called a continuous-time Markov chain.

Using this method of parameterisation, an imprecise continuous-time Markov chain (ICTMC) is similarly parameterised using a set of rate matrices $\mathcal{Q}$, and a set of initial distributions $\mathcal{M}$. The corresponding ICTMC, denoted by $\mathbb{P}_{\mathcal{Q},\mathcal{M}}$, is the set of all continuous-time stochastic processes whose dynamics can be described using the elements of $\mathcal{Q}$, and whose initial distributions are consistent with $\mathcal{M}$. That is, $\mathbb{P}_{\mathcal{Q},\mathcal{M}}$ is the set of stochastic processes $P$ for which $P(X_{0})\in\mathcal{M}$ and for which $Q_{t,x_{u}}\in\mathcal{Q}$ for every time $t$ and history $x_{u}$.

The lower expectation with respect to this set $\mathbb{P}_{\mathcal{Q},\mathcal{M}}$ is then defined as

[TABLE]

where $\mathbb{E}_{P}[\cdot\,|\,\cdot]$ denotes the expectation with respect to the (precise) stochastic process $P$. The upper expectation $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}$ is defined similarly, and is derived through the well-known conjugacy property $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\cdot\,|\,\cdot]=-\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[-\cdot\,|\,\cdot]$. Note that it suffices to focus on lower (or upper) expectations, and that lower (and upper) probabilities can be regarded as a special case; for example, for any $A\subseteq\mathcal{X}$, we have that $\underline{P}_{\mathcal{Q},\mathcal{M}}(X_{s}\in A\,|\,X_{t}):\!=\inf\{P(X_{s}\in A|X_{t})\,:\,P\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}\}=\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\mathbb{I}_{A}(X_{s})\,|\,X_{t}]$, where $\mathbb{I}_{A}$ is the indicator of $A$, defined for all $x\in\mathcal{X}$ by $\mathbb{I}_{A}(x):\!=1$ if $x\in A$ and $\mathbb{I}_{A}(x):\!=0$ otherwise.

In the sequel, we will assume that $\mathcal{M}$ is non-empty, and that $\mathcal{Q}$ is non-empty, bounded,111That is, that there exists a $c\in\mathbb{R}_{\geq 0}$ such that, for all $Q\in\mathcal{Q}$ and $x\in\mathcal{X}$, it holds that $\left|Q(x,x)\right|<c$. convex, and has separately specified rows. This latter property states that $\mathcal{Q}$ is closed under arbitrary recombination of rows from its elements; see (Krak et al., 2016, Definition 24) for a formal definition. Under these assumptions, $\mathbb{P}_{\mathcal{Q},\mathcal{M}}$ satisfies an imprecise Markov property, in the sense that $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[f(X_{s})\,|\,X_{t},X_{u}=x_{u}]=\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[f(X_{s})\,|\,X_{t}]$. This property explains why we call this model an imprecise continuous-time “Markov” chain.

2.2 Computing Lower Expectations for ICTMC’s

Because we want to focus in this paper on providing efficient methods of computation, we here briefly recall some previous results from Krak et al. (2016) about how to compute lower expectations for ICTMC’s. We focus in particular on how to do this for functions on a single time-point.

To this end, it is useful to introduce the lower transition rate operator $\underline{Q}$ that corresponds to $\mathcal{Q}$. This operator is a map from $\mathcal{L}(\mathcal{X})$ to $\mathcal{L}(\mathcal{X})$, defined for every $f\in\mathcal{L}(\mathcal{X})$ by

[TABLE]

Using this lower transition rate operator $\underline{Q}$, we can compute conditional lower expectations in the following way. For any $t,s\in\mathbb{R}_{\geq 0}$, with $t\leq s$, and any $f\in\mathcal{L}(\mathcal{X})$, it has been shown that

[TABLE]

where $I$ is the identity operator on $\mathcal{L}(\mathcal{X})$, in the sense that $Ig=g$ for every $g\in\mathcal{L}(\mathcal{X})$. The notation $\underline{\mathbb{E}}_{\mathcal{Q}}$ is meant to indicate that this conditional lower expectation only depends on $\mathcal{Q}$, and not on $\mathcal{M}$. The above implies that for large enough $n\in\mathbb{N}$, and writing $\Delta:\!=\nicefrac{{(s-t)}}{{n}}$, we have

[TABLE]

Concretely, this means that if one is able to solve the minimisation problem in Equation (1)—which is relatively straightforward for “nice enough” $\mathcal{Q}$, e.g., convex hulls of finite sets of rate matrices—then one can also compute conditional lower expectations using the expression in Equation 2. In practice, we do this by first computing $f_{1}^{\prime}:\!=\underline{Q}f$ using Equation (1), and then computing $f_{1}:\!=f+\Delta f_{1}^{\prime}$. Next, we compute $f_{2}^{\prime}:\!=\underline{Q}f_{1}$, from which we obtain $f_{2}:\!=f_{1}+\Delta f_{2}^{\prime}$. Proceeding in this fashion, after $n$ steps we then finally obtain f_{n}:\!=[I+\Delta\underline{Q}]f_{n-1}=\bigl{[}I+\Delta\underline{Q}\bigr{]}^{n}f, which is roughly the quantity of interest $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[f(X_{s})\,|\,X_{t}]$ provided that $n$ was taken large enough.222We refer the reader to (Krak et al., 2016, Proposition 8.5) for a theoretical bound on the minimum such $n$ that is required to ensure a given maximum error on the approximation in Equation (2). We here briefly note that this bound scales polynomially in every relevant parameter. This means that $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[f(X_{s})\,|\,X_{t}]$ is numerically computable in polynomial time, provided that $\mathcal{Q}$ is such that Equation (1) can also be solved in the same time-complexity order.

As noted above, the conditional lower expectation $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[f(X_{s})|X_{t}]$ only depends on $\mathcal{Q}$. Similarly, and in contrast, the unconditional lower expectation at time zero only depends on $\mathcal{M}$. That is,

[TABLE]

Furthermore, the unconditional lower expectation at an arbitrary time $t\in\mathbb{R}_{\geq 0}$, is given by

[TABLE]

which can therefore be computed by combining Equations (2) and (3). In particular, from a practical point of view, it suffices to first compute the conditional lower expectation $\underline{\mathbb{E}}_{\mathcal{Q}}[f(X_{t})\,|\,X_{0}]$, using Equation (2). Once this quantity is obtained, it remains to compute the right-hand side of Equation (3), which again is relatively straightforward when $\mathcal{M}$ is “nice enough”, e.g., the convex hull of some finite set of probability mass functions.

3 Imprecise Continuous-Time Hidden Markov Chains

In this section, we construct the hidden model that is the subject of this paper. Our aim is to augment the stochastic processes that were introduced in the previous section, by adding random output variables $Y_{t}$ whose distribution depends on the state $X_{t}$ at the same time point $t$.

We want to focus in this paper on the more practical aspect of solving the inference problem of interest, i.e., computing lower expectations on the state-space given some observations. Hence, we will assume that we are given some finite sequence of time points, and we then only consider these time points in augmenting the model. In order to disambiguate the notation, we will henceforth denote stochastic processes as $P_{\mathcal{X}}$, to emphasise that they are only concerned with the state-space.

3.1 Output Variables

We want to augment stochastic processes with random “output variables” $Y_{t}$, whose distribution depends on the state $X_{t}$. We here define the corresponding (conditional) distribution.

We want this definition to be fairly general, and in particular do not want to stipulate that $Y_{t}$ should be either a discrete or a continuous random variable. To this end, we simply consider some set $\mathcal{Y}$ to be the outcome space of the random variable. We then let $\Sigma$ be some algebra on $\mathcal{Y}$. Finally, for each $x\in\mathcal{X}$, we consider some finitely (and possibly $\sigma$-)additive probability measure $P_{\mathcal{Y}|\mathcal{X}}(\cdot|x)$ on $(\mathcal{Y},\Sigma)$, with respect to which the random variable $Y_{t}$ can be defined.

Definition 1

An output model is a tuple $(\mathcal{Y},\Sigma,P_{\mathcal{Y}|\mathcal{X}})$, where $\mathcal{Y}$ is an outcome space, $\Sigma$ is an algebra on $\mathcal{Y}$, and, for all $x\in\mathcal{X}$, $P_{\mathcal{Y}|\mathcal{X}}(\cdot|x)$ is a finitely additive probability measure on $(\mathcal{Y},\Sigma)$.

When considering (multiple) explicit time points, we use notation analogous to that used for states; so, $\mathcal{Y}_{t}:\!=\mathcal{Y}$ for any time $t\in\mathbb{R}_{\geq 0}$, and for any $u\in\mathcal{U}$, we write $\mathcal{Y}_{u}:\!=\prod_{t\in u}\mathcal{Y}_{t}$.

We let $\Sigma_{u}$ denote the set of all events of the type $O_{u}=\times_{t\in u}O_{t}$, where, for all $t\in u$, $O_{t}\in\Sigma$. This set $\Sigma_{u}$ lets us describe observations using assessments of the form (Y_{t}\in O_{t}\text{\leavevmode\nobreak\ for all t\in u}). For any $O_{u}\in\Sigma_{u}$ and $x_{u}\in\mathcal{X}_{u}$, we also adopt the shorthand notation $P_{\mathcal{Y}|\mathcal{X}}(O_{u}|x_{u}):\!=\prod_{t\in u}P_{\mathcal{Y}|\mathcal{X}}(O_{t}|x_{t})$.

3.2 Augmented Stochastic Processes

We now use this notion of an output model to define the stochastic model $P$ that corresponds to a—precise—continuous-time hidden stochastic process. So, consider some fixed output model $(\mathcal{Y},\Sigma,P_{\mathcal{Y}|\mathcal{X}})$, some fixed continuous-time stochastic process $P_{\mathcal{X}}$ and some fixed, non-empty and finite sequence of time-points $u\in\mathcal{U}$ on which observations of the outputs may take place.

We assume that $Y_{t}$ is conditionally independent of all other variables, given the state $X_{t}$. This means that the construction of the augmented process $P$ is relatively straightforward; we can simply multiply $P_{\mathcal{Y}|\mathcal{X}}(\cdot\,|\,X_{t})$ with any distribution $P_{\mathcal{X}}(X_{t},\cdot)$ that includes $X_{t}$ to obtain the joint distribution including $Y_{t}$: for any $t\in u$ and $v\in\mathcal{U}$ such that $t\notin v$, any $x_{t}\in\mathcal{X}_{t}$ and $x_{v}\in\mathcal{X}_{v}$, and any $O_{t}\in\Sigma$:

[TABLE]

Similarly, when considering multiple output observations at once—say for the entire sequence $u$—then for any $v\in\mathcal{U}$ such that $u\cap v=\emptyset$, any $x_{u}\in\mathcal{X}_{u}$ and $x_{v}\in\mathcal{X}_{v}$, and any $O_{u}\in\Sigma_{u}$:

[TABLE]

Other probabilities can be derived by appropriate marginalisation. We denote the resulting augmented stochastic process as $P=P_{\mathcal{Y}|\mathcal{X}}\otimes P_{\mathcal{X}}$, for the specific output model $P_{\mathcal{Y}|\mathcal{X}}$ and stochastic process $P_{\mathcal{X}}$ that were taken to be fixed in this section.

3.3 Imprecise Continuous-Time Hidden Markov Chains

An imprecise continuous-time hidden Markov chain (ICTHMC) is a set of augmented stochastic processes, obtained by augmenting all processes in an ICTMC with some given output model.

Definition 2

Consider any ICTMC $\mathbb{P}_{\mathcal{Q},\mathcal{M}}$, and any output model $(\mathcal{Y},\Sigma,P_{\mathcal{Y}|\mathcal{X}})$. Then, the corresponding imprecise continuous-time hidden Markov chain (ICTHMC) $\mathcal{Z}$ is the set of augmented stochastic processes that is defined by $\mathcal{Z}:\!=\left\{P_{\mathcal{Y}|\mathcal{X}}\otimes P_{\mathcal{X}}\,:\,P_{\mathcal{X}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}\right\}$. The lower expectation with respect to $\mathcal{Z}$ will be denoted by $\underline{\mathbb{E}}_{\mathcal{Z}}$.

Note that we leave the parameters $\mathcal{M}$, $\mathcal{Q}$ and $P_{\mathcal{Y}|\mathcal{X}}$ implicit in the notation of the ICTHMC $\mathcal{Z}$—we will henceforth take these parameters to be fixed.

Also, the output model is taken to be precise, and shared by all processes in the set. One further generalisation that we aim to make in the future is to allow for an imprecise specification of this output model. However, this would force us into choosing an appropriate notion of independence; e.g., whether to enforce the independence assumptions made in Section 3.2, leading to strong or complete independence, or to only enforce the lower envelopes to have these independence properties, leading to epistemic irrelevance. It is currently unclear which choice should be preferred, e.g. with regard to computability, so at present we prefer to focus on this simpler model.

4 Updating the Model

Suppose now that we have observed that some event $(Y_{u}\in O_{u})$ has taken place, with $O_{u}\in\Sigma_{u}$. We here use the terminology that we update our model with these observations, after which the updated model reflects our revised beliefs about some quantity of interest. These updated beliefs, about some function $f\in\mathcal{L}(\mathcal{X}_{v})$, say, are then denoted by $\mathbb{E}_{P}[f(X_{v})\,|\,Y_{u}\in O_{u}]$ or $\underline{\mathbb{E}}_{\mathcal{Z}}[f(X_{v})\,|\,Y_{u}\in O_{u}]$, depending on whether we are considering a precise or an imprecise model. In this section, we provide definitions and alternative expressions for such updated (lower) expectations.

4.1 Observations with Positive (Upper) Probability

When our assertion $(Y_{u}\in O_{u})$ about an observation at time points $u$ has positive probability, we can—in the precise case—update our model by application of Bayes’ rule. The following gives a convenient expression for the updated expectation $\mathbb{E}_{P}[f(X_{v})\,|\,Y_{u}\in O_{u}]$, which makes use of the independence assumptions in Section 3.2 for augmented stochastic processes.

Proposition 3

Let $P$ be an augmented stochastic process and consider any $u,v\in\mathcal{U}$, $O_{u}\in\Sigma_{u}$ and $f\in\mathcal{L}(\mathcal{X}_{v})$. Then the updated expectation is given by

[TABLE]

whenever $P(Y_{u}\in O_{u})=\mathbb{E}_{P_{\mathcal{X}}}[P_{\mathcal{Y}|\mathcal{X}}(O_{u}\,|\,X_{u})]>0$, and is left undefined, otherwise.

Having defined above how to update all the precise models $P\in\mathcal{Z}$, we will now update the imprecise model through regular extension (Walley, 1991). This corresponds to simply discarding from $\mathcal{Z}$ those precise models that assign zero probability to $(Y_{u}\in O_{u})$, updating the remaining models, and then computing their lower envelope.

Definition 4

Let $\mathcal{Z}$ be an ICTHMC and consider any $u,v\in\mathcal{U}$, $O_{u}\in\Sigma_{u}$ and $f\in\mathcal{L}(\mathcal{X}_{v})$. Then the updated lower expectation is defined by

[TABLE]

whenever $\overline{P}_{\mathcal{Z}}(Y_{u}\in O_{u})=\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[P_{\mathcal{Y}|\mathcal{X}}(O_{u}\,|\,X_{u})]>0$, and is left undefined, otherwise.

As is well known, the updated lower expectation that is obtained through regular extension satisfies Walley’s generalised Bayes’ rule (Walley, 1991). The following proposition gives an expression for this generalised Bayes’ rule, rewritten using some of the independence properties of the model. We will shortly see why this expression is useful from a computational perspective.

Proposition 5

Let $\mathcal{Z}$ be an ICTHMC and consider any $u,v\in\mathcal{U}$, $O_{u}\in\Sigma_{u}$ and $f\in\mathcal{L}(\mathcal{X}_{v})$. Then, if $\overline{P}_{\mathcal{Z}}(Y_{u}\in O_{u})=\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[P_{\mathcal{Y}|\mathcal{X}}(O_{u}\,|\,X_{u})]>0$, the quantity \underline{\mathbb{E}}_{\mathcal{Z}}\bigl{[}f(X_{v})\,|\,Y_{u}\in O_{u}\bigr{]} satisfies

[TABLE]

4.2 Uncountable Outcome Spaces, Point Observations, and Probability Zero

An important special case where observations have probability zero for all precise models, but where we can still make informative inferences, is when we have an uncountable outcome space $\mathcal{Y}$ and the observations are points $y_{u}\in\mathcal{Y}_{u}$—i.e., when $Y_{u}$ is continuous. In this case, it is common practice to define the updated expectation $\mathbb{E}_{P}[f(X_{v})\,|\,Y_{u}=y_{u}]$ as a limit of conditional expectations, where each conditioning event is an increasingly smaller region around this point $y_{u}$. We will start by formalising this idea in a relatively abstract way, but will shortly make this practicable. For the sake of intuition, note that we are working towards the introduction of probability density functions.

Fix any $P\in\mathcal{Z}$, consider any $y_{u}\in\mathcal{Y}_{u}$ and choose a sequence $\{O_{u}^{i}\}_{i\in\mathbb{N}}$ of events in $\Sigma_{u}$ which shrink to $y_{u}$—i.e., such that $O_{u}^{i}\supseteq O_{u}^{i+1}$ for all $i\in\mathbb{N}$, and such that $\cap_{i\in\mathbb{N}}O_{u}^{i}=\{y_{u}\}$. We then define

[TABLE]

This limit exists if there is a sequence $\{\lambda_{i}\}_{i\in\mathbb{N}}$ in $\mathbb{R}_{>0}$ such that, for every $x_{u}\in\mathcal{X}_{u}$, the limit

[TABLE]

exists, is real-valued—in particular, finite—and satisfies $\mathbb{E}_{P_{\mathcal{X}}}[\phi_{u}(y_{u}\,|\,X_{u})]>0$:

Proposition 6

Let $P$ be an augmented stochastic process and consider any $u,v\in\mathcal{U}$, $y_{u}\in\mathcal{Y}_{u}$ and $f\in\mathcal{L}(\mathcal{X}_{v})$. For any $\{O_{u}^{i}\}_{i\in\mathbb{N}}$ in $\Sigma_{u}$ that shrinks to $y_{u}$, if for some $\{\lambda_{i}\}_{i\in\mathbb{N}}$ in $\mathbb{R}_{>0}$ the quantity $\phi_{u}(y_{u}\,|\,X_{u})$ exists, is real-valued, and satisfies $\mathbb{E}_{P_{\mathcal{X}}}[\phi_{u}(y_{u}\,|\,X_{u})]>0$, then

[TABLE]

Note that $\mathbb{E}_{P}[f(X_{v})\,|\,Y_{u}=y_{u}]$ is clearly dependent on the exact sequence $\{O_{u}^{i}\}_{i\in\mathbb{N}}$. Unfortunately, this is the best we can hope for at the level of generality that we are currently dealing with. For brevity, we nevertheless omit from the notation the updated expectation’s dependency on this sequence. However, as we will explain below, this should not be problematic for most practical applications.

It is also useful to note that $\phi_{u}(y_{u}|x_{u})$ can often be constructed “piecewise”. That is, if for every $t\in u$ there is a sequence $\{\lambda_{\,t,i}\}_{i\in\mathbb{N}}$ in $\mathbb{R}_{>0}$ such that, for all $x_{t}\in\mathcal{X}_{t}$,

[TABLE]

exists and is real-valued, then choosing $\{\lambda_{i}\}_{i\in\mathbb{N}}$ as $\lambda_{i}:\!=\prod_{t\in u}\lambda_{\,t,i}$ yields $\phi_{u}(y_{u}|x_{u})=\prod_{t\in u}\phi_{t}(y_{t}|x_{t})$.

Now, to make the above practicable, we can for example assume that if $\mathcal{Y}$ is uncountable, then it is the set $\mathcal{Y}=\mathbb{R}^{d}$, for some $d\in\mathbb{N}$, and that $\Sigma$ is the Borel $\sigma$-algebra on $\mathbb{R}^{d}$. For each $x\in\mathcal{X}$, we then assume that the measure $P_{\mathcal{Y}|\mathcal{X}}(\cdot\,|x)$ is induced by some given probability density function: a measurable function $\psi(\cdot\,|x):\mathcal{Y}\to\mathbb{R}_{\geq 0}$ such that $\int_{\mathcal{Y}}\psi(y|x)\,\mathrm{d}y=1$ and, for every $O\in\Sigma$,

[TABLE]

where the integrals are understood in the Lebesgue sense.

Then choose any $y_{u}\in\mathcal{Y}_{u}$, any $t\in u$, any sequence $\{O_{t}^{i}\}_{i\in\mathbb{N}}$ of open balls in $\mathcal{Y}_{t}$ that are centred on, and shrink to, $y_{t}$, and fix any $x_{u}\in\mathcal{X}_{u}$. If $\psi(\cdot|x_{t})$ is continuous at $y_{t}$, it can be shown that

[TABLE]

where $\lambda(O_{t}^{i})$ denotes the Lebesgue measure of $O_{t}^{i}$. So, we can construct the sequence $\{O_{u}^{i}\}_{i\in\mathbb{N}}$ such that every $O_{u}^{i}:\!=\prod_{t\in u}O_{t}^{i}$, with each $O_{t}^{i}$ chosen as above. If we then choose the sequence $\{\lambda_{i}\}_{i\in\mathbb{N}}$ as $\lambda_{i}:\!=\prod_{t\in u}\lambda(O_{t}^{i})$ for each $i\in\mathbb{N}$, we find $\phi_{u}(y_{u}|x_{u})=\prod_{t\in u}\phi_{t}(y_{t}|x_{t})=\prod_{t\in u}\psi(y_{t}|x_{t})$, provided that each $\phi_{t}(y_{t}|x_{t})$ satisfies Equation (7). It can now be seen that, under these assumptions, the right-hand side of Equation (6) is simply the well-known Bayes’ rule for (finite) mixtures of densities.

In most practical applications, therefore, the function $\phi_{u}(\cdot\,|\,x_{u})$ is known explicitly; one may assume, for example, that $Y_{t}$ follows a Normal distribution with parameters depending on $X_{t}$, and the functions $\phi_{t}(\cdot\,|\,x_{t})$—and by extension, $\phi_{u}(\cdot|x_{u})$—then follow directly by identification with $\psi(\cdot\,|x_{t})$. Furthermore, arguably, most of the density functions that one encounters in practice will be continuous and strictly positive at $y_{t}$. This guarantees that the limit in Equation (7) exists, and largely solves the interpretation issue mentioned above: when $\phi_{u}(y_{u}|X_{u})=\prod_{t\in u}\psi(y_{t}|X_{t})$ is continuous and positive at $y_{u}$, $\mathbb{E}_{P}[f(X_{v})\,|\,Y_{u}=y_{u}]$ exists and is the same for almost333 It suffices if, for all $t\in u$, there is a sequence of open balls $\{B_{t}^{i}\}_{i\in\mathbb{N}}$ in $\mathcal{Y}$ that shrinks to $y_{t}$ such that, for all $i\in\mathbb{N}$, $O_{t}^{i}$ has positive Lebesgue measure and is contained in $B_{t}^{i}$. all sequences $\{O_{u}^{i}\}_{i\in\mathbb{N}}$.

Moving on, note that if $\phi_{u}(y_{u}|X_{u})$ exists and satisfies $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\phi_{u}(y_{u}|X_{u})]>0$, then the updated expectation $\mathbb{E}_{P}[f(X_{v})\,|\,Y_{u}=y_{u}]$ is well-defined for every $P\in\mathcal{Z}$. Hence, we can then update the imprecise model by updating each of the precise models that it consists of.

Definition 7

Let $\mathcal{Z}$ be an ICTHMC and consider any $u,v\in\mathcal{U}$, $y_{u}\in\mathcal{Y}_{u}$, and $f\in\mathcal{L}(\mathcal{X}_{v})$. For any $\{O_{u}^{i}\}_{i\in\mathbb{N}}$ in $\Sigma_{u}$ that shrinks to $y_{u}$, if for some $\{\lambda_{i}\}_{i\in\mathbb{N}}$ in $\mathbb{R}_{>0}$ the quantity $\phi_{u}(y_{u}\,|\,X_{u})$ exists and is real-valued, the updated lower expectation is defined by

[TABLE]

whenever $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\phi_{u}(y_{u}|X_{u})]>0$, and is left undefined, otherwise.

Similar to the results in Section 4.1, this updated lower expectation satisfies a “generalised Bayes’ rule for mixtures of densities”, in the following sense.

Proposition 8

Let $\mathcal{Z}$ be an ICTHMC and consider any $u,v\in\mathcal{U}$, $y_{u}\in\mathcal{Y}_{u}$ and $f\in\mathcal{L}(\mathcal{X}_{v})$. For any $\{O_{u}^{i}\}_{i\in\mathbb{N}}$ in $\Sigma_{u}$ that shrinks to $y_{u}$, if for some $\{\lambda_{i}\}_{i\in\mathbb{N}}$ in $\mathbb{R}_{>0}$ the quantity $\phi_{u}(y_{u}\,|\,X_{u})$ exists, is real-valued, and satisfies $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\phi_{u}(y_{u}|X_{u})]>0$, then

[TABLE]

Furthermore, this updated imprecise model can be given an intuitive limit interpretation.

Proposition 9

Let $\mathcal{Z}$ be an ICTHMC and consider any $u,v\in\mathcal{U}$, $y_{u}\in\mathcal{Y}_{u}$ and $f\in\mathcal{L}(\mathcal{X}_{v})$. For any $\{O_{u}^{i}\}_{i\in\mathbb{N}}$ in $\Sigma_{u}$ that shrinks to $y_{u}$, if for some $\{\lambda_{i}\}_{i\in\mathbb{N}}$ in $\mathbb{R}_{>0}$ the quantity $\phi_{u}(y_{u}\,|\,X_{u})$ exists, is real-valued, and satisfies $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\phi_{u}(y_{u}|X_{u})]>0$, then $\underline{\mathbb{E}}_{\mathcal{Z}}[f(X_{v})|Y_{u}=y_{u}]=\lim_{i\to+\infty}\underline{\mathbb{E}}_{\mathcal{Z}}[f(X_{v})|Y_{u}\in O_{u}^{i}]$.

Now, recall that the requirement $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\phi_{u}(y_{u}|X_{u})]>0$ for updating the imprecise model is a sufficient condition to guarantee that all the precise updated models are well-defined. However, one may wonder whether it is also possible to update the imprecise model under weaker conditions. Indeed, one obvious idea would be to define the updated model more generally as

[TABLE]

whenever $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\phi_{u}(y_{u}|X_{u})]>0$; this guarantees that some of the precise updated models are well-defined. This updated lower expectation satisfies the same generalised Bayes’ rule as above, i.e. the right-hand side of Equation (8) is equal to $\underline{\mathbb{E}}_{\mathcal{Z}}^{\mathrm{R}}[f(X_{v})\,|\,Y_{u}=y_{u}]$ whenever $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\phi_{u}(y_{u}|X_{u})]>0$. However, the limit interpretation then fails to hold, in the sense that it is possible to construct an example where $\underline{\mathbb{E}}_{\mathcal{Z}}^{\mathrm{R}}[f(X_{v})\,|\,Y_{u}=y_{u}]\neq\lim_{i\to+\infty}\underline{\mathbb{E}}_{\mathcal{Z}}[f(X_{v})\,|\,Y_{u}\in O_{u}^{i}]$, with $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\phi_{u}(y_{u}|X_{u})]>0$ but $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\phi_{u}(y_{u}|X_{u})]=0$. We feel that this makes this more general updating scheme somewhat troublesome from an interpretation (and hence philosophical) point of view.

On the other hand, we recall that the existence of $\phi_{u}(y_{u}|X_{u})$ and the positivity of $\mathbb{E}_{P_{\mathcal{X}}}[\phi_{u}(y_{u}|X_{u})]$ are necessary and sufficient conditions for the limit in Equation (5) to exist and be computable using Equation (6). However, these conditions are sufficient but non-necessary for that limit to simply exist. Therefore, a different way to generalise the imprecise updating method would be

[TABLE]

whenever \{P\in\mathcal{Z}\,:\,\leavevmode\nobreak\ \text{\mathbb{E}{P}[f(X{v}),|,Y_{u}=y_{u}] exists}\}\neq\emptyset. We conjecture that this updated model does satisfy the limit interpretation, but on the other hand, it is possible to show that this, in turn, no longer satisfies the above generalised Bayes’ rule. That makes this updating scheme somewhat troublesome from a practical point of view because, as we discuss below, the expression in Equation (8) is crucial for our method of efficient computation of the updated lower expectation.

5 Inference Algorithms

In the previous section, we have seen that we can use the generalised Bayes’ rule for updating our ICTHMC with some given observations. From a computational point of view, this is particularly useful because, rather than having to solve the non-linear optimisation problems in Definitions 4 or 7 directly, we can focus on evaluating the function \underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\bigl{[}P_{\mathcal{Y}|\mathcal{X}}(O_{u}|X_{u})\bigl{(}f(X_{v})-\mu\bigr{)}\bigr{]}, or its density-analogue, for some fixed value of $\mu$. Finding the updated lower expectation is then a matter of finding the maximum value of $\mu$ for which this quantity is non-negative. As we will discuss in Section 5.1, this is a relatively straightforward problem to solve numerically.

Therefore, in order for this approach to be computationally tractable, we require efficient algorithms that can evaluate this quantity for a given value of $\mu$. In Section 5.2, we provide such an algorithm for the important case where the function $f$ depends on a single time-point.

We first generalise the problem so that these results are applicable both for observations of the form $(Y_{u}\in O_{u})$, and for point-observations $(Y_{u}=y_{u})$ in an uncountable outcome space. Recall that

[TABLE]

In both cases, we can rewrite this expression as $\prod_{t\in u}g_{t}(X_{t})$, where, for all $t\in u$, $g_{t}\in\mathcal{L}(\mathcal{X}_{t})$ and $g_{t}\geq 0$. The function of interest is then \underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\left[\bigl{(}\prod_{t\in u}g_{t}(X_{t})\bigr{)}\bigl{(}f(X_{v})-\mu\bigr{)}\right] and the sign conditions in Propositions 5 and 8 reduce to $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\prod_{t\in u}g_{t}(X_{t})]>0$ and $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\prod_{t\in u}g_{t}(X_{t})]>0$, respectively.

5.1 Solving the Generalised Bayes’ Rule

Finding the maximum value of $\mu$ for which the function of interest in the generalised Bayes’ rule is non-negative, is relatively straightforward numerically. This is because this function, parameterised in $\mu$, is very well-behaved. The proposition below explicitly states some of its properties. These are essentially well-known, and can also be found in other work; see, e.g., (De Bock, 2015, Section 2.7.3). The statement below is therefore intended to briefly recall these properties, and is stated in a general form where we can also use it when working with densities.

Proposition 10

Let $\mathbb{P}_{\mathcal{Q},\mathcal{M}}$ be an ICTMC and consider any $u,v\in\mathcal{U}$, any $f\in\mathcal{L}(\mathcal{X}_{v})$ and, for all $t\in u$, any $g_{t}\in\mathcal{L}(\mathcal{X}_{t})$ such that $g_{t}\geq 0$. Consider the function $G:\mathbb{R}\to\mathbb{R}$ that is given, for all $\mu\in\mathbb{R}$, by G(\mu):\!=\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\left[\left(\prod_{t\in u}g_{t}(X_{t})\right)\bigl{(}f(X_{v})-\mu\bigr{)}\right]. Then the following properties hold:

1. G1:

$G$* is continuous, non-increasing, concave, and has a root, i.e. $\exists\mu\in\mathbb{R}:G(\mu)=0$.* 2. G2:

If \underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\bigl{[}\prod_{t\in u}g_{t}(X_{t})\bigr{]}>0, then $G$ is (strictly) decreasing, and has a unique root. 3. G3:

If \underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\bigl{[}\prod_{t\in u}g_{t}(X_{t})\bigr{]}=0 but \overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\bigl{[}\prod_{t\in u}g_{t}(X_{t})\bigr{]}>0, then $G$ has a maximum root $\mu_{*}$, satisfies $G(\mu)=0$ for all $\mu\leq\mu_{*}$, and is (strictly) decreasing for $\mu>\mu_{*}$. 4. G4:

If \overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\bigl{[}\prod_{t\in u}g_{t}(X_{t})\bigr{]}=0, then $G$ is identically zero, i.e. $\forall\mu\in\mathbb{R}:G(\mu)=0$.

Note that the function $G$ in Proposition 10 can behave in three essentially different ways. These correspond to the cases where the observed event has strictly positive probability(/density) for all processes in the set; to where it only has positive probability(/density) for some processes; and to where it has zero probability(/density) for all processes. In the first two cases—which are the important ones to apply the generalised Bayes’ rule—the function is “well-behaved” enough to make finding its maximum root a fairly simple task. For instance, a standard bisection/bracketing algorithm can be applied here, known in this context as Lavine’s algorithm (Cozman, 1997).

We sketch this method below. First, note that due to Propositions 5 and 8, the maximum root will always be found in the interval $[\min f,\max f]$. The properties above therefore provide us with a way to check the sign conditions for updating. That is, for any $\mu>\max f$, we see that $G(\mu)<0$ if and only if $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\prod_{t\in u}g_{t}(X_{t})]>0$; similarly, for any $\mu<\min f$, we see that $G(\mu)>0$ if and only if $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\prod_{t\in u}g_{t}(X_{t})]>0$. Evaluating $G$ at such values of $\mu$ is therefore sufficient to check the sign conditions in Propositions 5 and 8.

The algorithm now starts by setting $\mu_{-}:\!=\min f$, and $\mu_{+}:\!=\max f$; if $G(\mu_{+})=0$, we know that $\mu_{+}$ is the quantity of interest. Otherwise, proceed iteratively in the following way. Compute the half-way point $\mu:\!=\nicefrac{{1}}{{2}}(\mu_{+}-\mu_{-})$; then, if $G(\mu)\geq 0$ set $\mu_{-}:\!=\mu$, otherwise set $\mu_{+}:\!=\mu$; then repeat. Clearly, the interval $[\mu_{-},\mu_{+}]$ still contains the maximum root after each step. The procedure can be terminated whenever $(\mu_{+}-\mu_{-})<\epsilon$, for some desired numerical precision $\epsilon>0$. Since the width of the interval is halved at each iteration, the runtime of this procedure is O\bigl{(}\log\{(\max f-\min f)\epsilon^{-1}\}\bigr{)}. Methods for improving the numerical stability of this procedure can be found in (De Bock, 2015, Section 2.7.3).

5.2 Functions on a Single Time Point

Having discussed an efficient method to find the maximum root of the function $G(\mu)$ in Section 5.1, it now remains to provide an efficient method to numerically evaluate this function for a given value of $\mu$. Clearly, any such method will depend on the choice of $f$.

We focus on a particularly useful special case, which can be used to compute the updated lower expectation of a function $f\in\mathcal{L}(\mathcal{X}_{s})$ on a single time point $s$, given observations at time points $u$. If $s\notin u$, then it will be notationally convenient to define $g_{s}:\!=f-\mu$, and to let $u^{\prime}:\!=u\cup\{s\}$. We can then simply focus on computing

[TABLE]

On the other hand, if $s=t$ for some $t\in u$, we let $u^{\prime}:\!=u$ and replace $g_{t}$ by $(f-\mu)g_{t}$. Clearly, the above equality then also holds; the point is simply to establish a uniform indexing notation over all time-points and functions. The right hand side of the above equality can now be computed using the following dynamic programming technique.

For all $t\in u^{\prime}$, we define auxiliary functions $g_{t}^{+},g_{t}^{-}\in\mathcal{L}(\mathcal{X}_{t})$, as follows. Writing $u^{\prime}=t_{0},\ldots,t_{n}$, let $g_{t_{n}}^{+}:\!=g_{t_{n}}^{-}:\!=g_{t_{n}}$. Next, for all $i\in\{0,\dots,n-1\}$ and all $x_{t_{i}}\in\mathcal{X}_{t_{i}}$, let

[TABLE]

and

[TABLE]

Clearly, backward recursion allows us to compute all these functions in a time-complexity order that is linear in the number of time points in $u^{\prime}$. Practically, at each step, computing the quantities $\underline{\mathbb{E}}_{\mathcal{Q}}[g_{t_{i+1}}^{+}(X_{t_{i+1}})\,|\,X_{t_{i}}=x_{t_{i}}]$ and $\overline{\mathbb{E}}_{\mathcal{Q}}[g_{t_{i+1}}^{-}(X_{t_{i+1}})\,|\,X_{t_{i}}=x_{t_{i}}]$ can be done using Equation (2) and the method described in Section 2.2. Due to the results in (Krak et al., 2016), each of these quantities is computable in polynomial time. So, the total complexity of computing all these functions is clearly also polynomial. We now have the following result.

Proposition 11

For all $t\in u^{\prime}$, let $g_{t}$, $g_{t}^{+}$ and $g_{t}^{-}$ be as defined above. Then the function of interest is given by $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\left[\prod_{t\in u^{\prime}}g_{t}(X_{t})\right]=\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\left[g_{t_{0}}^{+}(X_{t_{0}})\right]$. Also, $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\left[\prod_{t\in u^{\prime}}g_{t}(X_{t})\right]=\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\left[g_{t_{0}}^{-}(X_{t_{0}})\right]$.

So, in order to evaluate the function of interest, it remains to compute $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\left[g_{t_{0}}^{+}(X_{t_{0}})\right]$. Since $g_{t_{0}}^{+}$ is a function on a single time point $t_{0}$, this can again be done in polynomial time, using Equation (4).

6 Conclusions and Future Work

We considered the problem of performing inference with imprecise continuous-time hidden Markov chains; an extension of imprecise continuous-time Markov chains obtained by augmenting them with random output variables, which may be either discrete or continuous. Our main result is an efficient, polynomial runtime, algorithm to compute lower expectations of functions that depend on the state-space at any given time-point, given a collection of observations of the output variables.

In future work, we intend to further generalise this model, by also allowing for imprecise output variables. Furthermore, we also aim to develop algorithms for other inference problems, such as the problem of computing updated lower expectations of functions $f\in\mathcal{L}(\mathcal{X}_{v})$ that depend on more than one time-point. Similarly, we aim to investigate predictive output inferences, i.e., the lower probability/density of observations, which has uses in classification problems. Another such problem is that of estimating state-sequences given observed output-sequences—as was previously done for (discrete-time) iHMM’s (De Bock and de Cooman, 2014).

Acknowledgments

The work in this paper was partially supported by the Research Foundation - Flanders (FWO) and the H2020-MSCA-ITN-2016 UTOPIAE, grant agreement 722734. The authors would also like to thank three anonymous referees for their helpful comments and suggestions.

A Extended Preliminaries

We here provide some additional notation and previous results that we use throughout the proofs of our results.

A.1 Additional Notation

For any sequence of time points $u=\{t_{0},\ldots,t_{n}\}$, we write for any $t\in\mathbb{R}_{\geq 0}$ that $u<t$ whenever $t_{i}<t$ for all $i\in\{0,\ldots,n\}$. Similarly, for any $u,v\in\mathcal{U}$, we write $u<v$ when all time-points in $u$ are strictly less than all time-points in $v$.

Recall that for any $u\in\mathcal{U}$, we denote with $\mathcal{L}(\mathcal{X}_{u})$ the set of all real-valued functions on $\mathcal{X}_{u}$. We endow these function spaces with the $L^{\infty}$-norm, i.e. the norm $\left\lVert f\right\rVert$ of any $f\in\mathcal{L}(\mathcal{X}_{u})$ is defined to be $\left\lVert f\right\rVert:\!=\left\lVert f\right\rVert_{\infty}=\max\{\left|f(x_{u})\right|\,:\,x_{u}\in\mathcal{X}_{u}\}$. Limits of functions are to be interpreted under this norm.

Furthermore, we sometimes use the shorthand notation $\{a_{i}\}_{i\in\mathbb{N}}\to c$ for convergent sequences of quantities, which should be read as $\lim_{i\to+\infty}a_{i}=c$. If this limit is approached from above or below, we write $\{a_{i}\}_{i\in\mathbb{N}}\to c^{+}$ or $\{a_{i}\}_{i\in\mathbb{N}}\to c^{-}$, respectively.

A.2 A Useful Property of ICTMC’s

The below states a very useful property of the lower expectation corresponding to ICTMC’s, that we will require in some of our proofs.

Lemma 12** (Iterated Lower Expectation)**

(Krak et al., 2016, Theorem 6.5)* Let $\mathcal{Q}$ be a non-empty, bounded and convex set of rate matrices, and let $\mathcal{M}$ be a non-empty set of probability mass functions on $\mathcal{X}_{0}$. Let $\mathbb{P}_{\mathcal{Q},\mathcal{M}}$ denote the corresponding ICTMC. Let $u\subset\mathbb{R}_{\geq 0}$ be a finite (possibly empty) sequence of time-points, and consider any $v,w\in\mathcal{U}$ such that $u<v<w$. Choose any $f\in\mathcal{L}(\mathcal{X}_{u\cup v\cup w})$. Then,*

[TABLE]

B Globally Required Proofs and Lemmas

The following property will be useful. The result is rather trivial, but we note it here explicitly to prevent confusion when we use it in our proofs.

Lemma 13

Let $P_{\mathcal{X}}$ be a stochastic process and consider any $u\in\mathcal{U}$ and any $\{f_{i}\}_{i\in\mathbb{N}}\to f$ in $\mathcal{L}(\mathcal{X}_{u})$. Then $\lim_{i\to+\infty}\mathbb{E}_{P_{\mathcal{X}}}[f_{i}(X_{u})]=\mathbb{E}_{P_{\mathcal{X}}}[f(X_{u})]$.

**Proof ** Trivial consequence of the definition of our norm $\left\lVert\cdot\right\rVert$ on $\mathcal{L}(\mathcal{X}_{u})$.

The following lemma states the imprecise analogue of the above result; this is essentially well-known, but we repeat it here for the sake of completeness.

Lemma 14

Let $\mathbb{P}_{\mathcal{Q},\mathcal{M}}$ be an ICTMC and consider any $u\in\mathcal{U}$ and any $\{f_{i}\}_{i\in\mathbb{N}}\to f$ in $\mathcal{L}(\mathcal{X}_{u})$. Then $\lim_{i\to+\infty}\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[f_{i}(X_{u})]=\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[f(X_{u})]$.

**Proof ** Keeping $u$ fixed, then since $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\,\cdot\,]\,:\,\mathcal{L}(\mathcal{X}_{u})\to\mathbb{R}$ is an infimum over precise expectations $\mathbb{E}_{P_{\mathcal{X}}}[\,\cdot\,]\,:\,\mathcal{L}(\mathcal{X}_{u})\to\mathbb{R}$, with $P_{\mathcal{X}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$, we know from (Walley, 1991, Theorem 3.3.3) that $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\,\cdot\,]$ is a coherent lower prevision on $\mathcal{L}(\mathcal{X}_{u})$. Therefore, the statement follows directly from (Walley, 1991, Proposition 2.6.1.$\ell$) and the definition of our norm $\left\lVert\cdot\right\rVert$ on $\mathcal{L}(\mathcal{X}_{u})$.

We provide the proof of Proposition 10 below; this is not in chronological order with respect to the main text, but it states a number of convenient properties that are required in the proofs of statements that appear before Proposition 10. We first need the following lemma.

Lemma 15

Let $P_{\mathcal{X}}$ be a stochastic process and consider any $u,v\in\mathcal{U}$, any $f\in\mathcal{L}(\mathcal{X}_{v})$ and any $g\in\mathcal{L}(\mathcal{X}_{u})$ such that $g\geq 0$. If $\mathbb{E}_{P_{\mathcal{X}}}[g(X_{u})]=0$, then for all $\mu\in\mathbb{R}$ it holds that

[TABLE]

**Proof ** Because $\mathbb{E}_{P_{\mathcal{X}}}[g(X_{u})]=0$, and since $g\geq 0$, we must clearly have that

[TABLE]

for all $x_{u}\in\mathcal{X}_{u}$ for which $g(x_{u})\neq 0$. Let $\mathcal{X}_{u}^{0}:\!=\{x_{u}\in\mathcal{X}_{u}\,:\,g(x_{u})\neq 0\}$. Then clearly for any $x_{u}\in\mathcal{X}_{u}^{0}$ it holds for all $x_{v\setminus u}\in\mathcal{X}_{v\setminus u}$ that also $P_{\mathcal{X}}(X_{u}=x_{u},X_{v\setminus u}=x_{v\setminus u})=0$. Hence, for all $x_{u}\in\mathcal{X}_{u}$ and $x_{v\setminus u}\in\mathcal{X}_{v\setminus u}$, we find that $P_{\mathcal{X}}(X_{u}=x_{u},X_{v\setminus u}=x_{v\setminus u})g(x_{u})=0$. Therefore,

[TABLE]

**Proof of Proposition 10 ** For brevity, define $g\in\mathcal{L}(\mathcal{X}_{u})$ as $g(x_{u}):\!=\prod_{t\in u}g_{t}(x_{t})$ for all $x_{u}\in\mathcal{X}_{u}$.

We start by proving Property G1. For continuity, consider any $\mu\in\mathbb{R}$. We will prove that $G$ is continuous in $\mu$, or in other words, that for every sequence $\{\mu_{i}\}_{i\in\mathbb{N}}\to\mu$ it holds that $\lim_{i\to\infty}G(\mu_{i})=G(\mu)$. So, choose any sequence $\{\mu_{i}\}_{i\in\mathbb{N}}\to\mu$, and consider the induced sequence of functions \bigl{\{}g(X_{u})\bigl{(}f(X_{v})-\mu_{i}\bigr{)}\bigr{\}}_{i\in\mathbb{N}} in $\mathcal{L}(\mathcal{X}_{u\cup v})$. Then, since $\{\mu_{i}\}_{i\in\mathbb{N}}\to\mu$, clearly also \lim_{i\to+\infty}g(X_{u})\bigl{(}f(X_{v})-\mu_{i}\bigr{)}=g(X_{u})\bigl{(}f(X_{v})-\mu\bigr{)}. Using Lemma 14, we therefore find that

[TABLE]

or in other words, that $\lim_{i\to+\infty}G(\mu_{i})=G(\mu)$. Since the sequence $\{\mu_{i}\}_{i\in\mathbb{N}}$ was arbitrary, this concludes the proof.

We next prove that $G$ is non-increasing. To this end, fix any $P_{\mathcal{X}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$. Then, by the linearity of expectation operators,

[TABLE]

Since by assumption $g\geq 0$, it must hold that $\mathbb{E}_{P_{\mathcal{X}}}[g(X_{u})]\geq 0$, and so, that \mathbb{E}_{P_{\mathcal{X}}}\bigl{[}g(X_{u})\bigl{(}f(X_{v})-\mu\bigr{)}\bigr{]} is non-increasing in $\mu$. Since this is true for all ${P_{\mathcal{X}}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$, we find that $G(\mu)$ is a lower envelope of non-increasing functions, which must therefore be non-increasing itself. This concludes the proof.

For concavity, fix any $\mu,\nu\in\mathbb{R}$, and choose any $\lambda\in[0,1]$. Let $\mu^{\prime}:\!=\lambda\mu+(1-\lambda)\nu$. We need to show that $\lambda G(\mu)+(1-\lambda)G(\nu)\leq G(\mu^{\prime})$. To this end, fix any $\epsilon\in\mathbb{R}_{>0}$. Then, there is some ${P_{\mathcal{X}}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$ such that

[TABLE]

By expanding the convex combination $\mu^{\prime}$ using the linearity of expectation operators, we find that

[TABLE]

where the inequality follows from the fact that $\lambda$ and $(1-\lambda)$ are non-negative, and $P_{\mathcal{X}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$—hence in particular $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\leq\mathbb{E}_{P_{\mathcal{X}}}$. Combining with Equation (10), we find that

[TABLE]

Since the $\epsilon\in\mathbb{R}_{>0}$ was arbitrary, this concludes the proof.

To prove that the function has a root, first consider any $\mu<\min f$. Then, for every $P_{\mathcal{X}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$, using the assumption that $g\geq 0$,

[TABLE]

and hence, using Equation (9), we find that \mathbb{E}_{P_{\mathcal{X}}}\bigl{[}g(X_{u})\bigl{(}f(X_{v})-\mu\bigr{)}\bigr{]}\geq 0. Since this is true for all $P_{\mathcal{X}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$, we find that $G(\mu)\geq 0$ for this choice of $\mu$. Next, consider any $\nu>\max f$. Then by a completely analogous argument—just reverse the inequalities—we find that $G(\nu)\leq 0$ for this choice of $\nu$. Therefore, and since we already know that $G$ is continuous, by the intermediate value theorem there must now be some $\mu_{*}\in[\mu,\nu]$ such that $G(\mu_{*})=0$. This concludes the proof.

We next prove G2, and start by showing that $G$ is (strictly) decreasing if $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]>0$. To this end, consider any $\mu\in\mathbb{R}$ and any $\Delta\in\mathbb{R}_{>0}$. We need to show that $G(\mu)>G(\mu+\Delta)$. Let $\epsilon:\!=\Delta\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]$; clearly then $\epsilon>0$. Therefore, there is some $P_{\mathcal{X}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$ such that

[TABLE]

Expanding the left hand side as in Equation (9), we find

[TABLE]

where the first inequality follows from the fact that $\Delta$ and $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]$ are strictly positive, and \underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]\leq\mathbb{E}_{P_{\mathcal{X}}}\bigl{[}g(X_{u})\bigr{]} since $P_{\mathcal{X}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$. Combining with Equation (11) shows that

[TABLE]

which concludes the proof.

To prove that $G$ has a unique root under the assumption that $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]>0$, note that we already know that $G$ has at least one root, i.e. $G(\mu)=0$ for some $\mu\in\mathbb{R}$. By combining with the fact that $G$ is strictly decreasing under the assumption that $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]>0$, the uniqueness of this root follows immediately.

We next prove G3. Lemma 16 below states that $G$ has a maximum root if $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]>0$, so we do not need to prove this here. So, let $\mu_{*}:\!=\max\{\mu\in\mathbb{R}\,:\,G(\mu)\geq 0\}$ be this maximum root.

We will now prove that $G(\mu)=0$ for all $\mu\leq\mu_{*}$ when $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]=0$ but $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]>0$. So, consider any $\mu<\mu_{*}$ (the case for $\mu=\mu_{*}$ is trivial). Since $\mu<\mu_{*}$, we already know that $G(\mu)$ is non-negative because $G$ is non-increasing; so, it suffices to show that $G(\mu)$ is non-positive. Now, for every $P_{\mathcal{X}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$ we have that

[TABLE]

where the first inequality follows from the fact that $g\geq 0$, the second inequality is due to the definition of the norm $\left\lVert\cdot\right\rVert$, and the second equality follows from the law of total probability (that is, $X_{v\setminus u}$ is marginalised out). Since G(\mu)\leq\mathbb{E}_{P_{\mathcal{X}}}\bigl{[}g(X_{u})\bigl{(}f(X_{v})-\mu\bigr{)}\bigr{]}, it follows from the above that if $\left\lVert f-\mu\right\rVert=0$ then $G(\mu)\leq 0$, in which case we are done.

Hence, we can assume without loss of generality that $\left\lVert f-\mu\right\rVert>0$. Choose any $\epsilon\in\mathbb{R}_{>0}$. Then, because $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]=0$, there is some $P_{\mathcal{X}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$ such that $\mathbb{E}_{P_{\mathcal{X}}}[g(X_{u})]<\nicefrac{{\epsilon}}{{\left\lVert f-\mu\right\rVert}}$, which implies that

[TABLE]

using Equation (12) for the second inequality. Since the $\epsilon\in\mathbb{R}_{>0}$ was arbitrary, this implies that $G(\mu)$ is non-positive, which concludes the proof.

We next show that $G$ is strictly decreasing for $\mu>\mu_{*}$. So consider any $\mu\in\mathbb{R}$ such that $\mu>\mu_{*}$, and any $\Delta\in\mathbb{R}_{>0}$; we need to show that $G(\mu)>G(\mu+\Delta)$. Because $\mu>\mu_{*}$, and since $\mu_{*}=\max\{\mu\in\mathbb{R}\,:\,G(\mu)\geq 0\}$, we know that $G(\mu)<0$.

First note that for any $\epsilon\in\mathbb{R}_{>0}$, there is some $P_{\mathcal{X}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$ for which

[TABLE]

and clearly also

[TABLE]

Therefore,

[TABLE]

and so, negating both sides and noting again that $G(\mu)<0$,

[TABLE]

and dividing through,

[TABLE]

for any $P_{\mathcal{X}}$ that $\epsilon$-approaches $G(\mu)$.

The idea is now the following: Equation (13) provides a lower bound on the (absolute magnitude of the) slope of the function \mathbb{E}_{P_{\mathcal{X}}}\bigl{[}g(X_{u})\bigl{(}f(X_{v})-\mu\bigr{)}\bigr{]} for any $P_{\mathcal{X}}$ that $\epsilon$-approaches $G(\mu)$. Since this function can still be a distance of $\epsilon$ above $G(\mu)$, we now need to make sure that the slope is such that this function will decrease by more than $\epsilon$ after increasing $\mu$ by $\Delta$; this will guarantee that this function becomes strictly lower than $G(\mu)$ when evaluated at $\mu+\Delta$, and since it is an upper bound on $G(\mu+\Delta)$, this will complete the proof.

Note that this lower bound increases as we decrease $\epsilon$. In order to ensure that we get a large enough slope, we now solve the following for $\epsilon$:

[TABLE]

So, choose any $\epsilon\in\mathbb{R}_{>0}$ such that $\epsilon<\nicefrac{{\Delta\left|G(\mu)\right|}}{{(\Delta+\mu-\mu_{*})}}$; since the right-hand side is clearly strictly positive, this is always possible. Then there is some $P_{\mathcal{X}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$ such that

[TABLE]

For this same $P_{\mathcal{X}}$, we then have

[TABLE]

where the second inequality is by Equation (13), the third inequality is by Equation (14), and the final inequality is by the choice of $P_{\mathcal{X}}$. So, we have found that indeed $G(\mu)>G(\mu+\Delta)$, which concludes the proof.

We finally prove G4, i.e. that $G(\mu)$ is identically zero whenever $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]=0$. Clearly, this assumption implies that $\mathbb{E}_{P_{\mathcal{X}}}[g(X_{u})]=0$ for all $P_{\mathcal{X}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$. Therefore, and since $g\geq 0$, it follows from Lemma 15 that for any $\mu\in\mathbb{R}$, it holds that \mathbb{E}_{P_{\mathcal{X}}}\bigl{[}g(X_{u})\bigl{(}f(X_{v})-\mu\bigr{)}\bigr{]}=0 for all $P_{\mathcal{X}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$. It immediately follows that indeed $G(\mu)=0$ for all $\mu\in\mathbb{R}$.

The following lemma states a more general version of the result that the generalised Bayes’ rule computes the updated lower expectation of a model under regular extension. We state it here because we use the result for various proofs throughout this appendix.

Lemma 16

Let $\mathbb{P}_{\mathcal{Q},\mathcal{M}}$ be an ICTMC, and consider any $u,v\in\mathcal{U}$, $f\in\mathcal{L}(\mathcal{X}_{v})$ and $g\in\mathcal{L}(\mathcal{X}_{u})$ such that $g\geq 0$. Then, if $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]>0$, it holds that

[TABLE]

**Proof ** Let $\mathcal{P}:\!=\left\{{P_{\mathcal{X}}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}\,:\,\mathbb{E}_{P_{\mathcal{X}}}[g(X_{u})]>0\right\}$, and note that $\mathcal{P}$ is non-empty due to the assumption that $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]>0$. For all ${P_{\mathcal{X}}}\in\mathcal{P}$, define

[TABLE]

and let $\mu_{*}$ be defined by

[TABLE]

Now define the following function, parameterised in $\mu\in\mathbb{R}$,

[TABLE]

and consider $\underline{\mathbb{E}}[g(X_{u})(f(X_{v})-\mu_{*})]$. We start by showing that this quantity is non-negative. To this end, fix any $\epsilon>0$. Then, there is some ${P_{\mathcal{X}}}\in\mathcal{P}$ such that

[TABLE]

Using Equation (9), the function $\mathbb{E}_{P_{\mathcal{X}}}[g(X_{u})(f(X_{v})-\mu)]$ is non-increasing in $\mu$. Therefore, and since $\mu_{*}\leq\mu_{P_{\mathcal{X}}}$, we have

[TABLE]

Due to the choice of $\mu_{P_{\mathcal{X}}}$, we have $\mathbb{E}_{P_{\mathcal{X}}}[g(X_{u})(f(X_{v})-\mu_{P_{\mathcal{X}}})]=0$, and so we find that

[TABLE]

Since this is true for every $\epsilon>0$, we conclude that $0\leq\underline{\mathbb{E}}[g(X_{u})(f(X_{v})-\mu_{*})]$. Next, we show that this quantity is also non-positive, or in other words, that $\mu_{*}$ is a root of this function.

To this end, fix any $\epsilon>0$, and define $\epsilon^{\prime}:\!=\nicefrac{{\epsilon}}{{\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]}}$; since by assumption $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]>0$, we have $\epsilon^{\prime}>0$. Now consider ${P_{\mathcal{X}}}\in\mathcal{P}$ such that

[TABLE]

Then, since $\underline{\mathbb{E}}[g(X_{u})(f(X_{v})-\mu)]$ is non-increasing in $\mu$—because it is a lower envelope of non-increasing functions—we have that

[TABLE]

Expanding the r.h.s. using the linearity of expectation operators, and by the definition of $\mu_{P_{\mathcal{X}}}$, we then have

[TABLE]

and since ${P_{\mathcal{X}}}\in\mathcal{P}\subseteq\mathbb{P}_{\mathcal{Q},\mathcal{M}}$,

[TABLE]

and so we find that

[TABLE]

Since this is true for every $\epsilon>0$, and since we already know that $\underline{\mathbb{E}}[g(X_{u})(f(X_{v})-\mu_{*})]$ is non-negative, we conclude that

[TABLE]

Now consider any $\mu^{\prime}>\mu_{*}$. There must then be some ${P_{\mathcal{X}}}\in\mathcal{P}$ such that $\mu_{*}\leq\mu_{P_{\mathcal{X}}}<\mu^{\prime}$, and furthermore,

[TABLE]

where the strict inequality follows from the fact that $\mathbb{E}_{P_{\mathcal{X}}}[g(X_{u})(f(X_{v})-\mu)]$ is strictly decreasing, and $\mu_{P_{\mathcal{X}}}<\mu^{\prime}$. Since this is true for every $\mu^{\prime}>\mu_{*}$, we conclude that

[TABLE]

Hence, because of our definition for $\mu_{*}$, we are left to prove that

[TABLE]

If $\mathcal{P}=\mathbb{P}_{\mathcal{Q},\mathcal{M}}$, this is trivially true. Therefore, we can assume without loss of generality that $\mathcal{P}\neq\mathbb{P}_{\mathcal{Q},\mathcal{M}}$. Let $\mathcal{P}_{0}:\!=\mathbb{P}_{\mathcal{Q},\mathcal{M}}\setminus\mathcal{P}$. Due to the definition of $\mathcal{P}$, we then have for every $P_{\mathcal{X}}\in\mathcal{P}_{0}$ that $\mathbb{E}_{P_{\mathcal{X}}}[g(X_{u})]=0$. It follows from Lemma 15 that \mathbb{E}_{P_{\mathcal{X}}}\bigl{[}g(X_{u})\bigl{(}f(X_{v})-\mu\bigr{)}\bigr{]}=0 for all $\mu\in\mathbb{R}$ and all $P_{\mathcal{X}}\in\mathcal{P}_{0}$. Hence,

[TABLE]

Because $\mathcal{P}\cup\mathcal{P}_{0}=\mathbb{P}_{\mathcal{Q},\mathcal{M}}$, we therefore have

[TABLE]

and so we conclude that \underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\bigl{[}g(X_{u})\bigl{(}f(X_{v})-\mu\bigr{)}\bigr{]} is negative if and only if $\underline{\mathbb{E}}[g(X_{u})(f(X_{v})-\mu)]$ is negative. This clearly implies Equation (15).

Corollary 17

Let $\mathcal{Z}$ be an ICTHMC, and consider any $u,v\in\mathcal{U}$, $f\in\mathcal{L}(\mathcal{X}_{v})$ and $g\in\mathcal{L}(\mathcal{X}_{u})$ such that $g\geq 0$. Then, if $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]>0$, it holds that

[TABLE]

**Proof ** Trivial consequence of Lemma 16 and Definition 2.

Due to the independence assumptions on augmented stochastic processes and the fact that we use a fixed output distribution $P_{\mathcal{Y}|\mathcal{X}}$, (lower) probabilities of outputs can be conveniently rewritten as follows.

Lemma 18

Let $P=P_{\mathcal{Y}|\mathcal{X}}\otimes P_{\mathcal{X}}$ be an augmented stochastic process and consider any $u\in\mathcal{U}$ and any $O_{u}\in\Sigma_{u}$. Then $P(Y_{u}\in O_{u})=\mathbb{E}_{P_{\mathcal{X}}}[P_{\mathcal{Y}|\mathcal{X}}(O_{u}|X_{u})]$.

**Proof ** Due to the construction of augmented stochastic processes in Section 3.2, we have that

[TABLE]

Lemma 19

Let $\mathcal{Z}$ be an ICTHMC with corresponding ICTMC $\mathbb{P}_{\mathcal{Q},\mathcal{M}}$ and consider any $u\in\mathcal{U}$ and $O_{u}\in\Sigma_{u}$. Then $\overline{P}_{\mathcal{Z}}(Y_{u}\in O_{u})=\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[P_{\mathcal{Y}|\mathcal{X}}(O_{u}|X_{u})]$.

**Proof ** Trivial consequence of Definition 2 and Lemma 18.

C Proofs of the Results in Section 4.1

**Proof of Proposition 3 ** The result follows from some simple manipulations, but we need to keep track of the overlap $u\cap v$ between the time-points of interest to prevent double-counting those time-points. Let $w:\!=u\setminus v$; then clearly $v\cup w=v\cup(u\setminus v)=u\cup(v\setminus u)=u\cup v$.

Starting from the definition of $\mathbb{E}_{P}[f(X_{v})\,|\,Y_{u}\in O_{u}]$ using Bayes’ rule, we find

[TABLE]

where the first equality is by definition, the second is by the basic rules of probability, the third is by changing the indexing using $v\cup w=u\cup(v\setminus u)$, and the fourth is by the definition of augmented stochastic processes in Section 3.2. Joining the indexes using $u\cup(v\setminus u)=u\cup v$, we find that

[TABLE]

where the third equality is by the definition of expectation. Applying Lemma 18 to the denominator, we finally obtain

[TABLE]

**Proof of Proposition 5 ** This is a special case of Corollary 17. To see this, set $g(X_{u}):\!=P_{\mathcal{Y}|\mathcal{X}}(O_{u}|X_{u})$ in that corollary’s statement, apply Proposition 3 to the quantities $\mathbb{E}_{P}[f(X_{v})\,|\,Y_{u}\in O_{u}]$ and apply Lemma 18 to the quantities $P(Y_{u}\in O_{u})$.

The hypothesis $\overline{P}_{\mathcal{Z}}(Y_{u}\in O_{u})>0$ then implies the hypothesis $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[g(X_{u})]>0$ of Corollary 17, due to Lemma 19.

D Proofs of the Results in Section 4.2

**Proof of Proposition 6 ** Assume that there is a sequence $\{\lambda_{i}\}_{i\in\mathbb{N}}$ in $\mathbb{R}_{>0}$ such that, for all $x_{u}\in\mathcal{X}_{u}$,

[TABLE]

exists, is real-valued, and satisfies $\mathbb{E}_{P_{\mathcal{X}}}[\phi_{u}(y_{u}|X_{u})]>0$.

Then, the existence of $\phi_{u}(y_{u}|X_{u})$ clearly implies that

[TABLE]

and so, using Lemma 13, we find that

[TABLE]

exists, and similarly that

[TABLE]

Furthermore, this latter inequality implies that there is some $n\in\mathbb{N}$ such that for all $i>n$,

[TABLE]

and hence in particular $\mathbb{E}_{P_{\mathcal{X}}}[P_{\mathcal{Y}|\mathcal{X}}(O_{u}^{i}|X_{u})]>0$ for all $i>n$. Furthermore, since by construction $O_{u}^{j}\supseteq O_{u}^{j+1}$ for all $j\in\mathbb{N}$, monotonicity of the measure $P_{\mathcal{Y}|\mathcal{X}}$ implies that $P_{\mathcal{Y}|\mathcal{X}}(O_{u}^{j}|x_{u})\geq P_{\mathcal{Y}|\mathcal{X}}(O_{u}^{j+1}|x_{u})$ for all $j\in\mathbb{N}$ and all $x_{u}\in\mathcal{X}_{u}$. So, for all $j\leq n$ we also have $\mathbb{E}_{P_{\mathcal{X}}}[P_{\mathcal{Y}|\mathcal{X}}(O_{u}^{j}|X_{u})]>0$, and so we have found that $\mathbb{E}_{P_{\mathcal{X}}}[P_{\mathcal{Y}|\mathcal{X}}(O_{u}^{i}|X_{u})]>0$ for all $i\in\mathbb{N}$. Due to Proposition 3, this implies that each $\mathbb{E}_{P}[f(X_{v})\,|\,Y_{u}\in O_{u}^{i}]$ is well-defined.

By the limit definition of $\mathbb{E}_{P}[f(X_{v})\,|\,Y_{u}=y_{u}]$, and applying Proposition 3 to each step,

[TABLE]

using the above established existence and properties of the individual limits for the penultimate step.

The below proves some of the properties that are claimed in the main text of Section 4.2. We first need the following result, which is essentially well-known, but which we repeat here for the sake of completeness.

Lemma 20

Fix $d\in\mathbb{N}$ and consider any absolutely integrable function $\psi:\mathbb{R}^{d}\to\mathbb{R}$. Then, for any $y\in\mathbb{R}^{d}$ and any sequence $\{B_{i}\}_{i\in\mathbb{N}}$ of open balls that are centred on, and shrink to, $y$, if $\psi$ is continuous at $y$ it holds that

[TABLE]

where the integral is understood in the Lebesgue sense, and where $\lambda(B_{i})$ denotes the Lebesgue measure of $B_{i}$.

**Proof ** Fix any $\epsilon>0$. We need to show that there is some $n\in\mathbb{N}$ such that, for all $i>n$, it holds that

[TABLE]

Now, because $\psi$ is continuous at $y$, there is some open ball $B$ that is centred on $y$, such that for all $\gamma\in B$, it holds that

[TABLE]

Furthermore, because the sequence $\{B_{i}\}_{i\in\mathbb{N}}$ is centred on, and shrinks to, $y$, there must be some $n\in\mathbb{N}$ such that for all $i>n$, it holds that $B_{i}\subset B$. Fix any such $i>n$. Then,

[TABLE]

**Proof of claims in Section 4.2 ** We start by proving the claim that $\phi_{u}(y_{u}|x_{u})$ exists and is real-valued if it can be constructed “piecewise”, as explained in the main text. So, fix any $y_{u}\in\mathcal{Y}_{u}$, choose any sequence $\{O_{u}^{i}\}_{i\in\mathbb{N}}$ in $\Sigma_{u}$ that shrinks to $y_{u}$, and suppose that for every $t\in u$, there is a sequence $\{\lambda_{\,t,i}\}_{i\in\mathbb{N}}$ in $\mathbb{R}_{>0}$ such that, for all $x_{t}\in\mathcal{X}_{t}$,

[TABLE]

exists and is real-valued. Recall that, for every $x_{u}\in\mathcal{X}_{u}$ and every $i\in\mathbb{N}$, we have $P_{\mathcal{Y}|\mathcal{X}}(O_{u}^{i}|x_{u})=\prod_{t\in u}P_{\mathcal{Y}|\mathcal{X}}(O_{t}^{i}|x_{t})$. So, by choosing $\{\lambda_{i}\}_{i\in\mathbb{N}}$ as $\lambda_{i}:\!=\prod_{t\in u}\lambda_{\,t,i}$, it follows that for every $x_{u}\in\mathcal{X}_{u}$,

[TABLE]

using the existence of the $\phi_{t}(y_{t}|x_{t})$ for the third equality. Hence, $\phi_{u}(y_{u}|x_{u})$ exists and, since each $\phi_{t}(y_{t}|x_{t})$, $t\in u$, is real-valued, so is $\phi_{u}(y_{u}|x_{u})$. This concludes the proof of this statement.

Next, we prove that the limit expression, and in particular the second equality, in Equation (7) are true when $\psi(\cdot|x_{t})$ is continuous (at $y_{t}$). To this end, note that $P_{\mathcal{Y}|\mathcal{X}}(\cdot\,|x_{t})$ was defined by

[TABLE]

for all $O\in\Sigma$. For a sequence of open balls $\{O_{t}^{i}\}_{i\in\mathbb{N}}$ that are centred on, and shrink to, $y_{t}$, we therefore need to prove that

[TABLE]

Because $\psi(y_{t}|x_{t})$ is by assumption continuous at $y_{t}$, this result follows immediately from Lemma 20.

We finally prove the claim that $\mathbb{E}_{P}[f(X_{v})\,|\,Y_{u}=y_{u}]$ is the same for almost every sequence $\{O_{u}^{i}\}_{i\in\mathbb{N}}$ that shrinks to $y_{u}$, provided that for all $x_{t}\in\mathcal{X}$, $P_{\mathcal{Y}|\mathcal{X}}(\cdot|x_{t})$ is constructed from a density $\psi(\cdot\,|x_{t})$ that is continuous and strictly positive at $y_{t}$. To this end, assume that $\{O_{u}^{i}\}_{i\in\mathbb{N}}$ satisfies the assumptions in Footnote 3; i.e. that for all $t\in u$, there is a sequence of open balls $\{B_{t}^{i}\}_{i\in\mathbb{N}}$ in $\mathcal{Y}$ that shrinks to $y_{t}$ such that, for all $i\in\mathbb{N}$, $\lambda(O_{t}^{i})>0$ and $O_{t}^{i}\subseteq B_{t}^{i}$.

We start by showing that

[TABLE]

is independent of the sequence $\{O_{u}^{i}\}_{i\in\mathbb{N}}$. We will prove this “piecewise”. In particular, we will show that for all $t\in u$,

[TABLE]

where we use the assumption $\lambda(O_{t}^{i})>0$ to ensure that each element of the sequence is well-defined.

So, consider any $t\in u$. Note that we have $P_{\mathcal{Y}|\mathcal{X}}(O_{t}^{i}|x_{t}):\!=\int_{O_{t}^{i}}\psi(y|x_{t})\,\mathrm{d}y$ by definition, and choose any $\epsilon\in\mathbb{R}_{>0}$. Because $\psi(\cdot|x_{t})$ is continuous at $y_{t}$, there is some open ball $B_{*}\in\mathcal{Y}$ that is centred on $y_{t}$, and such that for all $y\in B_{*}$,

[TABLE]

Furthermore, because the sequence $\{B_{t}^{i}\}_{i\in\mathbb{N}}$ shrinks to $y_{t}$, there is some $n\in\mathbb{N}$ such that, for all $i>n$, it holds that $B_{t}^{i}\subseteq B_{*}$. Furthermore, because each $O_{t}^{i}\subseteq B_{t}^{i}$, also clearly $O_{t}^{i}\subseteq B_{*}$ for all $i>n$. Now consider any $i>n$. Then,

[TABLE]

So, we conclude that indeed

[TABLE]

as claimed. Because this holds for all $t\in u$, it follows from what we discussed above that $\phi_{u}(y_{u}|x_{u})$ can be constructed “piecewise”, that is,

[TABLE]

which implies that $\phi_{u}(y_{u}|x_{u})$ exists and is the same for every sequence $\{O_{u}^{i}\}_{i\in\mathbb{N}}$ for which the assumed properties hold. Furthermore, since by assumption each $\psi(y_{t}|x_{t})>0$, we clearly also have $\phi_{u}(y_{u}|x_{u})>0$ for all $x_{u}\in\mathcal{X}_{u}$, and therefore in particular that $\mathbb{E}_{P_{\mathcal{X}}}[\phi_{u}(y_{u}|X_{u})]>0$.

It now follows from Proposition 6 that $\mathbb{E}_{P}[f(X_{v})\,|\,Y_{u}=y_{u}]$ exists and is the same for every sequence for which the assumed properties hold.

**Proof of Proposition 8 ** This is a special case of Corollary 17, obtained by setting $g(X_{u}):\!=\phi_{u}(y_{u}|X_{u})$ in that corollary’s statement, and applying Proposition 6 to the quantities $\mathbb{E}_{P}[f(X_{v})\,|\,Y_{u}=y_{u}]$.

**Proof of Proposition 9 ** Assume that there is a sequence $\{\lambda_{i}\}_{i\in\mathbb{N}}$ such that $\phi_{u}(y_{u}|X_{u})$ exists, is real-valued, and satisfies $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\phi_{u}(y_{u}|X_{u})]>0$.

Note that $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\phi_{u}(y_{u}|X_{u})]>0$ implies that $\mathbb{E}_{P_{\mathcal{X}}}[\phi_{u}(y_{u}|X_{u})]>0$ for all $P_{\mathcal{X}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$. Furthermore, for any $P_{\mathcal{X}}\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$, because by assumption $\lim_{i\to+\infty}\nicefrac{{P_{\mathcal{Y}|\mathcal{X}}(O_{u}^{i}|X_{u})}}{{\lambda_{i}}}=\phi_{u}(y_{u}|X_{u})$, it follows from Lemma 13 that

[TABLE]

This implies that there is some $n\in\mathbb{N}$ such that for all $j>n$,

[TABLE]

which implies that also $\mathbb{E}_{P_{\mathcal{X}}}[P_{\mathcal{Y}|\mathcal{X}}(O_{u}^{j}|X_{u})]>0$. Furthermore, since $O_{u}^{i}\supseteq O_{u}^{i+1}$, we have that $P_{\mathcal{Y}|\mathcal{X}}(O_{u}^{i}|x_{u})\geq P_{\mathcal{Y}|\mathcal{X}}(O_{u}^{i+1}|x_{u})$ for all $x_{u}\in\mathcal{X}_{u}$, by monotonicity of measure. It follows that also for all $k\leq n$

[TABLE]

Hence, we have found that $\mathbb{E}_{P_{\mathcal{X}}}[P_{\mathcal{Y}|\mathcal{X}}(O_{u}^{i}|X_{u})]>0$ for all $i\in\mathbb{N}$. Since $P\in\mathbb{P}_{\mathcal{Q},\mathcal{M}}$, it now follows that $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[P_{\mathcal{Y}|\mathcal{X}}(O_{u}^{i}|X_{u})]=\overline{P}_{\mathcal{Z}}(Y_{u}\in O_{u}^{i})>0$ for all $i\in\mathbb{N}$ by Lemma 19.

Now define the sequence $\{\mu_{i}\}_{i\in\mathbb{N}}$ as $\mu_{i}:\!=\underline{\mathbb{E}}_{\mathcal{Z}}[f(X_{v})\,|\,Y_{u}\in O_{u}^{i}]$, for all $i\in\mathbb{N}$. Fix any $i\in\mathbb{N}$. Then, because $\overline{P}_{\mathcal{Z}}(Y_{u}\in O_{u}^{i})>0$, it follows from Definition 4 that $\mu_{i}\in[\min f,\max f]$ and furthermore, by Proposition 5, that

[TABLE]

Therefore, and by the non-negative homogeneity of lower expectations, it also holds that

[TABLE]

Let now $\{\mu_{i_{k}}\}_{k\in\mathbb{N}}$ be any convergent subsequence; since the sequence $\{\mu_{i}\}_{i\in\mathbb{N}}$ is in the compact interval $[\min f,\max f]$, the Bolzano-Weierstrass theorem implies that at least one such subsequence exists. Let $\mu_{*}:\!=\lim_{k\to+\infty}\mu_{i_{k}}$.

We now clearly have that

[TABLE]

and therefore, by Lemma 14, that

[TABLE]

Furthermore, using Equation (16), we find that \underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\left[\phi_{u}(y_{u}|X_{u})\bigl{(}f(X_{v})-\mu_{*}\bigr{)}\right]=0.

Due to Proposition 10, and because $\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\phi_{u}(y_{u}|X_{u})]>0$, we conclude that $\mu_{*}$ corresponds to the unique root of the function G(\mu):\!=\underline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}\left[\phi_{u}(y_{u}|X_{u})\bigl{(}f(X_{v})-\mu\bigr{)}\right]. Furthermore, since the convergent subsequence $\{\mu_{i_{k}}\}_{k\in\mathbb{N}}$ was arbitrary, we find that $\mu_{*}$ is the limit of every convergent subsequence of $\{\mu_{i}\}_{i\in\mathbb{N}}$.

We next show that $\{\mu_{i}\}_{i\in\mathbb{N}}$ itself also converges to $\mu_{*}$. Assume ex absurdo that this is false. Then, there is some $\epsilon>0$ such that, for all $n\in\mathbb{N}$, there is some $k>n$ such that $\left|\mu_{k}-\mu_{*}\right|\geq\epsilon$. This implies that we can construct a subsequence $\{\mu_{i_{k}}\}_{k\in\mathbb{N}}$ such that $\left|\mu_{i_{k}}-\mu_{*}\right|\geq\epsilon$ for all $k\in\mathbb{N}$. This subsequence is again in the compact interval $[\min f,\max f]$, which implies that it has a convergent subsequence $\{\mu_{i_{k_{\ell}}}\}_{\ell\in\mathbb{N}}$, and clearly $\lim_{\ell\to+\infty}\mu_{i_{k_{\ell}}}\neq\mu_{*}$ because $\left|\mu_{i_{k_{\ell}}}-\mu_{*}\right|\geq\epsilon$ for all $\ell\in\mathbb{N}$. However, since $\{\mu_{i_{k_{\ell}}}\}_{\ell\in\mathbb{N}}$ is a convergent subsequence of the original sequence $\{\mu_{i}\}_{i\in\mathbb{N}}$, this contradicts our above conclusions. Hence, we must have that indeed $\lim_{i\to\infty}\mu_{i}=\mu_{*}$.

Since we already know that $\mu_{*}$ is the unique root of the function $G(\mu)$ defined above, and since this function is strictly decreasing due to Proposition 10, we must have that

[TABLE]

Therefore, due to Proposition 8, we conclude that

[TABLE]

The following two propositions prove the properties of the first of the two alternative imprecise updating methods that were suggested in Section 4.2.

Proposition 21

Let $\mathcal{Z}$ be an ICTHMC and consider any $u,v\in\mathcal{U}$, $y_{u}\in\mathcal{Y}_{u}$ and $f\in\mathcal{L}(\mathcal{X}_{v})$. For any $\{O_{u}^{i}\}_{i\in\mathbb{N}}$ in $\Sigma_{u}$ that shrinks to $y_{u}$, if for some $\{\lambda_{i}\}_{i\in\mathbb{N}}$ in $\mathbb{R}_{>0}$ the quantity $\phi_{u}(y_{u}|X_{u})$ exists, is real-valued and satisfies $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\phi_{u}(y_{u}|X_{u})]>0$, then the model defined by

[TABLE]

satisfies

[TABLE]

**Proof ** This is a special case of Corollary 17, obtained by setting $g(X_{u}):\!=\phi_{u}(y_{u}|X_{u})$ in that corollary’s statement, and applying Proposition 6 to the quantities $\mathbb{E}_{P}[f(X_{v})\,|\,Y_{u}=y_{u}]$.

Proposition 22

There exists some ICTHMC $\mathcal{Z}$, some sequences of time-points $u,v\in\mathcal{U}$, some function $f\in\mathcal{L}(\mathcal{X}_{v})$, some $y_{u}\in\mathcal{Y}_{u}$ and some sequence $\{O_{u}^{i}\}_{i\in\mathbb{N}}$ such that there is a sequence $\{\lambda_{i}\}_{i\in\mathbb{N}}$ for which $\phi_{u}(y_{u}|X_{u})$ exists, is real-valued and satisfies $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\phi_{u}(y_{u}|X_{u})]>0$, such that for

[TABLE]

it holds that

[TABLE]

**Proof ** Because the claim is existential, a proof by example suffices. To this end, let $\mathcal{X}:\!=\{x,\overline{x}\}$ be a binary state-space, and let $\mathcal{Y}:\!=[-1,1]$, with $\Sigma$ the Borel $\sigma$-algebra on $\mathcal{Y}$ under the usual topology. For the sequences of time-points, we choose $u=v=\{0\}$. Set $y_{0}:\!=0$.

The trick will be to choose the measures $P_{\mathcal{Y}|\mathcal{X}}(\cdot|x)$ and $P_{\mathcal{Y}|\mathcal{X}}(\cdot|\overline{x})$ so that the first gives $y_{0}$ strictly positive support (i.e. density), while the second assigns zero support (i.e. density) to $y_{0}$ but positive support (i.e. density) to the region around $y_{0}$. To this end, let $P_{\mathcal{Y}|\mathcal{X}}(\cdot|x)$ be the uniform distribution on $[-1,1]$.

We define the measure $P_{\mathcal{Y}|\mathcal{X}}(\cdot|\overline{x})$ by constructing $\phi(\cdot|\overline{x})$ explicitly. So, for every $y\in\mathcal{Y}$, let

[TABLE]

Clearly, $\int_{\mathcal{Y}}\phi(y|\overline{x})\,\mathrm{d}y=1$, and furthermore $\phi(y_{0}|\overline{x})=0$. For every $O\in\Sigma$, let $P_{\mathcal{Y}|\mathcal{X}}(O|\overline{x})$ be defined by

[TABLE]

For any sequence $\{O_{0}^{i}\}_{i\in\mathbb{N}}$ of open intervals $O_{0}^{i}:\!=(y_{0}-\delta_{i},y_{0}+\delta_{i})$, with $\delta_{i}>0$ such that $\{\delta_{i}\}_{i\in\mathbb{N}}\to 0^{+}$, we then clearly have

[TABLE]

and

[TABLE]

where $\lambda(O_{0}^{i})$ is the Lebesgue measure of $O_{0}^{i}$. Fix any such sequence $\{O_{0}^{i}\}_{i\in\mathbb{N}}$ and let $\{\lambda_{i}\}_{i\in\mathbb{N}}:\!=\{\lambda(O_{0}^{i})\}_{i\in\mathbb{N}}$.

Let the set of initial distributions $\mathcal{M}$ be the entire set of probability mass functions on $\mathcal{X}_{0}$. Consider the two probability mass functions $P_{1},P_{2}$ on $\mathcal{X}_{0}$ such that $P_{1}(x)=1$, $P_{1}(\overline{x})=0$, and $P_{2}(x)=0$, $P_{2}(\overline{x})=1$; clearly, $P_{1},P_{2}\in\mathcal{M}$. Let $\mathcal{Q}$ be any non-empty, bounded, and convex set of rate matrices with separately specified rows, and let $\mathbb{P}_{\mathcal{Q},\mathcal{M}}$ be the corresponding ICTMC. Construct $\mathcal{Z}$ from $\mathbb{P}_{\mathcal{Q},\mathcal{M}}$ and $(\mathcal{Y},\Sigma,P_{\mathcal{Y}|\mathcal{X}})$ as in Definition 2. Let $f\in\mathcal{L}(\mathcal{X}_{0})$ be defined by $f(x):\!=1$ and $f(\overline{x}):\!=-1$.

Clearly, $\overline{\mathbb{E}}_{\mathcal{Q},\mathcal{M}}[\phi(y_{0}|X_{0})]=\overline{\mathbb{E}}_{\mathcal{M}}[\phi(y_{0}|X_{0})]=P_{1}(x)\phi(y_{0}|x)+P_{1}(\overline{x})\phi(y_{0}|\overline{x})=P_{1}(x)\phi(y_{0}|x)=\frac{1}{2}>0$, and so the lower expectation $\underline{\mathbb{E}}_{\mathcal{Z}}^{\mathrm{R}}[f(X_{v})\,|\,Y_{u}=y_{u}]$ is well-defined.

Now, for any $P\in\mathcal{Z}$ such that $\mathbb{E}_{P}[\phi(y_{0}|X_{0})]>0$, we have by Proposition 6 that

[TABLE]

because $\phi(y_{0}|\overline{x})=0$. Hence,

[TABLE]

However, for any $i\in\mathbb{N}$, since $P_{2}\in\mathcal{M}$, there is some $P\in\mathcal{Z}$ such that

[TABLE]

and so

[TABLE]

Since we also know that $-1=\min f\leq\underline{\mathbb{E}}_{\mathcal{Z}}[f(X_{0})|Y_{0}\in O_{0}^{i}]\leq\mathbb{E}_{P}[f(X_{0})\,|\,Y_{0}\in O_{0}^{i}]$, this allows us to infer that $\underline{\mathbb{E}}_{\mathcal{Z}}[f(X_{0})|Y_{0}\in O_{0}^{i}]=-1$.

We conclude from all of the above that

[TABLE]

The following result proves the claim that the second alternative imprecise updating method from Section 4.2 does not in general satisfy the generalised Bayes’ rule for mixtures of densities.

Corollary 23

There exists some ICTHMC $\mathcal{Z}$, some sequences of time-points $u,v\in\mathcal{U}$, some function $f\in\mathcal{L}(\mathcal{X}_{v})$, some $y_{u}\in\mathcal{Y}_{u}$ and some sequence $\{O_{u}^{i}\}_{i\in\mathbb{N}}$ such that

[TABLE]

and

[TABLE]

with

[TABLE]

**Proof ** The claim is existential, so a proof by example suffices. Let $\mathcal{Z}$, $u,v\in\mathcal{U}$, $y_{u}\in\mathcal{Y}_{u}$, $f\in\mathcal{L}(\mathcal{X}_{v})$ and $\{O_{u}^{i}\}_{i\in\mathbb{N}}$ in $\Sigma_{u}$ be the same as in the example/proof of Proposition 22.

Now consider any $P\in\mathcal{Z}$, and note that clearly either $\mathbb{E}_{P_{\mathcal{X}}}[\phi(y_{0}|X_{0})]>0$, or $\mathbb{E}_{P_{\mathcal{X}}}[\phi(y_{0}|X_{0})]=0$. For the first case, as we established in the example/proof of Proposition 22, we know that $\mathbb{E}_{P}[f(X_{0})|Y_{0}=y_{0}]$ exists and is equal to $1$.

Suppose for the other case that $\mathbb{E}_{P_{\mathcal{X}}}[\phi(y_{0}|X_{0})]=0$. Because $\phi(y_{0}|x)>0$ and $\phi(y_{0}|\overline{x})=0$, this clearly implies that $P_{\mathcal{X}}(X_{0}=x)=0$, or in other words, that $P_{\mathcal{X}}(X_{0}=\overline{x})=1$. As we already established in the previous example, we then have for every $i\in\mathbb{N}$ that

[TABLE]

Because this holds for all $\in\mathbb{N}$, we clearly have that $\mathbb{E}_{P}[f(X_{0})|Y_{0}=y_{0}]$ exists, and

[TABLE]

Since this exhaustively covers all cases, we conclude that $\mathbb{E}_{P}[f(X_{0})|Y_{0}=y_{0}]$ exists for all $P\in\mathcal{Z}$. Therefore,

[TABLE]

which means that $\underline{\mathbb{E}}_{\mathcal{Z}}^{\mathrm{L}}[f(X_{v})\,|\,Y_{u}=y_{u}]$ is well-defined.

Furthermore, it follows from the above that, for all $P\in\mathcal{Z}$, we have either $\mathbb{E}_{P}[f(X_{0})|Y_{0}=y_{0}]=1$, or $\mathbb{E}_{P}[f(X_{0})|Y_{0}=y_{0}]=-1$. Also, since $P_{2}\in\mathcal{M}$, there is at least one $P\in\mathcal{Z}$ for which the second case applies. Hence, it follows that

[TABLE]

However, we also know that, for any $P\in\mathcal{Z}$, if $\mathbb{E}_{P_{\mathcal{X}}}[\phi(y_{0}|X_{0})]>0$, then $\mathbb{E}_{P}[f(X_{0})|Y_{0}=y_{0}]=1$. Since $P_{1}\in\mathcal{M}$, there is at least one $P\in\mathcal{Z}$ for which this holds. Therefore,

[TABLE]

where the second equality follows from Proposition 21. We conclude that indeed

[TABLE]

E Proofs of the Results in Section 5

We need the following lemma for the proof of Proposition 11.

Lemma 24

For all $i\in\{0,\dots,n\}$, let $g_{t_{i}}$, $g_{t_{i}}^{+}$ and $g_{t_{i}}^{-}$ be as defined in Section 5.2. Then for all $i\in\{0,\dots,n\}$:

[TABLE]

**Proof ** We provide a proof by induction. Clearly, the result is trivial for $i=n$. So, assume that it is true for $i$. We show that it is then also true for $i-1$ (with $i>0$).

We focus on $g_{t_{i-1}}^{+}$, and consider the two cases in its definition separately. So, consider any $x_{t_{i-1}}\in\mathcal{X}_{t_{i-1}}$. Then, if $g_{t_{i-1}}(x_{t_{i-1}})\geq 0$, we have

[TABLE]

where the first equality is by definition, the second is by the induction hypothesis, the third by iterated lower expectation (Lemma 12), and the final by the non-negative homogeneity of lower expectations and the assumption that $g_{t_{i-1}}(x_{t_{i-1}})\geq 0$.

For the other case, assume that $g_{t_{i-1}}(x_{t_{i-1}})<0$. Then,

[TABLE]

where the first equality is by definition, the second equality by the induction hypothesis, the third by iterated upper expectation (Lemma 12 combined with conjugacy), the fourth by conjugacy of upper- and lower expectation, and the final by the non-negative homogeneity of lower expectations and the assumption that $g_{t_{i-1}}(x_{t_{i-1}})<0$.

Since this covers both cases in the definition of $g_{t_{i-1}}^{+}(x_{t_{i-1}})$, we find that

[TABLE]

which concludes the proof for $g_{t_{i-1}}^{+}$. The proof for $g_{t_{i-1}}^{-}$ is completely analogous.

**Proof of Proposition 11 ** By combining Lemma 24 with iterated lower expectation (Lemma 12), we find that

[TABLE]

and similarly for the upper expectation.