Tractable Dual Optimal Stochastic Model Predictive Control: An Example in Healthcare

TL;DR

This paper introduces a tractable approach to stochastic model predictive control that preserves the dual control probing behavior, demonstrated through a healthcare decision-making example.

Contribution

It proposes approximating system dynamics with a POMDP to enable optimal dual control in a receding horizon framework, balancing exploration and exploitation.

Findings

Maintains probing behavior in stochastic control.

Solves finite horizon control for POMDPs efficiently.

Demonstrates approach with healthcare example.

Abstract

Output-Feedback Stochastic Model Predictive Control based on Stochastic Optimal Control for nonlinear systems is computationally intractable because of the need to solve a Finite Horizon Stochastic Optimal Control Problem. However, solving this problem leads to an optimal probing nature of the resulting control law, called dual control, which trades off benefits of exploration and exploitation. In practice, intractability of Stochastic Model Predictive Control is typically overcome by replacement of the underlying Stochastic Optimal Control problem by more amenable approximate surrogate problems, which however come at a loss of the optimal probing nature of the control signals. While probing can be superimposed in some approaches, this is done sub-optimally. In this paper, we examine approximation of the system dynamics by a Partially Observable Markov Decision Process with its own Finite Horizon Stochastic Optimal Control Problem, which can be solved for an optimal control policy, implemented in receding horizon fashion. This procedure enables maintaining probing in the control actions. We further discuss a numerical example in healthcare decision making, highlighting the duality in stochastic optimal receding horizon control.

Tables

Table 1. TABLE I : Problem data for healthcare decision making example.

Decision $a$	Transition Probabilities $P\left(a\right)$	Observation Probabilities $R\left(a\right)$	Cost $c\left(a\right)$
1: Skip next appointment slot			$\left[\begin{array}{c}0\\ 5\\ 5\end{array}\right]$
2: Schedule new appointment			$\left[\begin{array}{c}1\\ 1\\ 1\end{array}\right]$
3: Order rapid diagnostic test			$\left[\begin{array}{c}4\\ 3\\ 4\end{array}\right]$
4: Apply available treatment			$\left[\begin{array}{c}4\\ 2\\ 4\end{array}\right]$

Equations

$$x_{t+1}$$

$$y_t$$

$${\zeta }^t$$

$${\pi }_t$$

$${\pi }_{t+1\mid t}$$

$$u_t=g_t({\pi }_t),$$

$$J_N({\pi }_0,{\mathbf{g}}^{N-1})={\mathbb{E}}_0\left[\sum _{j=0}^{N-1}{\alpha }^{j}c(x_j,g_j({\pi }_j))+{\alpha }^N{c}_N(x_N)\right],$$

$$J_{\infty }({\pi }_0,{\mathbf{g}}^{\infty })$$

$${\mathbb{P}}_k\left[x_k\in {\mathcal{X}}_k\right]$$

$$u_k=g_k({\pi }_k)$$

$$V_k({\pi }_k)\triangleq \underset{g_k(\cdot )}{\inf }$$

$${\mathbb{P}}_{k+1}\left[x_{k+1}\in {\mathbb{X}}_{k+1}\right]\ge 1-{ϵ}_{k+1},$$

$$g_k({\pi }_k)\in {\mathbb{U}}_k,$$

$$V_N({\pi }_N)$$

$${\mathbf{g}}^{MPC}\triangleq \{g_0^{\star },g_0^{\star },g_0^{\star },\dots \}.$$

$$\begin{gathered} (1-\alpha \gamma )J_{\infty }({\pi }_0,{\mathbf{g}}^{{\star }^{\infty }})\le (1-\alpha \gamma )J_{\infty }({\pi }_0,{\mathbf{g}}^{MPC}) \\ \le J_N({\pi }_0,{\mathbf{g}}^{{\star }^{N-1}})+\frac{\alpha }{1-\alpha }\eta , \end{gathered}$$

$$\mathbb{P}\left(x_{t+1}=j\mid x_t=i,u_t=a\right)$$

$$\mathbb{P}\left(y_{t+1}=\theta \mid x_{t+1}=j,u_t=a\right)$$

$${\pi }_{t+1,j}=\frac{{\sum }_{i\in \mathbf{X}}{\pi }_{t,j}{p}_{ij}^a{r}_{j\theta }^a}{{\sum }_{i,j\in \mathbf{X}}{\pi }_{t,j}{p}_{ij}^a{r}_{j\theta }^a},$$

$$\begin{gathered} V_k({\pi }_k)= \\ \underset{u_k\in \mathbf{U}}{\min }\left\{{\pi }_{k}c(u_k)+\alpha \sum _{\theta \in \mathbf{Y}}\mathbb{P}\left(\theta \mid {\pi }_k,u_k\right)V_{k+1}({\pi }_{k+1})\right\}, \end{gathered}$$

$$V_N({\pi }_N)={\pi }_N{c}_N.$$

$$P=\left[\begin{array}{ccc}0.8& 0.2& 0.0\\ 0.0& 0.9& 0.1\\ 0.0& 0.0& 1.0\end{array}\right],$$

$$c_N={\left[\begin{array}{ccc}0& 4& 30\end{array}\right]}^T.$$

Full text

Glossary

FHSOCP

finite-horizon stochastic optimal control problem,

HMM

hidden Markov model,

i.i.d.

independent and identically distributed,

POMDP

partially observable Markov decision process,

SDPE

stochastic dynamic programming equation,

SMPC

stochastic model predictive control,

UAV

unmanned autonomous/aerial vehicle.

I Introduction

Stochastic Optimal Control on the infinite time horizon is a computationally intractable problem. Unfortunately, the same holds even after replacing the infinite-horizon control problem by receding horizon implementation of a finite-horizon stochastic optimal control law. While a number of approaches have been devoted to Stochastic Model Predictive Control (SMPC), including [1, 2, 3, 4], none of these involve solution of a Finite Horizon Stochastic Optimal Control Problem (FHSOCP), which is required for optimal probing in the resulting control inputs but computationally intractable in practice. Assuming an available solution of the FHSOCP in principle leads to a dual optimal SMPC law, which enjoys a number of desirable properties discussed in [5]. These properties include recursive feasibility, stochastic stability and, in particular, bounds relating infinite-horizon performance of the SMPC control with computed finite-horizon performance and optimal infinite-horizon performance. This result, which requires explicit solution of the FHSOCP, encourages a tractable version of dual optimal SMPC.

In this paper, we discuss a version of SMPC motivated by the structure of dual optimal SMPC as in [5]. In particular, we suggest approximation of the system dynamics by a Partially Observable Markov Decision Process (POMDP), the finite-horizon solution of which is tractable for small to moderate problem instances (see e.g. [6, 7]). The main benefit of this approach is that duality of the resulting control inputs is naturally maintained by the approximation in an optimal sense. Moreover, given that POMDPs are a subclass of general nonlinear systems, the results in [5] hold in modified form for the resulting approximate dynamics. These two observations lead us to propose SMPC on the approximate POMDP with explicit solution of the resulting approximate FHSOCP. Receding horizon control in POMDPs has been discussed for instance with respect to UAV guidance [8], sensor scheduling in [9], and robotic manipulation in [10]. We discuss in particular an example in medical decision making, involving decisions regarding appointment scheduling, the use of costly diagnostic tests, and patient treatment. This example highlights in particular the probing nature of the dual optimal SMPC on a POMDP.

The paper evolves as follows. After briefly introducing SMPC as suggested in [5] in Section II, we transition to receding horizon control of POMPDs in Section III. We transition to the use of POMDPs in healthcare in Section IV, which includes our numerical example highlighting duality in receding horizon implementations of optimal POMDP solutions. The paper closes with some concluding remarks in Section V.

II Stochastic Model Predictive Control

II-A Stochastic Optimal Control in a Nutshell

We consider receding horizon output-feedback control for nonlinear stochastic systems of the form

[TABLE]

where $x_{k}\in\mathbb{R}^{n}$, $u_{k}\in\mathbb{R}^{m}$, $y_{k}\in\mathbb{R}^{o}$, starting from known initial state density $\pi_{0|-1}=\operatorname{pdf}(x_{0})$. We denote the data available at time $t$ by

[TABLE]

We further impose the following standing assumption on the random variables and control inputs.

Assumption 1**.**

The signals in (1-2) satisfy:

$w_{t}$ and $v_{t}$ are i.i.d. sequences with known densities. 2. 2.

$x_{0},w_{t},v_{l}$ are mutually independent for all $t,l\geq 0$. 3. 3.

The control input $u_{t}$ at time instant $t\geq 0$ is a function of the data $\mathbf{\zeta}^{t}$ and given initial state density $\pi_{0\mid-1}$.

The information state, denoted $\pi_{t}$, is the conditional probability density function of state $x_{t}$ given data $\mathbf{\zeta}^{t}$,

[TABLE]

and is propagated via the Bayesian Filter (see e.g. [11, 12]):

[TABLE]

from initial density $\pi_{0\mid-1}$. As a result of the Markovian dynamics (1-2), optimal control inputs must inherently be separated feedback policies (e.g. [13, 14]). That is, control input $u_{t}$ depends on the data $\mathbf{\zeta}^{t}$ and initial density $\pi_{0\mid-1}$ solely through the current information state, $\pi_{t}$. Optimality thus requires propagating $\pi_{t}$ and policies $g_{t}$, where

[TABLE]

leading to the closed-loop architecture displayed in Figure 1.

Definition 1**.**

$\mathbb{E}_{k}[\,\cdot\,]$ and $\mathbb{P}_{k}[\,\cdot\,]$ are expected value and probability with respect to state $x_{k}$ – with conditional density $\pi_{k}$ – and i.i.d. random variables $\{(w_{j},v_{j+1}):j\geq k\}$.

Given control horizon $N\in\mathbb{N}$, stage cost $c:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}_{+}$, terminal cost $c_{N}:\mathbb{R}^{n}\to\mathbb{R}_{+}$, and discount factor $\alpha\in\mathbb{R}_{+}$, we further define the cost

[TABLE]

where $\mathbf{t}^{m}\triangleq\{t_{0},t_{1},\ldots,t_{m}\}$. Extending this cost to the infinite horizon typically requires $\alpha<1$ and omitting the terminal cost term $c_{N}(\cdot)$, leading to

[TABLE]

Additionally introducing the constraints

[TABLE]

for $\epsilon_{k}\in[0,1)$, finite-horizon stochastic optimal feedback policies may be computed, in principle, by solving the Stochastic Dynamic Programming Equation (SDPE),

[TABLE]

for $k=0,\ldots,N-1$. The equation is solved backwards in time, from its terminal value

[TABLE]

Solution of the SDPE is the primary source of intractability in Stochastic Optimal Control. The reason for this difficulty lies in the dependence of the future information states in (6-7) on current and future control inputs via (3-4). However, optimality via the SDPE leads to a control law of dual nature [15]. Notice that, while the Bayesian Filter (3-4) can be approximated to arbitrary accuracy using a Particle Filter [12], the SDPE cannot be easily simplified without loss of optimal probing in the control inputs. While control laws generated without solution of the SDPE can be modified artificially to include certain excitation properties, as discussed for instance in [16, 17], such approaches are suboptimal and do not enjoy the same theoretical guarantees.

II-B Dual Optimal SMPC Algorithm

While computationally intractable, solution of the SDPE and the associated feedback architecture in Figure 1 lead to the following dual optimal SMPC algorithm, the properties of which are discussed in [5].

This algorithm differs from common practice in stochastic model predictive control in that it explicitly uses the information state $\pi_{t}$. Throughout the literature, these conditional densities are commonly replaced by state estimates in order to generate certainty-equivalent control laws. While this makes the problem more tractable, one no longer solves the underlying Stochastic Optimal Control problem, leading to a loss of duality in the resulting control inputs. The central divergence to such approaches lies in Step 2 of the algorithm, in which the SDPE is presumed solved offline for the optimal feedback policies $g_{t}^{\star}(\pi_{t})$. This is an extraordinarily difficult proposition in many cases but captures the optimality, and hence duality, as a closed-loop feedback control law. The complexity of this step lies not only in computing a vector functional but also in the internal propagation of the information state within the SDPE.

Definition 2**.**

Denote by $\mathbf{g}^{MPC}$ the SMPC implementation of policy $g_{0}^{\star}(\cdot)$ on the infinite horizon, i.e.

[TABLE]

Similarly, $\mathbf{g}^{\star^{N-1}}$ and $\mathbf{g}^{\star^{\infty}}$ are the optimal sequences of policies in (6-7) and the corresponding infinite-horizon problem.

Subsuming assumptions akin to those used in deterministic MPC (e.g. [18]) to ensure stochastic recursive feasibility of the SMPC algorithm and constrain the growth of the underlying value function with increasing control horizon $N$, we re-state the following technical result from [5].

Theorem 1**.**

Under suitable assumptions, SMPC with discount factor $\alpha\in[0,1)$ yields

[TABLE]

for given values of $\gamma\in[0,1]$ and $\eta\in\mathbb{R}_{+}$.

This result relates the following quantities: design cost, $J_{N}(\pi_{0},\mathbf{g}^{*^{N-1}}),$ which is known as part of the SMPC calculation, optimal cost $J_{\infty}(\pi_{0},\mathbf{g}^{*^{\infty}})$ which is unknown (otherwise we would use $\mathbf{g}^{*^{\infty}}$), and unknown infinite-horizon SMPC achieved cost $J_{\infty}(\pi_{0},\mathbf{g}^{MPC})$. The result, which requires duality and solution of the SDPE, is special in that SMPC approaches relying of approximation of the SDPE, as commonly found in the literature, do generally not yield statements regarding performance of the implemented control law on the infinite horizon.

III Approximate Dual Optimal SMPC via POMDPs

We next approximate the nonlinear dynamics (1-2) by a POMDP. The above algorithm and observations are then applicable in slightly modified form for the resultant approximate model. However, we can solve finite-horizon POMDP problems of reasonable dimensions explicitly, allowing us to maintain duality in the approximate version of the SMPC algorithm in Section II. For approximation of the system dynamics and cost via POMDPs, see for example [9], which discusses a related approach. In contrast with this reference, we suggest choosing the approximation so that the resulting SDPE can be solved explicitly. This results in maintaining duality in a stochastic optimal sense, although on the approximate POMDP dynamics.

III-A Partially Observable Markov Decisions Processes

POMDPs are characterized by probabilistic dynamics on a finite state space $\mathbf{X}=\{1,\ldots,n\}$, finite action space $\mathbf{U}=\{1,\ldots,m\}$, and finite observation space $\mathbf{Y}=\{1,\ldots,o\}$. POMDP dynamics are defined by the conditional state transition and observation probabilities

[TABLE]

$t\geq 1$, $i,j\in\mathbf{X}$, $a\in\mathbf{U}$, $\theta\in\mathbf{Y}$. The state transition dynamics (8) correspond to a conventional Markov Decision Process (MDP). However, the control actions $u_{t}$ are to be chosen based on the initial state distribution $\pi_{0}=\operatorname{pdf}(x_{0})$ and the sequences of observations, $\{y_{1},\ldots,y_{t}\}$, and controls $\{u_{0},\ldots,u_{t-1}\}$, respectively. That is, we are choosing our control actions in a Hidden Markov Model (HMM [19]) setup.

Given control action $u_{t}=a$ and measured output $y_{t+1}=\theta$, the Bayesian Filter recursion (3-4) extends to the POMDP dynamics (8-9) as

[TABLE]

where $\pi_{t,j}$ denotes the $j^{\text{th}}$ entry of the row vector $\pi_{t}$. We define the cost as in Section II, with stage cost $c(x_{t},u_{t})=c_{i}^{a}$ if $x_{t}=i\in\mathbf{X}$ and $u_{t}=a\in\mathbf{U}$, or $e_{i}c(a)$ in vectorized form. The terminal cost $c_{N}(x_{t})$ is defined similarly. Omitting constraints for brevity, optimal control decisions may then be computed via SDPE

[TABLE]

from terminal value function

[TABLE]

IV Application in Healthcare

IV-A POMDPs in Healthcare Decision Making

A framework for the use of POMDPs in healthcare decision-making applications is discussed in detail in [20]. Particular examples of POMDPs capturing disease stages as well as appropriate tests and treatment decisions for individual patients include [21, 22]. In [22], for instance, a POMDP is used for mammography screening in individual patients. States in the underlying Markov chain model are

No cancer 2. 2.

In situ cancer 3. 3.

Invasive cancer 4. 4.

In situ post-cancer 5. 5.

Invasive post-cancer 6. 6.

Death

Available decisions are either to perform a mammogram, with possible follow-up biopsy, or to wait, which may lead to self-detection. The authors proceed with solution of the POMDP for optimal screening strategies depending on patient age and risk for in situ or invasive cancer. This highlights a successful application of POMDPs in healthcare.

IV-B An Illustrative POMDP in Healthcare

The remainder of this paper discusses a particular numerical example of decisions on treatment and diagnosis in healthcare, displaying specifically the use of dual control in SMPC applied to a POMDP. Consider a patient treated for a specific disease which can be managed but not cured. For simplicity, we assume that the patient does not die under treatment. While this transition would have to be added in practice, it results in a time-varying model, which we avoid in order to keep the following discussion compact.

The disease encompasses three stages with severity increasing from Stage 1 through Stage 2 to Stage 3, transitions between which are governed by a Markov chain with transition probability matrix

[TABLE]

where $P$ is the matrix with values $p_{ij}$ at row $i$ and column $j$. All transition and observation probability matrices below are defined similarly. Once our patient enters Stage 3, Stages 1 and 2 are inaccessible for all future times. However, Stage 3 can only be entered through Stage 2, a transition from which to Stage 1 is possible only under costly treatment. The same treatment inhibits transitions from Stage 2 to Stage 3. We have access to the patient state only through tests, which will result in one of three possible values, each of which is representative of one of the three disease stages. However, these tests are imperfect, with non-zero probability of returning an incorrect disease stage. All possible state transitions and observations are illustrated in Figure 2.

At each point in time, the current information state $\pi_{t}$ is available to make one of four possible decisions:

Skip next appointment slot 2. 2.

Schedule new appointment 3. 3.

Order rapid diagnostic test 4. 4.

Apply available treatment

Skipping an appointment slot results in the patient progressing through the Markov chain describing the transition probabilities of the disease without medical intervention, without new information being available after the current decision epoch. Scheduling an appointment does not alter the patient transition probabilities but provides a low-quality assessment of the current disease stage, which is used to refine the next information state. The third option, ordering a rapid diagnostic test, allows for a high-quality assessment of the patient’s state, leading to a more reliable refinement of the next information state than possible when choosing the previous decision option. The results from this diagnostic test are considered available sufficiently fast so that the patient state remains unchanged under this decision. The remaining option entails medical intervention, allowing transition from Stage 2 to Stage 1 while preventing transition from Stage 2 to Stage 3. Transition probabilities $P(a)$, observation probabilities $R(a)$, and stage cost vectors $c(a)$ for each decision are summarized in Table I. Additionally, we impose the terminal cost

[TABLE]

While we choose the SMPC discount factor as $\alpha=1$ for simplicity, one may also extend this discussion and choose a discount factor $\alpha\in(0,1)$ to exploit Theorem 1 explicitly. Given that we do not consider constraints in this example, such that recursive feasibility holds by default, this can be enabled via proper choice of the terminal cost term, $c_{N}$.

IV-C Rationale for Duality

Intuitively, we expect an efficient policy for this problem to attempt avoiding transitions to Stage 3 while managing the ressources required to schedule appointments, order tests, or apply medical intervention. This may, in principle, be achieved by a policy akin to the following structure:

Skip appointments when Stages 2 and 3 are unlikely. 2. 2.

Schedule appointments when Stages 2 and 3 are likely but the probability for Stage 2 is below some threshold. 3. 3.

Order diagnostic test if the probability of Stage 2 lies in a specific range. 4. 4.

Proceed with medical intervention if the probability of Stage 2 is high.

While the optimal policy may be somewhat more intricate, this simple decision structure could be acceptable in practice. However, even this simple structure includes duality in the decisions, demonstrated by including the diagnostic test even though it does not alter the patient state. That is, this decision improves the quality of available information at a cost, also called exploration. This improvement in the available information allows us to apply medical intervention at appropriate times, which is called exploitation.

IV-D Computational Results

The trade-off between these two principal decision categories is precisely what is encompassed by duality, which we can include in an optimal sense by solving the SDPE and applying the resulting initial policy in receding horizon fashion. This is demonstrated in Figure 3, which shows simulation results for SMPC with control horizon $N=6$. As anticipated, the stochastic optimal receding horizon policy shows a structure not drastically different from the decision structure motivated above. In particular, diagnostic tests are used effectively to decide on medical intervention.

In contrast, Figure 4 shows a similar simulation for certainty-equivalent SMPC with control horizon $N=6$. In this control, the information state $\pi_{t}$ is replaced by the most likely patient state $\hat{x}_{t}$ at each time, a simplification often used to ease the computational burden associated with solving the underlying control problem and propagating the information state $\pi_{t}$ over time. However, this simplification comes at a loss of duality in the control decisions. As shown in Figure 4, the certainty equivalent control does not choose diagnosis to refine the information state and subsequently make a more informed treatment decision. Given that the control is only based on one of three possible estimate patient states, we can give the entire certainty equivalent optimal policy as:

Skip appointments when Stage 1 is most likely. 2. 2.

Apply treatment when Stage 2 is most likely. 3. 3.

Schedule appointments when Stage 3 is most likely.

Clearly, this policy does not make effective use of the ressources and decision options available to us, caused by its loss of duality in the control signal. In fact, not classifying the patient state correctly towards the end of the simulation leads to a preventable transition to Stage 3 in this simulation. The failure of this seemingly similar certainty equivalent SMPC control strategy highlights the need for duality in the control signals, as is accommodated by dual optimal SMPC as motivated in this paper.

V Conclusions

We discussed approximation of intractable nonlinear output-feedback SMPC problems by POMDPs, which for reasonable problem dimensions can be solved explicitly. The benefit of this approach as opposed to common practice in SMPC is that duality of the control actions natural to Stochastic Optimal Control is maintained optimally. We demonstrated the use of SMPC and in particular the probing nature of the resulting control inputs by means of a particular numerical POMDP in healthcare decision making. The example displays in particular how an expensive diagnostic test is chosen to refine the current information state without influencing the patient state. Without duality in the control input, this diagnostic test is not used, resulting subsequently in poorly informed treatment decisions due to lack of refinement in the information state.