Risk Filtering and Risk-Averse Control of Markovian Systems Subject to Model Uncertainty

TL;DR

This paper develops a framework for risk-averse control of Markov decision processes under model uncertainty, introducing risk filters to handle unknown parameters and deriving Bellman optimality principles.

Contribution

It introduces risk filters for Bayesian model uncertainty in Markov decision processes and derives a Bellman principle for risk-averse control under these conditions.

Findings

Risk filters effectively incorporate model uncertainty into control decisions.

The Bellman principle is extended to non-standard risk-averse problems.

Applications include optimal investment and clinical trial management.

Abstract

We consider a Markov decision process subject to model uncertainty in a Bayesian framework, where we assume that the state process is observed but its law is unknown to the observer. In addition, while the state process and the controls are observed at time $t$, the actual cost that may depend on the unknown parameter is not known at time $t$. The controller optimizes total cost by using a family of special risk measures, that we call risk filters and that are appropriately defined to take into account the model uncertainty of the controlled system. These key features lead to non-standard and non-trivial risk-averse control problems, for which we derive the Bellman principle of optimality. We illustrate the general theory on two practical examples: optimal investment and clinical trials.