On Average Risk-sensitive Markov Control Processes

TL;DR

This paper develops a Lyapunov-based framework for optimizing average risk-sensitive Markov control processes with general, possibly non-convex risk maps, including applications to behavioral economics and unbounded costs.

Contribution

It introduces a novel Lyapunov approach for existence and uniqueness of optimal controls in risk-sensitive Markov processes with general risk maps and non-compact state spaces.

Findings

Established fixed point existence for optimal control

Derived contraction conditions using Lyapunov and Doeblin-type conditions

Extended results to entropic risk maps with specific growth and Lyapunov conditions

Abstract

We introduce the Lyapunov approach to optimal control problems of average risk-sensitive Markov control processes with general risk maps. Motivated by applications in particular to behavioral economics, we consider possibly non-convex risk maps, modeling behavior with mixed risk preference. We introduce classical objective functions to the risk-sensitive setting and we are in particular interested in optimizing the average risk in the infinite-time horizon for Markov Control Processes on general, possibly non-compact, state spaces allowing also unbounded cost. Existence and uniqueness of an optimal control is obtained with a fixed point theorem applied to the nonlinear map modeling the risk-sensitive expected total cost. The necessary contraction is obtained in a suitable chosen seminorm under a new set of conditions: 1) Lyapunov-type conditions on both risk maps and cost functions that control the growth of iterations, and 2) Doeblin-type conditions, known for Markov chains, generalized to nonlinear mappings. In the particular case of the entropic risk map, the above conditions can be replaced by the existence of a Lyapunov function, a local Doeblin-type condition for the underlying Markov chain, and a growth condition on the cost functions.