Policy stability and ultimate stationarity in discounted risk-sensitive stochastic control

TL;DR

This paper analyzes the stability and stationarity of optimal policies in finite-state Markov Decision Processes under risk-sensitive discounted criteria, exploring robustness, limiting regimes, and the non-stationarity of policies.

Contribution

It provides a comprehensive analysis of policy stability and stationarity in risk-sensitive MDPs, including robustness to parameter perturbations and the behavior under limiting regimes.

Findings

Optimal policies exhibit intrinsic non-stationarity in the discounted risk-sensitive setting.

Robustness of policies to perturbations in risk-aversion and discount factors is established.

Connections between discounted and average formulations are clarified under mixing conditions.

Abstract

We study discrete-time Markov Decision Processes (MDPs) on finite state-action spaces and analyze the stability of optimal policies and value functions in the long-run discounted risk-sensitive objective setting. Our analysis addresses robustness with respect to perturbations of the risk-aversion parameter and the discount factor, the emergence of ultimate stationarity, and the interaction between discounted and averaged formulations under suitable mixing assumptions. We further investigate limiting regimes associated with vanishing discount and vanishing risk sensitivity, and discuss the role of Blackwell-type stability properties in the discounted setting. Finally, we provide numerical illustrations that highlight the intrinsic non-stationarity of optimal discounted risk-sensitive policies.