Economic MPC of Markov Decision Processes: Dissipativity in Undiscounted Infinite-Horizon Optimal Control

TL;DR

This paper extends dissipativity theory in economic MPC to stochastic Markov Decision Processes, enabling stability analysis for a broader class of problems using nonlinear stage costs and probability measures.

Contribution

It introduces a generalized dissipativity framework for undiscounted infinite-horizon MDPs, addressing limitations of existing theories for stochastic systems.

Findings

Applied to stochastic LQ regulator with Gaussian noise

Explicit storage functional derived for the LQ case

Framework supports Lyapunov stability analysis in stochastic settings

Abstract

Economic Model Predictive Control (MPC) dissipativity theory is central to discussing the stability of policies resulting from minimizing economic stage costs. In its current form, the dissipativity theory for economic MPC applies to problems based on deterministic dynamics or to very specific classes of stochastic problems, and does not readily extend to generic Markov Decision Processes. In this paper, we clarify the core reason for this difficulty, and propose a generalization of the economic MPC dissipativity theory that circumvents it. This generalization focuses on undiscounted infinite-horizon problems and is based on nonlinear stage cost functionals, allowing one to discuss the Lyapunov asymptotic stability of policies for Markov Decision Processes in terms of the probability measures underlying their stochastic dynamics. This theory is illustrated for the stochastic Linear Quadratic Regulator with Gaussian process noise, for which a storage functional can be provided explicitly. For the sake of brevity, we limit our discussion to undiscounted Markov Decision Processes.