Stability-Constrained Markov Decision Processes Using MPC

TL;DR

This paper integrates stability constraints into Markov Decision Processes by leveraging Model Predictive Control, ensuring policies are stabilizing and reformulating discounted MDPs as undiscounted ones for better stability handling.

Contribution

It introduces a method to incorporate stability constraints into discounted MDPs using MPC, enabling stable policy synthesis within reinforcement learning.

Findings

Reformulation of discounted MDPs as undiscounted MDPs for stability analysis

MPC-based policies can produce optimal or stabilizing policies depending on stability

The approach ensures stability constraints are directly incorporated into policy design

Abstract

In this paper, we consider solving discounted Markov Decision Processes (MDPs) under the constraint that the resulting policy is stabilizing. In practice MDPs are solved based on some form of policy approximation. We will leverage recent results proposing to use Model Predictive Control (MPC) as a structured policy in the context of Reinforcement Learning to make it possible to introduce stability requirements directly inside the MPC-based policy. This will restrict the solution of the MDP to stabilizing policies by construction. The stability theory for MPC is most mature for the undiscounted MPC case. Hence, we will first show in this paper that stable discounted MDPs can be reformulated as undiscounted ones. This observation will entail that the MPC-based policy with stability requirements will produce the optimal policy for the discounted MDP if it is stable, and the best stabilizing policy otherwise.