Economic Model Predictive Control as a Solution to Markov Decision Processes

TL;DR

This paper explores how Economic Model Predictive Control (MPC) can serve as an approximate solution to Markov Decision Processes (MDPs), addressing computational challenges and conditions for optimality in stochastic dynamic systems.

Contribution

It clarifies the relationship between Economic MPC and MDPs, providing conditions under which MPC can achieve closed-loop optimality and highlighting its potential as a heuristic.

Findings

Economic MPC can approximate MDP solutions under certain conditions

Conditions for MPC to achieve optimality are identified

Economic MPC offers a computationally feasible alternative to solving MDPs

Abstract

Markov Decision Processes (MDPs) offer a fairly generic and powerful framework to discuss the notion of optimal policies for dynamic systems, in particular when the dynamics are stochastic. However, computing the optimal policy of an MDP can be very difficult due to the curse of dimensionality present in solving the underlying Bellman equations. Model Predictive Control (MPC) is a very popular technique for building control policies for complex dynamic systems. Historically, MPC has focused on constraint satisfaction and steering dynamic systems towards a user-defined reference. More recently, Economic MPC was proposed as a computationally tractable way of building optimal policies for dynamic systems. When stochsaticity is present, economic MPC is close to the MDP framework. In that context, Economic MPC can be construed as attractable heuristic to provide approximate solutions to MDPs. However, there is arguably a knowledge gap in the literature regarding these approximate solutions and the conditions for an MPC scheme to achieve closed-loop optimality. This chapter aims to clarify this approximation pedagogically, to provide the conditions for MPC to deliver optimal policies, and to explore some of their consequences.