Reinforcement Learning based on MPC/MHE for Unmodeled and Partially Observable Dynamics

TL;DR

This paper introduces a reinforcement learning approach to tune MPC and MHE parameters for better control of systems with unmodeled and partially observable dynamics, improving closed-loop performance.

Contribution

It presents a novel RL-based method to jointly tune MPC and MHE parameters for POMDPs without relying on accurate models.

Findings

Enhanced control performance in POMDP systems

Effective parameter tuning via RL for unmodeled dynamics

Improved closed-loop control outcomes

Abstract

This paper proposes an observer-based framework for solving Partially Observable Markov Decision Processes (POMDPs) when an accurate model is not available. We first propose to use a Moving Horizon Estimation-Model Predictive Control (MHE-MPC) scheme in order to provide a policy for the POMDP problem, where the full state of the real process is not measured and necessarily known. We propose to parameterize both MPC and MHE formulations, where certain adjustable parameters are regarded for tuning the policy. In this paper, for the sake of tackling the unmodeled and partially observable dynamics, we leverage the Reinforcement Learning (RL) to tune the parameters of MPC and MHE schemes jointly, with the closed-loop performance of the policy as a goal rather than model fitting or the MHE performance. Illustrations show that the proposed approach can effectively increase the performance of close-loop control of systems formulated as POMDPs.