Integrating Reinforcement Learning and Model Predictive Control with Applications to Microgrids

TL;DR

This paper introduces a novel method combining reinforcement learning and model predictive control to efficiently solve optimal control problems in microgrids, significantly reducing computation time while maintaining control quality.

Contribution

It proposes a decoupled Q-function approach with neural network approximation to simplify mixed-integer MPC problems, enabling faster online control in microgrid systems.

Findings

Reduces MPC online computation time substantially.

Maintains high feasibility and low suboptimality.

Demonstrates effectiveness on real-world microgrid data.

Abstract

This work proposes an approach that integrates reinforcement learning and model predictive control (MPC) to solve finite-horizon optimal control problems in mixed-logical dynamical systems efficiently. Optimization-based control of such systems with discrete and continuous decision variables entails the online solution of mixed-integer linear programs, which suffer from the curse of dimensionality. Our approach aims to mitigate this issue by decoupling the decision on the discrete variables from the decision on the continuous variables. In the proposed approach, reinforcement learning determines the discrete decision variables and simplifies the online optimization problem of the MPC controller from a mixed-integer linear program to a linear program, significantly reducing the computational time. A fundamental contribution of this work is the definition of the decoupled Q-function, which plays a crucial role in making the learning problem tractable in a combinatorial action space. We motivate the use of recurrent neural networks to approximate the decoupled Q-function and show how they can be employed in a reinforcement learning setting. Simulation experiments on a microgrid system using real-world data demonstrate that the proposed method substantially reduces the online computation time of MPC while maintaining high feasibility and low suboptimality.