Should we use model-free or model-based control? A case study of battery management systems

TL;DR

This paper compares model-free reinforcement learning and model-based predictive control for battery management, evaluating their efficiency, robustness, and practicality in optimizing energy costs under operational constraints.

Contribution

It provides a comprehensive benchmark analysis of RL and MPC for battery management systems, highlighting their respective advantages and limitations in real-world scenarios.

Findings

RL achieves faster optimal solutions and robustness to demand shifts.

MPC is less time-consuming to implement but less adaptable.

RL requires more training time but offers better robustness.

Abstract

Reinforcement learning (RL) and model predictive control (MPC) each offer distinct advantages and limitations when applied to control problems in power and energy systems. Despite various studies on these methods, benchmarks remain lacking and the preference for RL over traditional controls is not well understood. In this work, we put forth a comparative analysis using RL- and MPC-based controllers for optimizing a battery management system (BMS). The BMS problem aims to minimize costs while adhering to operational limits. by adjusting the battery (dis)charging in response to fluctuating electricity prices over a time horizon. The MPC controller uses a learningbased forecast of future demand and price changes to formulate a multi-period linear program, that can be solved using off-the-shelf solvers. Meanwhile, the RL controller requires no timeseries modeling but instead is trained from the sample trajectories using the proximal policy optimization (PPO) algorithm. Numerical tests compare these controllers across optimality, training time, testing time, and robustness, providing a comprehensive evaluation of their efficacy. RL not only yields optimal solutions quickly but also ensures robustness to shifts in customer behavior, such as changes in demand distribution. However, as expected, training the RL agent is more time-consuming than MPC.