Optimal Control of District Cooling Energy Plant with Reinforcement Learning and MPC

TL;DR

This paper compares reinforcement learning and model predictive control for optimizing district cooling energy plants, demonstrating both methods achieve significant cost savings with different computational approaches.

Contribution

It introduces a novel RL controller based on least-squares policy iteration and compares it with an MPC controller that avoids mixed-integer programming.

Findings

Both controllers achieve about 17% cost savings.

The RL controller uses a new Q-learning algorithm with effective design choices.

The MPC controller only requires a nonlinear program solver.

Abstract

We consider the problem of optimal control of district cooling energy plants (DCEPs) consisting of multiple chillers, a cooling tower, and a thermal energy storage (TES), in the presence of time-varying electricity price. A straightforward application of model predictive control (MPC) requires solving a challenging mixed-integer nonlinear program (MINLP) because of the on/off of chillers and the complexity of the DCEP model. Reinforcement learning (RL) is an attractive alternative since its real-time control computation is much simpler. But designing an RL controller is challenging due to myriad design choices and computationally intensive training. In this paper, we propose an RL controller and an MPC controller for minimizing the electricity cost of a DCEP, and compare them via simulations. The two controllers are designed to be comparable in terms of objective and information requirements. The RL controller uses a novel Q-learning algorithm that is based on least-squares policy iteration. We describe the design choices for the RL controller, including the choice of state space and basis functions, that are found to be effective. The proposed MPC controller does not need a mixed integer solver for implementation, but only a nonlinear program (NLP) solver. A rule-based baseline controller is also proposed to aid in comparison. Simulation results show that the proposed RL and MPC controllers achieve similar savings over the baseline controller, about 17%.