Reinforcement Learning for Optimal Control of a District Cooling Energy Plant

TL;DR

This paper introduces a Q-learning based reinforcement learning controller for district cooling energy plants, achieving cost savings comparable to traditional MPC methods while enabling real-time control in continuous state and action spaces.

Contribution

It presents a novel RL controller tailored for continuous spaces and details its design choices, improving upon existing RL approaches for DCEPs.

Findings

RL controller reduces energy costs by approximately 8%.

Performance is comparable to model predictive control (MPC).

Designed for continuous state and action spaces.

Abstract

District cooling energy plants (DCEPs) consisting of chillers, cooling towers, and thermal energy storage (TES) systems consume a considerable amount of electricity. Optimizing the scheduling of the TES and chillers to take advantage of time-varying electricity price is a challenging optimal control problem. The classical method, model predictive control (MPC), requires solving a high dimensional mixed-integer nonlinear program (MINLP) because of the on/off actuation of the chillers and charging/discharging of TES, which are computationally challenging. RL is an attractive alternative to MPC: the real time control computation is a low-dimensional optimization problem that can be easily solved. However, the performance of an RL controller depends on many design choices. In this paper, we propose a Q-learning based reinforcement learning (RL) controller for this problem. Numerical simulation results show that the proposed RL controller is able to reduce energy cost over a rule-based baseline controller by approximately 8%, comparable to savings reported in the literature with MPC for similar DCEPs. We describe the design choices in the RL controller, including basis functions, reward function shaping, and learning algorithm parameters. Compared to existing work on RL for DCEPs, the proposed controller is designed for continuous state and actions spaces.