Reinforcement Learning Applied to an Electric Water Heater: From Theory to Practice

TL;DR

This paper demonstrates how reinforcement learning, combined with auto-encoder feature extraction, can effectively control electric water heaters for demand response, reducing energy costs by 15% in real-world experiments.

Contribution

It introduces a reinforcement learning approach with auto-encoder features for efficient control of electric water heaters, bridging theory and practical application.

Findings

Faster policy learning with reduced state space in simulations.

Achieved 15% energy cost reduction in a 40-day lab experiment.

Auto-encoder features help mitigate high-dimensional sensor data challenges.

Abstract

Electric water heaters have the ability to store energy in their water buffer without impacting the comfort of the end user. This feature makes them a prime candidate for residential demand response. However, the stochastic and nonlinear dynamics of electric water heaters, makes it challenging to harness their flexibility. Driven by this challenge, this paper formulates the underlying sequential decision-making problem as a Markov decision process and uses techniques from reinforcement learning. Specifically, we apply an auto-encoder network to find a compact feature representation of the sensor measurements, which helps to mitigate the curse of dimensionality. A wellknown batch reinforcement learning technique, fitted Q-iteration, is used to find a control policy, given this feature representation. In a simulation-based experiment using an electric water heater with 50 temperature sensors, the proposed method was able to achieve good policies much faster than when using the full state information. In a lab experiment, we apply fitted Q-iteration to an electric water heater with eight temperature sensors. Further reducing the state vector did not improve the results of fitted Q-iteration. The results of the lab experiment, spanning 40 days, indicate that compared to a thermostat controller, the presented approach was able to reduce the total cost of energy consumption of the electric water heater by 15%.