A Review of Reinforcement Learning for Autonomous Building Energy Management

TL;DR

This paper reviews how reinforcement learning has been applied to autonomous building energy management, highlighting recent advances, challenges, and future research directions in optimizing energy use with intelligent control algorithms.

Contribution

It provides a comprehensive overview of reinforcement learning applications in building energy management and discusses future challenges and research opportunities.

Findings

Reinforcement learning has been successfully applied to optimize building energy systems.

The review identifies key challenges and future directions in RL-based energy management.

The paper summarizes recent advancements and gaps in the literature.

Abstract

The area of building energy management has received a significant amount of interest in recent years. This area is concerned with combining advancements in sensor technologies, communications and advanced control algorithms to optimize energy utilization. Reinforcement learning is one of the most prominent machine learning algorithms used for control problems and has had many successful applications in the area of building energy management. This research gives a comprehensive review of the literature relating to the application of reinforcement learning to developing autonomous building energy management systems. The main direction for future research and challenges in reinforcement learning are also outlined.

Equations

$$V^{\pi }(s)=E[{\Sigma }_{k=0}^{\infty }{\gamma }^k{r}_{t+k+1}\mid s,\pi ]$$

$$V^{{\pi }^{\ast }}(s)=maxE[r_{t+1}+\gamma V^{{\pi }^{\ast }}(s_{t+1})\mid s,{\pi }^{\ast }]$$

$$Q(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha [r+\gamma \underset{a_t}{\max }Q(s_{t+1},a_t)-Q(s_t,a_t)]$$

$$Q(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha [r+\gamma Q(s_{t+1},a_{t+1})-Q(s_t,a_t)]$$

$${\delta }_t=r_{t+1}+\gamma V(s_{t+1})-V(s_t)$$

$$V(s_t)\leftarrow V(s_t)+\alpha {\delta }_t$$

$$p(s_t,a_t)\leftarrow p(s_t,a_t)+\beta {\delta }_t$$

$$\pi (s_t,a_t)=\frac{exp(\frac{Q(s,a)}{\tau })}{{\Sigma }_{b=1}^{B}exp(\frac{Q(s,a)}{\tau })}$$

$$L_i({\theta }_i)={\mathbb{E}}_{(s,a,r,s^{\prime })\sim U(D)}(r_i+\gamma ma{x}_{a^{\prime }}\hat{Q}(s^{\prime },a^{\prime };{\theta }_i^{-})-Q(s,a;{\theta }_i){)}^2$$

$$d\theta \leftarrow d\theta +{▽}_{{\theta }^{\prime }}log\pi (a_i|s_i;{\theta }^{\prime })(R-V(s_i;{\theta }_v^{\prime }))$$

$$d{\theta }_v\leftarrow d{\theta }_v+\partial (R-V(s_i;{\theta }_v^{\prime }){)}^2/\partial {\theta }_v^{\prime }$$

Full text

K****eywords reinforcement learning  $\cdot$ building energy management  $\cdot$ smart homes  $\cdot$ smart grid  $\cdot$ deep learning  $\cdot$ machine learning

1 Introduction

Meeting the global energy needs in an efficient and sustainable manner is one of the most pressing issues facing society today. The energy consumption from buildings accounts for approximately 40% of the total global energy consumption [1]. It is therefore imperative to find innovative ways to help reduce and optimize the energy consumed by buildings. Some of the main sources of energy consumption in buildings include: Heating Ventilation and Air Conditioning (HVAC), water heating, and lighting. Reducing the energy consumed by buildings has many benefits. The consumer is benefited with lower energy bills. Electricity providers benefit from reduced peak loads that must be met. There is also a wider benefit to society as a whole from reducing building energy consumption and reduction of emissions. Much of the energy that powers the grid comes from greenhouse gas emitting sources, e.g. coal. Therefore, a reduction in building energy consumption would also correspond to a reduction in the emission of greenhouse gasses such as carbon dioxide.

The prevalence of digital systems in every aspect of modern society is largely due to the steady reduction in the cost and size of micro-processors, along with their increased computing power. The widespread availability of computing has manifested itself in the area of building energy consumption in the form of advanced sensors and actuators, and building energy management systems [2]. These systems consist of monitoring and controlling the activity of each device within the building with the aim of reducing the demand placed onto the grid and the cost to the consumer. The difficulty of this management process is increased with the addition of photovoltaic (PV) panels, batteries, electric vehicles, and smart appliances. Building energy management systems are comprised of many components. Sensors are needed to monitor the building, e.g. temperature, humidity and load sensors. Communication is needed between devices in order to facilitate the monitoring and scheduling of their operation. This communication between devices is an example of the “Internet of Things” (IoT). Smart meters are needed to record energy consumption and to communicate with energy providers. Many devices are equipped with rudimentary control mechanisms such as rule-based decision systems. For example, the thermostat that controls a HVAC system turns heating and cooling on/off when the temperature of the room crosses a user-defined threshold. This paper will provide an overview of how more sophisticated machine learning control systems, in particular Reinforcement Learning (RL), have improved the operation of building energy management systems.

Reinforcement Learning (RL) is a form of machine learning that consists of an agent interacting in an environment, learning what actions to take depending on the state of the environment [3]. The agent learns by trial and error and is rewarded for taking desirable actions. The environment is often modelled as a Markov Decision Process (MDP). RL algorithms date back to the 1970s and 1980s. One of the more popular RL algorithms, Q Learning, was first proposed in 1989 [4]. These algorithms have been applied to a wide range of problems over the years, everything from traffic light control [5] to watershed management [6]. The combination of RL algorithms with deep neural networks has significantly increased the effectiveness of RL methods and has enabled them to be applied to tasks involving computer visions, e.g. self-driving cars. Recent notable achievements of RL have resulted in a significant amount of research into RL and its applications. These achievements include mastering the game of Go and playing Atari games.

Many researchers have since begun applying reinforcement learning to some of the challenging areas of building energy management. RL has been applied to tasks such as HVAC control, water heater control, electric vehicle charging, lighting control and appliance scheduling. An advantage of applying RL to address these problems is that the algorithm learns by itself what the best control policy is. When implementing more traditional rule-based approaches, the designer must handcraft the thresholds that the system will adhere to. This naïve approach is not necessarily able to minimize energy consumption to the extent that RL can. There are several factors that increase the complexity of applying RL to these problems, such as identifying what state information is needed, conflicting objectives and simulator design. These will be discussed in detail throughout the paper. The main contributions of this research are:

To give a comprehensive review of the literature relating to the application of RL to building energy management. 2. 2.

To quantify the impact of RL on building energy management systems in terms of energy savings. 3. 3.

To establish the limitations of RL for building energy management and outline potential areas of future research.

The outline of the paper is as follows. Sections 2 will give outline the research area of building energy management. An overview of reinforcement learning will be provided in Section 3. Section 4 will give a comprehensive account of the applications of reinforcement learning within the area of building energy management. Section 5 will discuss some of the limitations and potential areas for future work concerning the application of reinforcement learning to building energy management. Finally, Section 6 will outline what can be concluded as a result of this research.

2 Autonomous Building Energy Management

The development of autonomous building energy management systems has received a lot of interest in recent years for a number of reasons, one of the primary reasons being the drive to increase the energy efficiency in buildings through operational methods. As stated in the introduction, energy consumption in buildings accounts for a significant portion of the total worldwide energy consumption [1]. There are a number of factors that influence how energy efficient a building is, e.g. insulation, construction materials and overall design. The aim of developing autonomous building energy management systems is to further reduce both overall energy consumption and the cost of powering buildings by utilizing advancements in technologies such as:

• •

Autonomous control systems

• •

Internet of Things (IoT)

• •

Batteries

• •

Smart grid

2.1 HVAC

Heating Ventilation and Air Conditioning (HVAC) is one of the most energy intensive consumers of energy in buildings. It is estimated that HVAC is responsible for 50% of building energy consumption in the US and between 10 - 20% of overall consumption in developed countries [7]. This is due to expectation of thermal comfort in developed countries, rather than it being regarded as a luxury [8]. It is also well known that there is a strong correlation between outside temperatures and HVAC energy consumption [9]. This energy consumption is expected to increase due to the increased number of extreme weather events observed globally [10]. The most basic HVAC control system would be a threshold-based approach in which a thermostat is used to regulate the temperature. If the temperature crosses some predefined threshold, the HVAC system activates to heat/cool the environment as required. There have been a wide number of strategies proposed to control the operation of HVAC systems [11]. Section 4 will outline previous applications of RL to HVAC control.

2.2 Lighting and Appliances

There are of course many other energy intensive devices within the buildings besides HVAC. Much of the energy consumed in domestic dwellings comes from: lighting, televisions, water heaters, washing machines, dryers, fridge freezers, etc. In order to reduce energy consumption of appliances, the design of these devices has been significantly improved in recent years. For example, some dryers are designed to have moisture sensors to prevent over drying. A significant amount of effort has focused on changing consumer behaviour, e.g. turning lights off when the room is not in use or turning off television sets when not in use. It is known that consumer behaviour plays a significant role in energy consumption [12]. Previous studies have shown that providing regular feedback to consumers of their energy consumption can reduce their consumption by 15% on average [13]. Another crucial issue with regards to the energy consumption of appliances is the variation in energy demand throughout the day. This affects the load demand placed on the electrical grid. This will be discussed further in Section 2.5. This provides an opportunity for autonomous building energy management systems to help reduce these peak demands by managing energy consumption, e.g. dimming lights at peak times. Autonomous control systems are now being developed to effectively schedule appliances to minimize energy cost [14].

2.3 Electric Vehicles and Batteries

The automotive industry is also heavily influencing the behaviour of residential energy consumption due to the current transition from fossil fuel powered vehicles to electric vehicles. The need to draw power from the grid in order to charge electric vehicles significantly increases the electrical load placed on the grid. Grahn et al. investigated the affect Plug in Hybrid Electric Vehicles (PHEVs) have on the domestic energy consumption in contrast to other appliances in the home [15]. The authors of this study estimate that if all of the 4.3 million vehicles in Sweden were electric, this would correspond to an additional 34 GWh of electricity consumption. This is approximately 10 % of Sweden’s daily electricity consumption [15]. This is a significant additional load for the grid to adapt to.

Vehicle to grid (V2G) technology is a promising solution to address this problem [16]. Vehicle to grid technology works by selling electricity back to the grid when the car is not being used. A variation V2G is to consider the electric vehicle as a deferrable load. When the grid needs more power, charging of electric vehicle can stop for a few seconds or minutes. In this process there is no actual injection of power from the vehicle to the grid, but a pause in charging that can help the grid. V2G also has the potential to alleviate some of the load on the grid during peak demand. This further motivates the need for effective building energy management systems. For example, a potentially effective strategy would be to charge the vehicle overnight when electricity is cheapest and to sell energy back to the grid at 7 pm when demand is high. Electric vehicles could also contribute to address the problem of incorporating variable renewable energy into the grid. The variability of renewable sources is a challenge to incorporating technologies such as Photovoltaic (PV) into the grid [17]. This will be discussed further in Section 2.5. Battery technology in general is expected to play a major role in addressing some of the issues raised above [18]. The Tesla Powerwall is a prime example of this [19]. Energy management systems have already been developed that utilize battery technology to help manage the electrical demand [20].

2.4 Smart Meters and the Internet of Things

Smart meters are a key development that will enable the large-scale deployment of autonomous building energy management systems [21]. They enable communication with electricity providers that aids their operation. Smart meters also enable the control of appliances within the home. Smart meters are already in widespread use. Italy and Sweden were two of the first to countries to have completed their deployment of smart meters nation-wide, with many other countries making significant progress in their deployment [22].

The term “Internet of Things” or IoT, refers to a network of devices, appliances, sensors and electronics that can connect with one another [23]. It is this communication between devices that enables building energy management systems to operate.

2.5 Electrical Grid and Renewables

One of the main challenges when developing autonomous building control systems is integrating it with the electrical grid. These systems need to schedule their operation in order to minimize the cost of electricity to the consumer. This is further complicated with many individuals installing PV solar panels, as previously discussed. This gives rise to the recent paradigm of “prosumers” (producers and consumers) [24]. Within the prosumer paradigm, individuals both consume energy from the grid and produce energy which is sold to the grid. This creates issues of instability within the grid which must be addressed [25]. This is due to feeders with high levels of PV penetration producing an excess of electricity that the grid struggles to cope with [26].

Another factor which increases the complexity of implementing autonomous building control systems is the price of energy [27]. Electricity prices vary based on demand and are therefore considered by many control systems [28]. Weather also has a significant effect on energy consumption. As stated previously, energy consumption via HVAC increases significantly with both exceedingly high and low outside weather temperatures [9]. In addition to this, sunny weather results in a higher output from PV systems, this further exacerbates the problems mentioned relating to grid instabilities [25].

The next section will outline Reinforcement learning, a popular machine learning method that has been applied to many problems relating to smart grid, building energy management, and smart homes.

3 Reinforcement Learning

Reinforcement Learning (RL) is a subgroup of machine learning research that involves an agent learning by itself what actions to take in an environment so that it maximizes some reward [3]. This typically involves a significant amount of trial and error from the agent as it learns what actions result in the highest reward. Figure 1 illustrates this interaction with its environment. Algorithm 1 presents a generic pseudocode, outlining the broad steps taken in a typical RL algorithm. The agent interacts with its environment in discrete time steps. An agent typically has a policy ($\pi$) which determines what actions it will take. The goal is then to find the optimum policy $\pi\textsuperscript{*}\in\Pi$, where $\Pi$ is the set of possible policies. The value of a policy $\pi$ in a given state s is calculated using the value function in Equation 1.

[TABLE]

Where E is the expected future return and $\gamma$ is the discount factor. In order for the agent to assess the value of its current state, it must also consider the expected future rewards it will obtain if it follows its current policy. Equation 1 quantifies this mathematically. The discount factor $\gamma$ determines how much weighting the agent gives to future rewards, as discussed in Section 3.1. The value function of state s for the optimum policy $\pi^{*}$ is stated in Equation 2, calculated using the Bellman equation.

[TABLE]

One of the first significant achievements of RL was its application to learn to play backgammon in 1995 by Gerald Tesauro [29]. Some of the more recent achievements include learning to play Atari games at a level comparable to an expert human player [30] in 2015 and mastering the game of GO [31] in 2016 by beating the world champion Lee Sedol. This is by no means a comprehensive list of the many achievements of RL algorithms. RL has been successfully applied to a vast range of problems including building energy management, which will be discussed later.

3.1 Problem Types

The most common way to model an RL problem is as a Markov Decision Process (MDP) [3]. A MDP is a discrete time framework for modelling decision making. A MDP is defined as a tuple $\langle S,A,T,R,\gamma\rangle$, where $S$ is the set of states the environment can be, A is the set of possible actions, $T=Pr_{a}(s,s^{\prime})=Pr(s_{t+1}=s^{\prime}\mid s_{t}=s,a_{t}=a)$ is the probability that taking action a in state s will lead to state s*′* at time t+1, and finally $R=R_{a}(s,s^{\prime})$ is the reward received after transitioning from state s to s*′*. RL agents account for future rewards using a discount factor $\gamma\in[0,1]$, a higher value makes the agent more forward thinking.

An extension to the standard MDP is the Partially Observable Markov Decision Process (POMDP) [32]. In POMDPs it is assumed that the underlying problem is a MDP, however the agent does not have complete knowledge of the state of the environment. A POMDP is defined as the tuple $\langle S,A,T,R,\Omega,O,\gamma\rangle$. Here $\Omega$ and $O$ are the set of observations and the set of conditional observation probabilities respectively. When the environment transitions to state s*′*, the agent receives the observation $o\in\Omega$ with probability $O(o\mid s^{\prime},a)$.

For problems that require multiple agents, RL is often combined with Multi-Agent Systems (MAS) [33]. This is referred to as Multi-Agent Reinforcement Learning (MARL) [34]. One of the key challenges of MARL is to ensure that agents coordinate their actions to achieve a global optimal result. This can be difficult to achieve in practice as each agent may only have partial information of the environment, making it a decentralized problem. These sorts of problems can be modeled as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) [35]. Dec-POMDPs are defined by the tuple $\langle I,S,\textbf{A},T,R,\boldsymbol{\Omega},O,h\rangle$. In this case, $I$ is the set of agents, $\textbf{A}=\times A_{i}$ is the joint action space, $\boldsymbol{\Omega}=\times\Omega_{i}$ is the joint observation space and finally h is the horizon.

Another type of RL problem domain are problems that consist of multiple tasks, i.e. Multi Task Learning (MTL). In this type of problem, each task $T_{j}$ has a tuple $\langle D,T_{j},R_{j},O_{j}\rangle$ where $D$ is the underlying domain, $T_{j}$ is the task specific transition, $R_{j}$ is the task specific reward and $O_{j}$ is the task specific observation. This is closely related to Transfer Learning (TL), in which the goal is to transfer knowledge learned when solving one problem to solve a different problem [36]. The distinction being that in MTL, the tasks are variants of the same underlying problem.

Many RL problems have multiple objectives. These problems are referred to as Multi-Objective RL (MORL) problems [37]. In MORL, the MDP is extended to be a Multi-Objective MDP (MOMDP) where the agent receives a vector reward signal $\textbf{R}(s,a)=(R_{1}(s,a),R_{2}(s,a),...,R_{m}(s,a))$ where m is the number of objectives. The goal here is to learn multiple policies simultaneously for each objective [38, 39].

The task of learning, itself can be viewed as a learning problem. This is referred to as Meta learning [40]. Meta learning has been applied to RL algorithms. For example the RL2 algorithm consists of a fast and slow RL algorithm [41]. The fast RL learner learns how to complete the task while the slow RL learner learns the optimum parameters of the fast learner.

Multi-Agent RL problems can also be either cooperative or competitive [42]. In cooperative MARL, each agents’ objective aligns with one another either in the form of a shared goal or goals that do not conflict with one another. In competitive environments, agents have conflicting objectives whereby a gain for one agent results in a loss for another.

3.2 Model-Free vs Model-Based

RL algorithms can be subcategorized into two groups, model-based and model-free algorithms. In model-based algorithms, the agent learns a model of the environment based on its observing how the state of the environment changes when certain actions are taken. These observations are used to estimate the environments state transition function $T(s^{\prime}\mid s,a)$ and reward $R$. Once the algorithm learns a model of the environment, it can be combined with a planning algorithm to decide what actions to take [43]. Examples of model-based algorithms include: Explicit Explore and Exploit (E 3) [44], Prioritized sweeping [45], Dyna [46] and Queue-Dyna [47].

Model-free approaches do not need to develop a model of the environment. Instead model-free approaches learn a policy, via trial and error, with the aim of approximating the optimum policy. Many of the routinely used RL algorithms are model-free, e.g. Q Learning [4] and SARSA [48]. Model-free approaches are more popular in the literature as they are generally less computationally expensive. Model-based approaches first require building an accurate model of the problem, which can be difficult. Once an accurate model of the environment is obtained, the algorithm must find the optimum policy by planning ahead.

3.3 Discrete and Continuous Search Spaces

Many RL algorithms, such as Q learning [4], represent the values of each state action pair in a look up table (a Q table for Q learning). This approach works well for problems where the state action space is small. However, a fundamental limitation with the tabular approach for representing Q values is its scalability. It’s clear that when the number of states and actions increase, the size of the table quickly becomes exceedingly large. The state action space becomes infinite for problems with continuous variables. The widely adopted solution to this is to replace the Q table with a function approximator, most commonly a neural network. When implementing a neural network to represent the Q function, the network reads in the state of the environment as input. The output of the network is the Q value for each action. This function approximation allows RL algorithms to deal with exceedingly large state spaces such as those with images. This is the approach taken for all of the recent work in deep RL and Atari games [30]. The image in Figure 2 illustrates a deep neural network that can be used as a function approximator for RL algorithms.

Function approximation works well for problems with continuous state spaces. This approach still presents a problem for domains with continuous action spaces. One approach would be to discretize the action space, however this will limit the performance of the RL algorithm. A number of RL algorithms have been proposed to address problems with continuous action variables, including: Deep Deterministic Policy Gradient (DDPG) [50] and Sequential Monte Carlo (SMC) [51].

3.4 Q Learning

One of the most prominent RL algorithms is Q Learning [4]. The Q Learning algorithm has the properties of being an off-policy and model-free reinforcement learning method. Off-policy refers to an agent that learns the value of its policy independent of its actions [52]. When applied to discrete state action spaces, the agents policy is determined by its Q table, a matrix where $Q={S\times A}$. These Q values are used to select what action to take in a given state. Once the agent takes an action (a) for the current environmental state (s), the agent receives a reward (r). The agent then updates it’s Q table based on Equation 3.

[TABLE]

Where $\alpha\in[0,1]$ is the learning rate and $\gamma\in[0,1]$ is the discount factor. It is also common to implement Q Learning with an $\epsilon$-greedy selection policy where a random action is selected with probability $\epsilon\in[0,1]$. The purpose of this is to encourage exploration. Another option is to use a softmax action selection policy, outlined in Section 3.6.

3.5 SARSA

Another popular RL algorithm is SARSA (State Action Reward State Action) [48]. As is the case with Q Learning, SARSA is a model-free learning algorithm. Unlike Q Learning, SARSA is an on-policy learning approach meaning that it estimates the value of its policy at a given time while using that particular policy. SARSA updates its Q values using Equation 4.

[TABLE]

Where each parameter has the same meaning as in Equation 3.

3.6 Actor Critic

Actor Critic (AC) is a RL algorithm that consists of two agents: an actor and a critic. The actor makes decisions based on its observations of the environment and current policy. The purpose of the critic is to observe both the state of the environment and the reward obtained from the environment based on the actor’s decision. The critic then gives feedback to the actor [53]. Actor critic is an on-policy and model-free RL algorithm.

In actor critic learning, the actor takes an action a. This results in a Temporal Difference (TD) error, calculated using Equation 5.

[TABLE]

Where V is the current estimate of the state value. The critic then updates its estimate of the value function using Equation 6.

[TABLE]

Where $\alpha$ is the learning rate. The actor also uses the TD error to update its preference (p) to select that action again via Equation 7

[TABLE]

Where $\beta\in[0,1]$. The actor then uses this preference in combination with a softmax selection method to choose its action. Its policy is outlined in Equation 8.

[TABLE]

Where $\tau$ is the temperature and is used to calibrate the Softmax selection policy. Lower $\tau$ values increase the likelihood of selecting the best estimated action while higher values make each action selection equally likely. As stated in Section 3.4, $\epsilon$-greedy is another popular action selection policy. While Figure 1 illustrates how an agent generally interacts with its environment, Figure 3 specifically illustrates how AC agents interact with their environment.

3.7 Recent Methods

The algorithms outlined in the previous sections are still routinely used in the literature. There have been some significant advances made in RL that have led to new and powerful techniques. The most significant of these being the combination of RL with deep neural networks as previously mentioned in Section 3.3. One of the more popular recent methods is the Deep Q Network (DQN) algorithm [30]. DQN utilizes both an experience replay of past state action pairs to evaluate policies and a periodic update for the target network. The parameters ($\theta$) of the policy function (Q) are updated by calculating the error between the policy network (Q) and the target network (Q̂) parameterised by $\theta^{-}$, as shown in Equation 9.

[TABLE]

Where $\mathbb{E}_{(s,a,r,s^{\prime})\sim U(D)}$ is an episode $(s,a,r,s^{\prime})$ sampled uniformly from the dataset D of previous experiences. The DQN algorithm was applied to a range of Atari games and found to exceed human level performance.

Another recent RL algorithm is the Asynchronous Advantage Actor Critic (A3C) algorithm [54]. As mentioned in Section 3.6, actor critic methods consists of a policy $\pi(s)$ (actor) and a value function $V(s)$ (critic). This algorithm utilizes multiple learners that interact with their environment at the same time. The “Advantage” portion of the A3C algorithm consists of estimating how much better an action was than expected, calculated as $A=R-V(s)$. The “Asynchronous” portion of A3C comes from the fact that each learner is running individually in parallel. This dramatically speeds up learning. Each learner is initialized to the global policy. They then interact with each individual environment and calculate the value and loss. This is then used to calculate the gradients for the policy (Equation 10) and value function (Equation 11).

[TABLE]

[TABLE]

Where $\theta$ and $\theta_{v}$ are the parameters of the global policy and value function, and $\theta^{\prime}$ and $\theta^{\prime}_{v}$ are the parameters of the thread specific individual policy and value function. These are used to asynchronously update the global $\theta$ and $\theta_{v}$.

DQN and A3C are two state of the art deep RL methods. The aim of this section was to give an overview of some recent developments in RL and is by no means a comprehensive review of RL as this is outside of the scope of this paper. For further reading on the state of the art in RL see the work of Arulkumaran et al. [55]. The next section will give a comprehensive account of the applications of RL to building energy management systems.

4 Autonomous Building Energy Management via Reinforcement Learning

An overview of both the problem of building energy management and RL was provided in Sections 2 and 3. This section will now provide a comprehensive review of the literature that combines these two areas of research.

4.1 HVAC

Reinforcement learning has been successfully applied to many areas of building energy management, including HVAC control. Typically, the environment states for the RL algorithm include factors such as: time of day, outdoor temperature, indoor temperature, weather forecast and occupancy. Typical RL actions include: temperature set points, air flow control, heating control and cooling control. The RL algorithm also needs some sort of reward to operate, all previous studies in the literature calculate rewards based on energy cost, thermal comfort or a combination of both.

The first application in the literature of RL to HVAC control was in 1996. Anderson et al. applied Q learning in conjunction with a PI controller to modulate the output of the PI controller for a heating coil [56, 57]. This approach performed better than the PI controller alone. A 2006 study applied a combined RL - Model Predictive Control (MPC) approach to control HVAC operation at the Energy Resource Station Laboratory building in Ankeny, Iowa [58]. It was found that the combined approach performed better than either Q learning or MPC individually. An actor critic - neural network learning approach was applied to adjust the signal of a local control for HVAC control in 2008 by Du and Fei [59]. This study reports significant improvements from a combined PID actor critic learning approach than a stand-alone PID controller. Dalamagkidis and Kolokotsa implemented the Recursive Least-Squares Temporal Difference (RLS TD) for HVAC control in 2008 and reported better performance with an RL controlled HVAC system than with a fuzzy PD controller [60]. Yu and Dexter implemented a Q($\lambda$) learning approach with fuzzy discretization of the state space variables to select rules that determine the operation of the HVAC system [61]. A 2013 study by Urieli and Stone apply Tree Search to control a HVAC heat pump [62]. The Tree Search method resulted in a 7 - 14% saving when compared to the standard rule-based control. In 2015 Barrett and Linder applied Q learning to the problem of HVAC control combined with Bayesian Learning for occupancy prediction [63]. The results outline a 10% energy savings improvement over a programmable control method. This work was then extended to apply parallel Q learning to HVAC control [64]. Another 2015 study by Ruelens et al. implemented a combined Auto-Encoder (AE) Q learning to control a heat pump for a HVAC system [65]. The authors reported energy savings in the range of 4 - 11%, comparable to the previous study.

Yang et al. implemented a Batch Q learning with a neural network to control a PV powered heating system [66]. The results report a 10% improvement on a standard rule-based system. In 2017 Wei et al. utilized deep neural network and applied Deep RL (DQN) to the problem of HVAC control and report energy saving improvements over conventional Q learning in the range of 20 - 70% [67, 68]. Another 2017 study by Wang et al. applied a Monte Carlo Actor Critic with Long Short Term Memory (LSTM) neural network to the task of HVAC control [69]. Their results indicate a 15% thermal comfort improvement and a 2.5% energy efficiency improvement when compared to other methods. More recently, Marantos et al. proposed using a Neural Fitted Q-Iteration (NFQ) approach for HVAC control in 2018 [70]. This approach also used a neural network to approximate the Q function. The results of this study report significant savings in terms of both energy consumption and thermal comfort when compared to rule-based controllers. In 2018, Zhang et al. applied Asynchronous Advantage Actor Critic (A3C) (a deep RL approach) to control the HVAC system for a simulation of the Intelligent Workplace building in Pittsburgh, USA [71]. The authors report a 15% energy saving improvement upon the base case. Chen et al. applied Q learning to control both the HVAC and window systems [72]. The authors report energy savings of 13 and 23% and lowered discomfort ratings by 62 and 80% in the two buildings tested. In a 2018 study, Patyn et al. compared the performance difference of Q learning on three different neural network architectures (convolutional neural networks (CNN), LSTM networks and a feed forward multi-layer neural network) for HVAC control [73]. Their results indicate that the LSTM and multi-layer network perform best however the LSTM training time was twice as long.

This section has outlined numerous successful applications of RL to HVAC control with a variety of RL algorithms. The most common RL algorithm implemented was Q learning. The vast majority of studies report energy savings in the range of $\approx 10\%$ when compared to rule-based approaches. There is also a recent trend to apply deep RL to HVAC control, which has outperformed traditional tabular RL. This trend is unsurprising given the recent success and attention that deep learning has received.

4.2 Water Heater

As discussed in Section 2, water heaters consume a significant amount of energy. This section will explore some of the applications of RL to control water heaters with the aim of reducing energy costs. Some of the state variables include: time of day, current water temperature and forecast usage. The action that the RL agent makes is generally to turn the heater on or off. The reward given to the agent is the electricity consumption.

A 2014 study by Al-Jabery et al. applied Q learning and a fuzzy Q learning to control a water heater [74]. Their results indicate that their proposed fuzzy Q learning algorithm gives smoother convergence than the standard Q learning. Al-Jabery et al. then explored applying an Actor Critic - Q learning approach to control a water heater in 2017, where they included the grid load in the state space for the RL agent [75]. This study reports energy cost savings between 6 and 26% using RL. In 2014 Ruelens et al. applied batch Q learning to control a cluster of 100 water heaters [76]. The authors reported that within 45 days, the RL algorithm reduced electricity costs when compared to a hysteresis controller. In a more recent 2018 study, Ruelens et al. applied Q iteration to a physical water heater in a laboratory setting and found that Q iteration was able to reduce the energy consumption by 15% over 40 days when compared to a thermostat controller [77]. De Somer et al. applied Q learning to control the heating cycle of a domestic water heater to maximize the consumption from a local PV source. The authors then evaluate the Q learning agent on 6 residential houses and report a 20% increase in the amount of local PV consumption [78]. Kazmi et al. did a similar study in the Netherlands where deep RL was applied to 32 homes [79]. The authors report a 20% reduction in the energy consumption for water heating and no loss in user comfort.

Although there are significantly fewer studies in the literature applying RL to water heaters than HVAC. Based on the papers reviewed, there appears to be a significant reduction in energy consumption when applying RL to water heaters. Many of the studies report reductions of $\approx 20\%$ when compared to the baseline. A similar trend can be observed that recent research is more focused on the application of deep RL as it has demonstrated its effectiveness.

4.3 Home Management Systems

The studies outlined in Sections 4.1 and 4.2 consider applying RL to HVAC and water heaters respectively in isolation. This section explores applications of RL within the home to manage multiple appliances, lighting, PV, battery, etc. Applying RL in this manner is needed as the task of building energy management is a complex problem with multiple factors that must be managed to reduce the overall energy consumption. Similar to the previously mentioned sections, the states for the RL algorithm when exploring building energy management in a more holistic manner, generally consist of the time of day, temperature information and the current usage state of the various appliances. Many of these studies also include other state information such as electricity prices, grid load and information in relation to PV panels such as solar irradiance. The actions available to the RL agent in the studies outlined in this section are similar to those outlined in the previous sections, i.e. turning a device or appliance on or off. In the cases of studies which incorporate batteries, the actions are to charge/discharge the battery or to do nothing.

In 2018, a study by Reymond et al. applied Fitted Q Iteration to learn to schedule a number of household appliances including a heat pump, water heater and dishwasher [80]. Their results indicate that independent learning performs 9.65% better than a centralized learning approach. Wei et al. implemented a dual iterative Q Learning algorithm for residential battery management [81]. The authors report a 32.16% savings in energy cost when compared to the baseline. A 2015 study by Guan et al. utilize temporal difference learning to control the energy storage of a battery in the presence of a PV panel [82]. It was found that temporal difference learning resulted in improvements of 59.8% reduction in energy costs. Wan et al. implemented actor critic learning with two deep neural networks as a residential energy management system [83]. In this study, the RL algorithm learned when to purchase electricity from the utility provider based on previous prices, house hold loads and current battery charge. The authors report an 11.38% reduction in energy cost by using actor critic learning when evaluated over 100 days. Remani et al. applied Q learning to schedule multiple devices within the home including lighting, clothes dryer, dish washer, etc. [84]. The authors implemented a price based demand response and included a PV panel in the system model and reported an approximate daily energy cost saving of 15%. Wen et al. proposed a demand response energy management systems for small buildings that enables automated device scheduling in response to electrical price fluctuations [85]. The authors implement Q learning with the aim of taking advantage of the estimated 65% of potential energy savings for small buildings by efficient device scheduling and report improvements upon the baseline. Bazenkov and Goubko utilize inverse reinforcement learning (IRL) to predict consumer appliance usage and report a higher accuracy using IRL than other machine learning methods such as random forest [86]. A 2018 study by Mocanu et al. implement both Deep Q Learning (DQL) and Deep Policy Gradients (DPG) to optimize the energy management system for 10, 20 and 48 houses [87]. This study explored the use of electric vehicles, PV panels and building appliances. The authors report electricity cost savings of 27.4 % for DPG and 14.1% for DQL.

A combined approach was proposed by Wang et al. that implements Q learning to manage a battery in a residential setting with installed PV [88]. Q learning was combined with a model-based control algorithm to minimize energy costs. This hybrid approach demonstrated significant cost savings in the range of 23 - 72%. A similar study was conducted by Mbuwir et al. that implemented fitted Q iteration for battery management in order to maximize the energy provided by PV [89]. This was simulated using Belgian consumer data and demonstrated a cost saving of 19%. Rayati et al. utilize Q learning for home energy management in a setting with PV installation and energy storage [90]. This study also considers home comfort and CO2 emissions when learning the optimal control policy. The authors report a maximum energy saving of 40%, peak load reduction of 17% and CO2 social cost reduction of 50%. Sheikhi et al. implement Q learning to control a smart energy hub that consists of a combined heat and power, auxiliary boiler, electricity storage and heating storage [91]. The authors report a saving of 30% and 50% for energy cost and peak loads respectively.

The results outlined in the literature relating to the application of RL to home energy management varies significantly more than in the previous two sections. The highest reported energy cost reduction was 72% when implementing RL with a model-based controller [88]. There is a greater capacity to generate savings when applying RL to home energy management systems with batteries and solar than simply applying RL to devices. This is reflected in the studies mentioned here. In terms of algorithms, Q learning is again the most routinely used method.

4.4 Smart Homes and the Electrical Grid

While the studies in the previous sections have demonstrated the effectiveness of RL when applied within the home, many studies in the literature examine how RL can also have a positive impact when considering how these smart homes are incorporated into the grid. This section will outline the applications of RL that relate to smart homes and the grid. Many of these studies evaluate control systems for a community of homes, rather than single homes in isolation, and therefore utilize multi-agent RL in many cases. There are many studies that focus on the application of RL to the operation of a microgrid to address problems such as thermal generator scheduling and incorporating PV into a microgrid. These are outside of the scope of this paper and will therefore not be discussed. This section solely focuses on studies that explore incorporating smart homes into the electrical grid. For a more general review of smart grid technologies, a comprehensive review is provided by Tuballa et al. [92].

Jiang et al. implemented a hierarchical multi-agent Q learning approach for dynamic demand response and distributed energy resource management in a microgrid [93]. This microgrid also includes wind power and a battery. This study reports a 19% reduction in energy costs for the whole community. Kim et al. implemented a multi-agent Q learning approach for residential energy consumption based on electricity prices in a microgrid [94]. Significant savings were reported over the baseline myopic purchasing strategy. A multi-agent RL approach was implemented by Anvari-Moghaddam et al. to integrate intelligent homes into the grid where batteries and renewables are also considered [95]. The authors’ proposed structure consisted of: a central coordinator agent, a building management agent, a renewables agent, a battery agent and a services agent. The authors report an energy cost saving of 5%. A recent 2018 paper by Prasad and Dusparic implement a multi-agent deep RL approach to enable homes to share energy with one another in order to minimize cost for the community as a whole [96]. The authors implemented a multi-agent DQN approach where each agent controlled a house. The agent could consume/store energy, request energy from or grant energy to neighbours, deny energy to neighbours or purchase from the grid. Each home was had an installed battery. The results indicate savings of 97 and 156kWh during the winter and summer respectively for a 10 home community.

Lu et al. recently implemented Q learning for electricity pricing with the aim to schedule pricing to maximize the providers profit when purchasing from an electricity wholesaler and minimize customers cost [97]. A 2018 study by Kofinas et al. implemented fuzzy Q learning to control the operation of a battery and desalination plant in conjunction with a PV panel for a microgrid that also includes residential electricity consumers [98]. Claessens et al. applied Q learning with a convolutional neural network for residential load control [99]. It was found that the proposed approach provided near optimum performance when compared to a theoretically optimal benchmark. In 2018, Kim and Lim utilized Q learning to learn when to charge and discharge a battery and when to buy and sell from the grid [100]. The authors report significant energy cost savings when compared to other methods.

The studies outlined in this section focus on how RL can reduce energy costs for multiple smart homes and how RL can help integrate these homes into the grid. As was observed in the previous sections, the studies here also demonstrate the effectiveness of RL for these problems. Multi-agent RL is a commonly used approach to learn policies for multiple homes as it is well suited to the distributed nature of this control task. Based on these studies, electricity cost savings can range from 5% - 19%. In terms of utilizing RL to balance residential loads with the grid, it was demonstrated that RL performs with near optimum performance and significantly reduces electricity costs for the grid.

5 Discussion

A common theme across all of the various applications of RL within building energy management systems is that RL can provide significant savings in each problem it is applied to. As new and effective RL algorithms are developed, these new methods gradually make their way into the smart homes literature, e.g. deep RL methods. This section will outline some of the limitations of RL for building energy management and also some directions for future research.

5.1 Limitations

There are however limitations of applying RL to building energy management systems. The overwhelming majority of studies outlined in this paper discuss applications of RL to simulated versions of problems relating to building energy management. While this is perfectly acceptable and a natural way to apply RL to these problems, this approach relies heavily on accurate simulator design and data that is representative of real-world scenarios. Since RL is an online learning algorithm, it could be directly applied to a building energy management system for a physical building without ever learning within a simulated environment. The problem with this approach is that in order for the RL algorithm to learn an effective control policy that minimizes energy cost, it must learn by trial and error. This would result in an initial exploration period where the RL agent would evaluate different policies, many of which would have an unacceptably high energy cost associated with it. Accurate simulators are needed for this reason, so that the RL agent can learn in simulation which policies are best. These pre-trained agents can then further refine their policies when implemented in physical systems. This is a viable approach as was demonstrated by the application of RL to the HVAC operation at the Energy Resource Station Laboratory building in Ankeny, Iowa [58].

5.2 Future Directions

The first and most obvious trend that will be experienced relating to the application of RL to building energy management is the move towards deep RL algorithms. These deep RL methods are capable of learning more complex policies that are more sophisticated than those represented by shallow neural network or look up tables. As the volume of data collected by sensors will only increase, deep RL methods will become necessary to develop effective policies when interacting in environments with very large state action spaces. It is worth noting here that increasingly powerful computers will also enable researchers to train RL agents with increasingly complex policies.

Another potential route for future research in RL for building energy management is to apply many of the variants of traditional RL, some of which were outlined in Section 3.1. For example, based on our extensive literature review, there are no examples within the literature that apply multi task RL to building energy management. Multi-task learning could be potentially useful for learning different HVAC control policies depending on how many occupants are in the house. Transfer learning could be useful for transferring knowledge learnt when learning to schedule appliances to scheduling water heaters.

Some of the problems relating to building energy management can be framed as multi-objective RL problems. The most obvious example of this is the trade of between thermal comfort and energy cost. There are numerous examples in the literature that apply multi-objective optimization algorithms to address this problem, however none of these studies utilize any state of the art multi-objective RL algorithms, e.g. Pareto Q learning [37].

An additional avenue for future research would be the application of meta learning to RL problems in building energy management, e.g. the RL2 algorithm mentioned in Section 3.1 [41]. There are no applications of meta reinforcement learning to building energy management. Methods such as RL2 could be useful for learning more effective control policies for the various within the studies outlined in this paper.

In terms of the tasks that RL is applied to, many studies in the literature are beginning to explore implementing multi-agent RL to control a community of homes rather than a single home in isolation. Developing control policies for groups of homes has the added benefit of allowing homes to pool their resources in a manner that results in greater savings for everyone. A natural route for this research to follow into the future would be to apply multi-agent RL to larger communities of homes to explore the scalability of multi-agent RL in this domain. Conducting game theory analysis in this area would also be a promising area of research. Decentralized multi-agent control policies for large communities of homes would rely heavily on cooperation to achieve cost savings for all, it would therefore be worthwhile to explore the issue of cooperation and defection in such systems.

Based on the extensive literature review conducted, all of the current research relating to applications of RL in building energy management evaluates systems in which the environment does not significantly change outside of its normal environmental conditions. This raises the question of how well do these learned control policies perform when faced with significant environmental changes, e.g. extreme weather events, battery degradation or failure, PV panal installation or failure, homes where the number of occupants increases or decreases, etc. It is not clear based on the current research, how the learned control policies can adapt to these significant environmental changes, i.e. are the learned policies brittle and would result in catastrophic failure with sub optimal performance in the event of these changes? If the RL agent is robust and capable of adapting to such environmental changes, how long would it take for the RL agent to relearn new policies in such scenarios? These questions would be worth investigating in depth in future studies.

6 Conclusion

This paper has discussed the key ideas behind developing building energy management systems, given a brief introduction into reinforcement learning and provided a comprehensive account of the applications of reinforcement learning to building energy management. As a result of this literature review, it is clear that:

Reinforcement learning algorithms significantly improve the energy efficiency of homes. Energy savings vary significantly depending on the specific application. Broadly speaking RL can typically provide savings of $\approx 10\%$ for HVAC applications, $\approx 20\%$ for water heaters and $>20\%$ for more complete buildings energy management systems. 2. 2.

The vast majority of studies that implement RL for problems relating to building energy management are in simulation only. With accurate simulator design however, RL is a viable approach for building energy management outside of simulation. 3. 3.

Much of the recent research for RL and building energy management is focused on applications on deep RL algorithms due to their increased effectiveness over traditional approaches. This is likely to become more pronounced into the future.