Learning Dynamical Demand Response Model in Real-Time Pricing Program

TL;DR

This paper introduces a dynamical demand response model that captures temporal customer behavior in real-time pricing, significantly improving prediction accuracy over traditional static models.

Contribution

It proposes a novel dynamical DR modeling framework using linear, neural network, and RNN-based approaches, learned from historical data.

Findings

Dynamical models outperform static models in prediction accuracy.

Recurrent neural networks effectively capture temporal EUC behaviors.

The proposed models adapt to different state representations.

Abstract

Price responsiveness is a major feature of end use customers (EUCs) that participate in demand response (DR) programs, and has been conventionally modeled with static demand functions, which take the electricity price as the input and the aggregate energy consumption as the output. This, however, neglects the inherent temporal correlation of the EUC behaviors, and may result in large errors when predicting the actual responses of EUCs in real-time pricing (RTP) programs. In this paper, we propose a dynamical DR model so as to capture the temporal behavior of the EUCs. The states in the proposed dynamical DR model can be explicitly chosen, in which case the model can be represented by a linear function or a multi-layer feedforward neural network, or implicitly chosen, in which case the model can be represented by a recurrent neural network or a long short-term memory unit network. In both cases, the dynamical DR model can be learned from historical price and energy consumption data. Numerical simulation illustrated how the states are chosen and also showed the proposed dynamical DR model significantly outperforms the static ones.

Tables

Table 1. TABLE I: Performance of Dynamic DR Model Using Linear Function.

	order $n$	0	1	2	3	4	5
Train	MAPE (%)	19.67	6.19	4.46	3.99	3.95	3.82
	SDAPE (%)	15.22	4.97	3.89	3.61	3.56	3.46
Test	MAPE (%)	16.46	6.02	4.35	4.16	4.29	4.10
	SDAPE (%)	13.91	4.80	3.55	3.38	3.43	3.31

Table 2. TABLE II: Performance of Dynamic DR Model Using FNN.

	order $n$	0	1	2	3	4	5
Train	MAPE (%)	17.54	4.73	2.98	2.95	2.92	2.87
	SDAPE (%)	12.45	3.64	2.55	2.46	2.45	2.38
Test	MAPE (%)	15.48	5.26	3.46	3.47	3.48	3.34
	SDAPE (%)	11.74	3.77	2.74	2.64	2.64	2.63

Table 3. TABLE III: Performance of Dynamic DR Model Using RNN/LSTM.

		RNN	LSTM
Train	MAPE (%)	3.02	2.83
	SDAPE (%)	2.68	2.56
Test	MAPE (%)	3.25	3.16
	SDAPE (%)	2.55	2.47

Equations

$$\underset{e_t^{k,c}}{\mathrm{maximize}}{\beta }_t^k(e_t^{k,d},e_t^{k,c})-p_t{e}_t^{k,c},$$

$$e_{t+1}^{k,d}={\alpha }_t^k(e_t^{k,d}-e_t^{k,c})+{\xi }_{t+1}^k,$$

$$e_t^{k,c}\in {\mathcal{E}}_t^{k,c},$$

$$p_t⟼e_t^c:=\sum _{k\in \mathcal{K}}{e}_t^{k,c}.$$

$$e_t^c=\psi (p_t).$$

$$e_t^c=\phi ({\mathbf{s}}_t,p_t),$$

$${\mathbf{s}}_t=(p_{t-n},e_{t-n}^c,\cdots ,p_{t-1},e_{t-1}^c,t\mathrm{mod}T),$$

$$e_t^c={\mathbf{w}}^{\top }[{\mathbf{s}}_t^{\top },p_t]+b,$$

$${\mathbf{h}}_t^{[l]}=\mathrm{relu}({\mathbf{W}}^{[l]}{\mathbf{x}}_t^{[l]}+{\mathbf{b}}^{[l]}),$$

$$y_t=\mathbf{W}{\mathbf{h}}^{[L]}+\mathbf{b},$$

$$\ell =\frac{1}{m}\sum _{l=1}^m(y_t-e_t^c{)}^2.$$

$${\mathbf{h}}_t^{[l]}$$

$${\mathbf{f}}_t$$

$${\mathbf{i}}_t$$

$${\mathbf{o}}_t$$

$${\tilde{\mathbf{C}}}_t$$

$${\mathbf{C}}_t$$

$${\mathbf{h}}_t$$

$${\beta }_t^k(e_t^{k,d},e_t^{k,c})={\rho }^k(e_t^{k,d}-e_t^{k,c}{)}^2.$$

Full text

I Introduction

Demand response (DR) has proven to be a successful paradigm that emerges in the context of smart grid [1, 2, 3]. The DR enables the participation of demand-side resources, including commercial buildings [4, 5], industrial loads [6], electric vehicles [7], in maintaining the supply-demand-balance in power systems through some economic means such as incentives or price signals [1]. A variety of DR programs have been developed so far on both the wholesale market level and the retail market level. On the wholesale level, qualified demand-side resources can offer in almost all markets including capacity market, energy market, as well as ancillary markets, and will be cleared by the independent system operator (ISO) in the same way as the conventional generating resources. DR programs on the retail level, however, have more variety than those on the wholesale level. Yet, the retail DR programs can, by-and-large, be categorized into two classes—incentive based and price based. The former includes, e.g, direct load control and some interruptible programs, while the latter includes the real-time pricing (RTP), time-of-use-pricing, critical peaking pricing.

In price-based DR programs, the load serving entity (LSE)—the operator of the retail market—needs to design effective pricing schemes such that certain objective such as minimizing load fluctuation [8], maximizing profit [9] or utility [10], can be achieved. Particularly in RTP, the LSE sends out a price signal for every time interval, and the end use customers (EUCs) then respond to the price signal by adjusting their energy consumptions. It is therefore critical for the LSE to know the price response characteristics of the EUCs in order to make informed decisions on the price signal sent to the EUCs. Conventionally, price response characteristics are modeled using a static function such as linear, exponential, logarithmic, and potential functions [11]. These static functions, however, neglects the inherent temporal correlation of the EUC behaviors, and as we will see in the later part of the paper, may result in large errors when predicting the actual responses of EUCs in RTP programs.

To explicitly capture the temporal behavior of the EUCs, we propose a dynamical DR model which has states that evolve over time. The states in the proposed dynamical DR model keep necessary information from previous time intervals, and allows more accurate prediction of the EUCs’ response. These states can be explicitly chosen, in which case the model can be represented by a linear function or a multi-layer feedforward neural network (FNN), or implicitly chosen, in which case the model can be represented by a recurrent neural network (RNN) or a long short-term memory (LSTM) unit network. The proposed dynamical DR model can be learned from historical price and energy consumption data. In particular, the states can be determined empirically from the historical data. We will also show through numerical simulation that the proposed dynamical DR model significantly outperforms the conventional static models.

The remaining part of this paper is organized as follows. Section II reviews decision process of EUCs and proposes a dynamical DR model that is based on available data. The neural network representations of the dynamical DR model are developed in Section III, and results from numerical simulation are presented in Section IV. Some concluding remarks are made in Section V.

II Demand Response Model

In this section, we first briefly review how a myopic EUC may determine its energy consumption at a given electricity price. In light of the deficiencies in static DR models, we propose a dynamical DR model in the end. Throughout this paper, we use a subscript $t$ to denote the time interval, and assume one day is decomposed into $T$ segments.

II-A Decision Process of End Use Customers

In a RTP program, the LSE sends out a energy price for time interval $t$, denoted by $p_{t}$, before the beginning of that time interval. Each EUC then responds to the price signal by optimizing its energy consumption in a way that maximizes its overall benefit or equivalently, minimizes its overall cost. While there exist many detailed models for EUCs’, a generic model that is agnostic to the underlying components is to model the energy demand and energy consumption separately, where the energy demand is the needed energy while the energy consumption is the actual consumed energy [12, 10].

Let $\mathcal{K}=\{1,\cdots,K\}$ denote the set of EUCs served by the LSE. Let $e_{t}^{k,d}$ and $e_{t}^{k,c}$ denote the energy demand and energy consumption of EUC $k\in\mathcal{K}$ at time interval $t$, respectively. A myopic EUC finds its optimal energy consumption by solving the following utility maximization problem [10]:

[TABLE]

where $\beta_{t}^{k}(\cdot)$ is the benefit function, which gives the benefit of the EUC at certain energy demand and energy consumption, $\alpha_{t}^{k}$ is the backlog rate that represents the percentage of unmet energy demand that is carried over to the next time interval, $\xi_{t+1}^{k}$ is a random variable that models the new demand, $\mathcal{E}_{t}^{k,c}$ is the feasible set of energy consumption.

From the perspective of the LSE, the collective behavior of all EUCs are of more interest since its total profit demands on the aggregate energy consumption. The DR model that is of interest to the LSE is essentially a mapping from the price to the aggregate energy consumption as follow:

[TABLE]

II-B Dynamical Demand Response Model

The mapping from price to energy consumption is conventionally modeled using a static function such as linear, exponential, logarithmic, and potential functions [11], which can be conceptually expressed as:

[TABLE]

These static functions generally work well on wholesale level since the price response characteristics of DR resources in the wholesale market are mainly determined by their offers. Once the energy demand are cleared for each time interval, the DR resources will be controlled such that their energy consumptions follow the schedule, since otherwise they will incur costs. Therefore, given a certain price, the energy consumption can be determined from the relative static offer stacks directly, i.e., the mapping is relatively static.

However, in a RTP problem, the EUCs do not have the responsibility to follow any energy consumption profile. Indeed, it is obvious from (1) that given a price, the energy consumption of an EUC will depend on its current energy demand, which itself depends on the previous energy consumption. Therefore, the mapping in (2) is dynamical.

In light of this observation, we propose a dynamical model for the DR model for the EUCs in a RTP program as follows:

[TABLE]

where $\bm{s}_{t}$ is a state vector that captures all the factors besides the price that impact the aggregate energy consumption, and evolves over time given $p_{t}$.

While it is difficult to identify the “perfect” state vector, we may still be able to construct one that is good enough in the sense it gives good prediction accuracy of the aggregate demand. From a practical view, the state vector can be compromised of elements from the set of information that is available to the LSE at time interval $t$, denoted by $\mathcal{I}_{t}=\{(p_{\tau},e_{\tau}^{c})~{}\forall\tau<t,p_{t},t\}$. There are two potential approaches to construct such a state vector, a direct approach, and an indirect approach. In the direct approach, we select the state vector to be

[TABLE]

where $n$—referred to as the order of the DR model—is a parameter that can be determined from the historical data. We adopt the convention that $\bm{s}_{t}=\{t\bmod T\}$ when $n=0$. In the indirect approach, the state vector in (5) is further transformed in a series of nonlinear operations. With the state vector and the price information, we can predict the aggregate energy consumption using supervised learning techniques.

III Neural Network Representations of Dynamical Demand Response Model

In this section, we develop neural network representations for the dynamical DR model. A multi-layer FNN is applied in the direct approach, and a multi-layer RNN is applied in the indirect approach.

III-A Direct Approach Using Feedforward Neural Network

When using the direct approach, the state vector is manually selected from the set of available information. Therefore, the input vector and the output of the DR model are both readily available. We can thus fit the dynamical DR model in (4) using, for example, a linear function as follows:

[TABLE]

where $\bm{w}$ is a weight vector, and $b$ is the bias, the superscript $\top$ denotes the transpose of a vector or matrix.

Also, we may represent $\varphi$ with a nonlinear function such as a multi-layer FNN, which consists of one input layer, $L$ hidden layer, and one output layer as illustrated in the left part in Fig. 1. The hidden layer $l$ takes an input vector $\bm{x}_{t}^{[l]}$, and computes a (hidden) output vector $\bm{h}_{t}^{[l]}$ according to

[TABLE]

where $\mathrm{relu}(\cdot)$ denotes a rectified linear unit function that is applied element-wise, $\bm{W}^{[l]}$ is a weight matrix, and $\bm{b}^{[l]}$ is a bias vector. Note that the output vector of one hidden layer is the input vector for the next hidden layer, i.e, $\bm{x}^{[l+1]}=\bm{h}^{[l]}$, except the last hidden layer, the output of which is mapped to the output through a fully connected unit as follows:

[TABLE]

where $\bm{W}$ is a weight matrix, and $\bm{b}$ is a bias vector. Note that $\bm{x}_{t}^{[1]}=(\bm{s}_{t},p_{t})$. The multi-layer FNN can be trained using back-propagation such that the mean squared error between the predicted output $y_{t}$ and the true value $e_{t}^{c}$ is minimized, i.e., by minimizing the following loss function:

[TABLE]

III-B Indirect Approach Using Recurrent Neural Network

Alternatively, the states can be implicitly constructed within the neural network, which leads to RNNs [13]. The right part in Fig. 1 illustrate a multi-layer RNN with one input layer, $L$ hidden layers, and one output layer. The hidden state of the RNN unit in layer $l$, denoted by $\bm{h}_{t}^{[l]}$, is the input for RNN unit in the next layer as well as the input for itself at the next time step, as indicated by the arrows in Fig.1. This RNN takes a sequence $\{\bm{x}_{1},\cdots,\bm{x}_{T}\}$ as the input, and outputs a sequence $\{y_{1},\cdots,y_{T}\}$. Meanwhile, $L$ sequences of hidden states $\{\bm{h}_{t}^{[l]}\}$, $l=1,\cdots,L$, are generated along the trajectory, based on the following equations:

[TABLE]

where $\tanh(\cdot)$ is applied element-wise, $\bm{W}_{h}$ and $\bm{W}_{x}$ are weight matrices, $\bm{b}$ is a bias vector. Note that $\bm{h}_{-1}^{[l]}$ are initialized to zeros, $\bm{x}_{t}^{[l]}=\bm{h}_{t}^{[l-1]}$ for $l=2,\cdots,L$, and $\bm{x}_{t}^{[1]}=(\bm{s}_{t},p_{t})$. The hidden states in RNN are dynamical since their values also depend on their previous values, while those in the FNN are static since their values purely depend on the inputs. The output of the last hidden state vector is mapped to the output through a fully connected unit as in the case of multi-layer FNN. The RNN can be trained by minimizing the same loss function as in (9) using backpropagation through time technique (see, e.g., [14]). The input vector only has to include the most-recent information, i.e., when RNN is used, $n=1$ and $\bm{s}_{t}=\{p_{t-1},e_{t-1}^{c},t\bmod T\}$.

The key difference between the RNN and FNN in representing the dynamical DR model is that the FNN captures the temporal impacts by explicitly specifying a set of historical data as inputs, while the RNN keeps the temporal impacts by implicitly computing a dynamical hidden state.

One of the deficiency of the basic RNN unit is the lack of ability to model long-term dependencies. As a significant improvement over the basic RNN unit, the LSTM is proposed in [16]. The structure of an LSTM unit is illustrated in Fig. 2, in which $\sigma(\cdot)$ denotes a sigmoid function. For the purpose of simplicity, we drop the superscript that indicates the layer and focus on structure inside one LSTM unit. The LSTM unit introduces a new hidden state vector $\bm{C}_{t}$, which is used to keep long-term memories. The LSTM unit works as follows. First, a forget gate vector $\bm{f}_{t}$, an information gate vector $\bm{i}_{t}$, and an output gate vector $\bm{o}_{t}$ is computed from previous hidden state $\bm{h}_{t-1}$ and new input vector $\bm{x}_{t}$ as follows:

[TABLE]

Then, the two hidden state vectors are updated as follows:

[TABLE]

where $\circ$ represents element-wise multiplication. This structure has proven to be very effective in capturing long-term temporal dependencies, and therefore, is expected to outperform the basic RNN unit when representing the dynamical DR model. The multi-layer LSTM network is similar to the RNN in Fig. 1. We simply replace the RNN unit with the LSTM unit.

IV Simulation Results

In this section, we numerically investigate the performance of the proposed dynamical DR model under the direct and indirect approaches.

IV-A Simulation Setup

Assume $T=24$, i.e., each time interval covers one hour. We assume the benefit function has a quadratic form, the parameter in which is a constant, i.e.,

[TABLE]

The feasible set is $\mathcal{E}_{t}^{k,c}=\{e_{t}^{k,c}\geq 0.5e_{t}^{k,d}\}$. The backlog rate is uniformly sampled from $[0,1]$. The new demand is generated according the following procedures. First, the peak demand of each EUC is sampled uniformly from $[0.1,2]$ MW, then the new demand of the EUC is computed as the product of the peak demand, the normalized annual load profile from one of the zones in PJM in 2017 [17], and a Gaussian random variable with a mean of $1$ and a standard devision of $0.1$. The value of $\rho^{k}$ is taken as the ratio of $-100$ $/MWh2 and the EUC’s peak demand. It is chosen in this way such that the responses from EUCs are reasonable. In reality, the value of$\rho^{k}$is a choice of the EUCs. The number of EUCs served by the LSE is$100$. The historical data are simulated using the parameters described above, and a time series of prices that are sampled uniformly from$[20,50]$$/MWh, which is shown in Fig. 3. The first $7296$ sets of data (corresponding to January to October) are used for training, and the last $1464$ sets of data for testing (corresponding to November and December). The neural networks are implemented using Tensorflow [18], trained with the Adam optimizer with a learning rate of $0.001$ and training step of $10000$.

IV-B Performance Comparison

The mean absolute percentage error (MAPE), and the standard deviation of absolute error (SDAPE) are used to evaluate the performance of the learned model.

IV-B1 Linear Function

We first test the performance of dynamical DR model under direct approach. Table I shows the results when the linear function is used in the direct approach. Note that when order $n=0$, i.e., no information on previous time interval is utilized, both the training error and testing error are high, with MAPEs being $19.67\%$ and $16.46\%$, respectively. The errors drop significantly when $n>0$, which verifies the claim that the price responsive characteristics of the EUCs depend on their states in previous time intervals, rather than being static over time. Increasing the order significantly improves the model performance when $n<3$, however, When $n\geq 3$, the improvement becomes negligible. Therefore, when the linear function is used, an appropriate order of the dynamical DR model would be $3$. Intuitively, this means the response of the aggregate EUC demand to the price signal depends on the previous $3$ hours, rather than being static.

IV-B2 FNN

A FNN with $L=2$ hidden layers, each with $32$ neurons, is also used to represent the dynamical DR model. Table II shows the results when the FNN is used in the direct approach. It is clear that the FNN outperforms the linear function when the order of the model is the same, which indicates that the linear function is underfitting the price and energy consumption data. In addtion, when the FNN is used in the direct approach, the number of order of the dynamical DR model can be much lower than the case with the linear function. For example, the testing MAPE of the FNN is $3.46\%$ when $n=2$, much better than that of the linear function when $n=5$, which is $4.10\%$.

IV-B3 RNN/LSTM

In the indirect approach, a RNN/LSTM with $L=1$ hidden layer and $32$ neurons is adopted. The results are presented in Table II. In the indirect approach, only the most recent price and energy consumption is sent into the RNN/LSTM for prediction; The testing MAPEs of the indirect approach, which are $3.25\%$ with RNN and $3.16\%$ with LSTM, are better than that of the direct approach with FNN of order $5$, which is $3.34\%$, while the LSTM performs better than the RNN by a small margin.

The testing MAPE obtained using linear function with order $5$, FNN with order $5$, RNN, and LSTM are presented in the violin plot in Fig. 4. To sum up, both the direct and indirect approach can be applied to learn a good dynamical DR model. FNN, RNN, and LSTM outperforms the linear function, at the cost of higher model complexity. In particular, when RNN or LSTM is used, no manual selection of states are required.

IV-C Discussion

When the neural networks are used to represent the dynamical DR model, we also investigated the impacts of hyperparameter such as the order the dynamical DR model, the number of layers, and the number of neurons in each layer. Among all factors, the order of the dynamical DR model has the most significant impact on the performance, which has already been discussed in details in the above section. The impacts of other parameters are relatively small.

Also, we would like to emphasize that the learned dynamical DR model can find its application in a variety of scenarios. For example, it can be used to predict the energy consumption profile over several time intervals, given the prices over the same intervals. It can also be used in an agent that simulates the collectively behavior of a set of EUCs without necessarily modeling all the underlying components; such an agent can be very helpful in developing certain algorithms.

V Concluding Remarks

In this paper, we proposed a dynamical DR model that captures the temporal behavior of EUCs. A key element in the proposed model is the choice of state, which can be determined using either a direct approach that selects the state manually from the available information, or an indirect approach that computes the states from input information. The dynamical DR model can be represented by a linear function or FNN when the direct approach is used, and a RNN or LSTM network when the indirect approach is used. Both these two approaches can achieve small MAPE when predicting the response of the aggregate energy consumption to the price. Numerical simulation results validated that the dynamical DR models are indeed necessary to model the price response characteristics of the EUCs, which are inherently temporally correlated.

Future research will utilize the proposed dynamical DR model to learn a pricing policy of the LSE purely from historical data using agent-based algorithms.