Extensions of the Dynamic Programming Framework: Battery Scheduling, Demand Charges, and Renewable Integration

TL;DR

This paper extends the dynamic programming framework to handle non-separable objectives, introducing a tractable augmentation scheme and applying it to optimize battery scheduling with demand charges using stochastic models.

Contribution

It develops a state-augmentation approach for non-separable DP problems, especially those with Naturally Forward Separable objectives, and applies it to renewable energy management.

Findings

Proposed a tractable augmented-state DP method for NFS objectives.

Extended the framework to stochastic DP problems with a new Principle of Optimality.

Applied the method to optimize battery scheduling with demand charges.

Abstract

We consider a general class of Dynamic Programming (DP) problems with non-separable objective functions. We show that for any problem in this class, there exists an augmented-state DP problem which satisfies the Principle of Optimality and the solutions to which yield solutions to the original problem. Furthermore, we identify a subclass of DP problems with Naturally Forward Separable (NFS) objective functions for which this state-augmentation scheme is tractable. We extend this framework to stochastic DP problems, proposing a suitable definition of the Principle of Optimality. We then apply the resulting algorithms to the problem of optimal battery scheduling with demand charges using a data-based stochastic model for electricity usage and solar generation by the consumer.

Tables

Table 1. Table I: This table shows the corresponding cost of each feasible policy used in the counter example in lemma 1

feasible $𝐮$	objective value	feasible $𝐮$	objective value
$(0,0,0)$	0	$(h,0,-h)$	h/2
$(0,0,h)$	h/2	$(h,0,0)$	0
$(0,h,0)$	2h	$(h,-h,0)$	-h
$(0,h,-h)$	(5/2)h	$(h,-h,h)$	-(3/2)h

Table 2. Table II: List of constant values (prices correspond to Salt River Project E21 price plan)

Constant	Value	Constant	Value
$\alpha$	0.999791667 (W/h)	$t_{\text{off}}$	41
$\eta$	0.92 (%)	$p_{\text{on}}$	$0.0633\times {10}^{-3}$ ($/KWh)
$\overline{u}$	4000 (Wh)	$p_{\text{off}}$	$0.0423\times {10}^{-3}$ ($/KWh)
$\underset{¯}{u}$	-4000 (Wh)	$p_{\text{d}}$	0.2973 ($/KWh)
$\overline{e}$	8000 (Wh)	$\Delta t$	0.5 (h)
$t_{\text{on}}$	27

Equations

$$({\mathbf{u}}^{\ast },{\mathbf{x}}^{\ast })\in \arg \underset{\mathbf{u},\mathbf{x}}{\min }{J}_{t_0}(\mathbf{u},\mathbf{x})$$

$$x(t+1)=f(x(t),u(t),t)\text{ for }t=t_0,..,T,$$

$$x(t_0)=x_0,x(t)\in X_t\subset {\mathbb{R}}^n\text{ for }t=t_0,..,T-1,$$

$$u(t)\in U\subset {\mathbb{R}}^m\text{ for }t=t_0,..,T-1,$$

$$\mathbf{u}=(u(t_0),...,u(T-1))\text{ and }\mathbf{x}=(x(t_0),...,x(T)),$$

$$J_{t_0}(\mathbf{u},\mathbf{x})=\sum _{t=t_0}^{T-1}{c}_t(x(t),u(t))+c_T(x(T)),$$

$${\mathbf{u}}^{\ast }=({\pi }^{\ast }(x_0,t_0),....,{\pi }^{\ast }(x(T-1),T-1))$$

$$\underset{\mathbf{u},\mathbf{x}}{\min }{J}_{t_0}(\mathbf{u},\mathbf{x})=\sum _{t=t_0}^{T-1}{c}_t(x(t),u(t))+c_T(x(T))$$

$$x(t+1)=f(x(t),u(t),t)\text{ for }t=t_0,..,T-1,$$

$$x(t_0)=x_0,x(t)\in X_t\subset {\mathbb{R}}^n\text{ for }t=t_0,..,T,$$

$$u(t)\in U\subset {\mathbb{R}}^m\text{ for }t=t_0,..,T-1,$$

$$\mathbf{u}=(u(t_0),...,u(T-1))\text{ and }\mathbf{x}=(x(t_0),...,x(T)).$$

$$J_s(\mathbf{w},\mathbf{z})$$

$$<\sum _{t=s}^{T-1}{c}_t(x(t),u(t))+c_T(x(T))=J_s(\mathbf{v},\mathbf{h})$$

$$J_t(\hat{\mathbf{u}},\hat{\mathbf{x}})$$

$$=\sum _{k=t}^{s-1}{c}_k(x(k),u(k))+\sum _{k=s}^{T-1}{c}_k(z(k),w(k))+c_T(z(T))$$

$$<\sum _{k=t}^{s-1}{c}_k(x(k),u(k))+\sum _{k=s}^{T-1}{c}_k(x(k),u(k))+c_T(x(T))$$

$$=J_t({\mathbf{u}}^{\ast },{\mathbf{x}}^{\ast })$$

$$F(x,t)=\underset{u\in {\Gamma }_{t,x}}{\inf }\{c_t(x,u)+F(f(x,u,t),t+1)\}$$

$$\forall x\in X_t\text{ and }\forall t\in \{t_0,..,T-1\},$$

$$F(x,T)=c_T(x)\forall x\in X_T$$

$$\theta (x,t)\in \arg \underset{u\in {\Gamma }_{t,x}}{\min }\{c_t(x,u)+F(f(x,u,t),t+1)\},$$

$$\underset{\mathbf{u},\mathbf{x}}{\min }{J}_{t_0}(\mathbf{u},\mathbf{x}):=\sum _{t=t_0}^{T-1}{c}_t(x(t),u(t))+c_T(x(T))+\underset{t_0\le k\le T}{\max }{d}_k(x(k))$$

$$x(t+1)=f(x(t),u(t),t)\text{ for }t=t_0,..,T-1,$$

$$x(t_0)=x_0,x(t)\in X_t\subset {\mathbb{R}}^n\text{ for }t=t_0,..,T,$$

$$u(t)\in U\subset {\mathbb{R}}^m\text{ for }t=t_0,..,T-1,$$

$$\mathbf{u}=(u(t_0),...,u(T-1))\text{ and }\mathbf{x}=(x(t_0),...,x(T)),$$

$$\underset{\mathbf{u}\in {\mathbb{R}}^3,\mathbf{x}\in {\mathbb{R}}^4}{\min }\sum _{t=0}^2{c}_t(u(t))+\underset{0\le k\le 3}{\max }x(k)$$

$$\text{subject to: }x(t+1)=x(t)+u(t)\forall t\in \{0,1,2\},$$

$$x(0)=0,0\le x(t)\le h,u(t)\in \{-h,0,h\},$$

$$\underset{u(2)\in \mathbb{R},0\le x(3)\le h}{\min }{c}_2(u(2))+\underset{2\le k\le 3}{\max }x(k)$$

$$\text{subject to: }x(t+1)=x(t)+u(t),$$

$$x(2)=0,0\le x(t)\le h,u(t)\in \{-h,0,h\}.$$

$$J(\mathbf{u},\mathbf{x})={\varphi }_T(x(T),{\varphi }_{T-1}(x(T-1),u(T-1),{\varphi }_{T-2}(....,$$

$${\varphi }_{t_0+1}(x(t_0+1),u(t_0+1),{\varphi }_{t_0}(x(t_0),u(t_0)))),....,))),$$

$${\varphi }_{t_0}(x,u)=c_{t_0}(x,u)$$

$${\varphi }_i(x,u,w)=c_i(x,u)+w\text{for }i=t_0+1,\cdots ,T-1$$

$${\varphi }_T(x,w)=c_T(x)+w.$$

$$\underset{\mathbf{u},\mathbf{x}}{\min }{J}_{t_0}(\mathbf{u},\mathbf{x})$$

$$x(t+1)=f(x(t),u(t),t)\text{ for }t=t_0,..,T-1,$$

$$x(t_0)=x_0,x(t)\in X_t\subset {\mathbb{R}}^n\text{ for }t=t_0,..,T,$$

$$u(t)\in U\subset {\mathbb{R}}^m\text{ for }t=t_0,..,T-1,$$

$$\mathbf{u}=(u(t_0),...,u(T-1))\text{ and }\mathbf{x}=(x(t_0),...,x(T)),$$

$$\underset{\mathbf{u},\mathbf{z}}{\min }{L}_{t_0}(\mathbf{u},\mathbf{z}):={\varphi }_T(z_1(T),z_2(T))$$

$$\left[\begin{array}{c}{z}_1(t+1)\\ z_2(t+1)\end{array}\right]=\left[\begin{array}{c}f(z_1(t),u(t),t)\\ {\varphi }_t(z_1(t),u(t),z_2(t))\end{array}\right]t_0+1\le t<T-1$$

Full text

I Introduction

The optimal use of battery storage can be formulated as a discrete time process combined with decision variables and an objective function - a formulation commonly known as Dynamic Programming (DP) [1]. DP is a class of algorithms that break down complex optimization problems into simpler sequential subproblems, each of which is solved using Bellman’s Equation. For DP to work, however, we require that the optimization problem satisfies the Principle of Optimality [2] (a.k.a time-consistency [3, 4]); from any point on an optimal trajectory, the remaining portion of the optimal trajectory is also optimal for the problem initiated at that point [5]. DP problems commonly have an additively separable objective function of the form $J(\mathbf{u},\mathbf{x})=\sum_{t=0}^{T-1}c_{t}(x(t),u(t))+c_{T}(x(T))$. Problems of this form can be shown to satisfy the Principle of Optimality. However in problems such as optimal battery scheduling, we find non-additively separable objective functions. For example, if the objective is of the form $J(\mathbf{u},\mathbf{x})=\sum_{t=0}^{T-1}c_{t}(x(t),u(t))+\max_{t_{0}\leq k\leq T}{d_{{k}}(x({k}))}$ then the problem does not satisfy the Principle of Optimality. In this paper we propose a method for solving DP problems with non-separable objective functions by constructing equivalent DP problems with additively separable objective functions. Such reformulated problems then satisfy the Principle of Optimality and can therefore be solved using Bellman’s Equation. Moreover, we identify a class of problems, defined by Naturally Forward Separable (NFS) objectived, wherein state augmentation method does not substantially increase the complexity of the problem.

For stochastic DP we generalize our framework and propose a suitable definition of the Principle of Optimality. As discussed in [6] such a definition is non trivial. Inspired by [7], we construct probability measures on the sets the state variable can take at each time stage induced by the underlying random variables. We then say a stochastic problem satisfies the Principle of Optimality if from any point on a trajectory followed using an optimal policy, $\pi$, the policy $\pi$ is also optimal for the problem initiated from that point with probability one.

Dynamic programming for problems which do not satisfy the Principle of Optimality has received relatively little attention. The only general approach to the problem seems to be that taken in [8] which considered the use of multi-objective optimization in the case where the objective function is “backward separable”. Our approach differs from [8] as we consider the class of “forward separable” objective functions. In this paper we show that almost any objective function is forward separable in a certain sense and that for such problems there exists an additively separable augmented-state dynamic programming problem that satisfies the Principle of Optimality and from which solutions to the original forward separable problem can be recovered - See Section III. However, the resulting augmented-state DP problem has a higher dimensional state space than the original DP problem - an issue that can potentially render the augmented problem intractable due to the “curse of dimensionality”. For this reason, we propose a complexity metric for the forward separable representation and show that in certain cases the dimensionality of the augmented system does not significantly exceed the dimensionality of the original problem - a case where Bellman’s equation can be used effectively [9] and which we refer to as Naturally Forward Separable (NFS).

Note that although state augmentation has been used in the context of DP [10, 11], the only tractable use of state augmentation to recover the Principle of Optimality appears to be [12] and [13], who considered a DP problem with objective function of the form $J(\mathbf{u},\mathbf{x})=U(\sum_{t=0}^{T-1}c_{t}(x(t),u(t)))$; and [14], who considered a DP problem with variance-type objective function. Both these results can be considered special cases of the NFS class of objective functions proposed in this paper.

In practice it is rare to be able to analytically solve Bellman’s Equation. Therefore, once the augmented-state DP problem is formulated we propose a map to an approximated DP problem that can be analytically solved. Using an optimal solution from the approximated DP problem a feasible policy for the original problem is then constructed using a discretization scheme based on [15] and [16]. We show that as the number of discrete points increases, the resulting policies converge to the optimal policy.

To illustrate the proposed methods, we consider battery scheduling for mitigating the effect of variability in renewable energy resources. Specifically, renewable energy sources are most accurately modeled as an uncontrollable Gauss-Markov (G-M) process and the battery (for both consumers and utilities) attempts to minimize energy costs based on time-of-use while also minimizing the maximum rate of energy consumption. Based on this model, we formulate the battery storage problem as a DP with a non-separable objective function consisting of both integrated time-of-use charges and a maximum term representing the demand charge. Furthermore, we propose a model of solar generation as a G-M process and minimize the expected value of the proposed objective. The fundamental mathematical challenge with dynamic programming problems of this form is that, as shown in Section II, problems which include maximum terms in the objective do not satisfy the Principle of Optimality and thus the recursive solution of the Bellman equation ([1]) does not yield an optimal policy. To overcome this difficulty, we show that the battery scheduling problem is a special case of a forward separable DP problem with an NFS objective function. We then apply our state-augmentation technique to numerically solve the deterministic battery scheduling problem for given forecast solar data. In section IX-E we apply our approach to the battery scheduling problem using a Gauss-Markov model of solar generation extracted from data provided by local utility SRP. Note that this result extends previous work which considered a more limited non-separable DP framework as applied to battery scheduling in [17].

Remarkably, almost no work has been done on optimal use of batteries for reduction of demand charges. The exceptions include the heuristic algorithms of [18] and the pioneering work of [19] , which considered only a demand charge. Recently this group used an ad-hoc algorithm to consider a combined demand/consumption charge in [20] using detailed models of cooling/load. Furthermore, in [21] a similar energy storage problem is solved using optimized curtailment and load shedding. An $L_{p}$ approximation of the demand charge was used in combination with multi-objective optimization in [22] and, in addition, the optimal use of building mass for energy storage was considered in [23], wherein a bisection on the demand charges was used. We note that none of these approaches resolve the fundamental mathematical problem of DP with a non-separable cost function.

The paper is organized as follows. In Section II we introduce a formal definition of the DP problem along with associated notation and use this framework to define a Principle of Optimality. Next, we consider a class of objective functions we refer to as forward separable. In Section III, we show that for any DP problem with forward separable objective, there exists an augmented-state DP problem with separable objective for which the Principle of Optimality holds and from which solutions to the original DP with FS objective can be recovered. In Section IV we define a class of objective functions, termed Naturally Forward Separable (NFS). We show DP problems with naturally forward separable objective functions can be tractable solved using state augmentation. In Section V we show how to approximate and numerically solve augmented-state dynamic programming problems. Furthermore, we extend our framework to stochastic DP problems in Section VI and give a discretization scheme to solve stochastic DP’s with additively separable objective functions in Section VII. We summarize how state augmentation can be used with discretization methods to solve DP problems with NFS objective functions in Section VIII. In Section IX we formulate and solve the battery scheduling problem as a DP with NFS objective function.

II Background: Dynamic Programming

In this paper, we propose a framework for representing a general class of Dynamic Programming (DP) problems. Specifically, we define a general DP problem as a sequence of optimization problems $\mathcal{G}(t_{0},x_{0})$, indexed by $t_{0}\in\mathbb{N}$, and defined by an indexed sequence of objective functions $J_{t_{0}}:\mathbb{R}^{m\times(T-t_{0})}\times\mathbb{R}^{n\times(T-t_{0}+1)}{\to\mathbb{R}}$ where we say that the state and input sequence, $\mathbf{u}^{*}\in\mathbb{R}^{m\times(T-t_{0})}$ and $\mathbf{x}^{*}\in\mathbb{R}^{n\times(T-t_{0}+1)}$, solve $\mathcal{G}(t_{0},x_{0})$ if,

[TABLE]

where $f$ : $\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{N}\to\mathbb{R}^{n}$, $U\subset\mathbb{R}^{m}$ and $X_{t}\subset\mathbb{R}^{n}$ for all $t$. We denote $J_{t_{0}}^{*}=J_{t_{0}}(\mathbf{u}^{*},\mathbf{x}^{*})$.

We call $\{x(t)\}_{t_{0}\leq t\leq T}$ the set of state variables and $n$ the state space dimension. Similarly we will call $\{u(t)\}_{t_{0}\leq t\leq T-1}$ the input (control) variables and $m=\dim(U)$ the input (control) space dimension. For cases where the dimension of the state variable, $x(t)$, varies with time, we slightly abuse notation and define the state space dimension as $\max_{t_{0}\leq t\leq T}{\dim(X_{t})}$.

Definition 1**.**

The function $J_{t_{0}}:\mathbb{R}^{m\times(T-t_{0})}\times\mathbb{R}^{n\times(T-t_{0}+1)}{\to\mathbb{R}}$ is said to be additively separable if there exist functions, $c_{T}(x):\mathbb{R}^{n}\to\mathbb{R}$, and $c_{t}(x,u):\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}$ for $t=t_{0},\cdots T-1$ such that,

[TABLE]

where $\mathbf{u}=(u(t_{0}),...,u(T-1))\text{ and }\mathbf{x}=(x(t_{0}),...,x(T))$.

To illustrate, we note that the average value of a function $a_{t}:\mathbb{R}^{n}\to\mathbb{R}$, defined as $J(\mathbf{u},\mathbf{x})=\frac{1}{T}\sum_{t=0}^{T}a_{t}(x(t))$, is clearly an additively separable function. Variance type functions (11), however, are not additively separable.

Definition 2**.**

We say the sequence of inputs $\mathbf{u}=(u({t_{0}}),...,u({T-1}))\in\mathbb{R}^{m\times(T-t_{0})}$ is feasible if $u(t)\in U\text{ for }t=t_{0},..,T-1$ and for $x(t+1)=f(x(t),u(t),t)$ and $x(t_{0})=x_{0}$, then $x(t)\in X$ for all $t$. For a given $x$, we denote by $\Gamma_{t,x}$, the set $u\in U$ such that $f(x,u,t)\in X_{t+1}$. In this paper we only consider problems where $\Gamma_{t,x}$ is nonempty for all $x$ and $t$.

Note that for this class of DP problems, feasibility is inherited. That is, if $\mathbf{u}=(u({t}),....,u({T-1}))$ is feasible with $\mathbf{x}=(x(t),\cdots,x(T))$ for $\mathcal{G}(t,x(t))$ and $\mathbf{v}=(v({s}),....,v({T-1}))$ is feasible with $\mathbf{h}=(h(s),\cdots,h(T))$ for $\mathcal{G}(s,x(s))$ where $s>t$, then $\mathbf{w}=(u(t),\cdots,u(s-1),v({s}),....,v({T-1}))$ with $\mathbf{z}=(x(t),\cdots,x(s-1),h(s),\cdots,h(T))$ is feasible for $\mathcal{G}(t,x(t))$.

Definition 3**.**

A (Markov) policy is any map from the present state and time to a feasible input $(x,t)\mapsto u(t)\in\Gamma_{x,t}$, as $u(t)=\pi(x,t)$. We denote the set of policies consistent with some DP problem as $\Pi$. We say that $\pi^{*}$ is an optimal policy for Problem (1) if

[TABLE]

where $x({t+1})^{*}=f(x(t)^{*},\pi^{*}(x(t)^{*},t),t)$ for all $t$.

We now define a “Principle of Optimality” consistent with our DP formulation and which provides a necessary condition for such DP problems to be solvable using Bellman’s equation (4). If a DP problem is solvable using Bellman’s equation, then this equation yields an optimal policy.

Definition 4**.**

We say a DP problem, $\mathcal{G}(t_{0},x_{0})$, of the Form (1) satisfies the Principle of Optimality if the following holds. For any $s$ and $t$ with $t_{0}\leq t<s<T$, if $\mathbf{u}^{*}=(u(t),...,u(T-1))$ and $\mathbf{x}^{*}=(x(t),...,x(T))$ solve $\mathcal{G}(t,x(t))$ then $\mathbf{v}=(u(s),...,u(T-1))$ and $\mathbf{h}=(x(s),...,x(T))$ solve $\mathcal{G}(s,x(s))$.

The standard form of DP, equivalent to that defined in [1], solves indexed DP problems of the Form (1) with an additively separable objective function. We denote this class of DP problems by $\mathcal{P}(t_{0},x_{0})$:

[TABLE]

Note that $J_{T}(x)=c_{T}({x})$. We will refer to $x_{0}\in\mathbb{R}^{n}$ as the initial state, $J_{t_{0}}$ is the objective function, $c_{t}$ : $\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}$ for $t=t_{0},..,T-1$, $c_{T}$ $\mathbb{R}^{n}\to\mathbb{R}$ are given functions and $f$ : $\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{N}\to\mathbb{R}^{n}$ is a given vector field. The following lemma shows that this class of problems satisfies the proposed Principle of Optimality.

Lemma 5**.**

Any problem of form $\mathcal{P}(t_{0},x_{0})$ in (3) satisfies the Principle of Optimality.

Proof.

Suppose $\mathbf{u}^{*}=(u(t),...,u(T-1))$ and $\mathbf{x}^{*}=(x(t),...,x(T))$ solve $\mathcal{P}(t,x(t))$ in (2). Now we suppose by contradiction that there exists some $s>t$ such that $\mathbf{v}=(u(s),...,u(T-1))$ and $\mathbf{h}=(x(s),...,x(T))$ do not solve $\mathcal{P}(s,x(s))$. We will show that this implies that $\mathbf{u}^{*}$ and $\mathbf{x}^{*}$ do not solve $\mathcal{P}(t,x)$ in (2), thus verifying the conditions of the Principle of Optimality. If $\mathbf{v}$ and $\mathbf{h}$ do not solve $\mathcal{P}(s,x(s))$, then there exist feasible $\mathbf{w}$, $\mathbf{z}$ such that $J_{s}(\mathbf{w},\mathbf{z})<J_{s}(\mathbf{v},\mathbf{h})$. i.e.

[TABLE]

Now consider the proposed feasible sequences $\hat{\mathbf{u}}=(u(t),...,u(s-1),w(s),...,w(T-1))$ and $\hat{\mathbf{x}}=(x(t),...,x(s-1),z(s),...,z(T-1))$. It follows:

[TABLE]

which contradicts optimality of $\mathbf{u}^{*},\mathbf{x}^{*}$. Therefore, this class of problems satisfies the Principle of Optimality. ∎

Proposition 6** ([24]).**

For DP problems of the form $\mathcal{P}(t,x)$ in (3) with optimal objective, $J_{t}^{*}=J_{t}(\mathbf{u}^{*},\mathbf{x}^{*})$, define the function $F(x,t)=J_{t}^{*}$. Then the following holds.

[TABLE]

Equation (4) is often referred to as Bellman’s equation and a function $F$ which satisfies Bellman’s equation is often referred to as the “optimal cost-to-go” function. Prop. 6 shows that problems of the Form $\mathcal{P}(t_{0},x_{0})$ define a solution to Bellman’s equation which in turn indexes the optimal objective to the problem. Furthermore, for problems $\mathcal{P}(t_{0},x_{0})$, the solution to Bellman’s equation can be obtained recursively backwards in time using a minimization on $u$. A solution to Bellman’s equation provides a state-feedback law or optimal policy as follows.

Corollary 7** ([24]).**

Consider $\mathcal{P}(t_{0},x_{0})$ in (3). Suppose $F(x,t)$ satisfies Equation (4) for $\mathcal{P}(t_{0},x_{0})$. Then if there exists a policy such that,

[TABLE]

then $\theta$ is an optimal policy for the problem $\mathcal{P}(t_{0},x_{0})$.

Dynamic Programming with Maximum Terms In this paper we consider the special class of indexed DP problems, denoted by $\mathcal{S}(t_{0},x_{0})$ and given in (5). In contrast to problems of the form $\mathcal{P}(t_{0},x_{0})$ in (1), class $\mathcal{S}(t_{0},x_{0})$ has maximum terms in the objective. Specifically, these problems have the following form:

[TABLE]

where $c_{T}(x):\mathbb{R}^{n}\to\mathbb{R}$; $c_{t}(x,u):\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}$ for $t_{0}\leq t\leq T-1$; $d_{t}(x):\mathbb{R}^{n}\to\mathbb{R}$ for $t=t_{0},\cdots T$; $f$ : $\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{N}\to\mathbb{R}^{n}$.

Counterexample 8**.**

The class of DP problems of the form $\mathcal{S}(t_{0},x_{0})$ in (5) does not satisfy the Principle of Optimality.

Proof.

We give a counterexample. For $h>0$, we consider the following problem $\mathcal{S}(0,0)$:

[TABLE]

where here we define $c_{0}(u(0))=-u(0),$ $c_{1}(u(1))=u(1),$ $c_{2}(u(2))=-u(2)/2$.

Since $\mathbf{u}\in\{-h,0,h\}^{3}$, there are 27 input sequences, only 8 of which are feasible. In Table I, we calculate the objective value of each feasible input sequence and deduce the optimal input is $\mathbf{u}^{*}=(h,-h,h)$, yielding an optimal trajectory of $\mathbf{x}^{*}=\{0,h,0,h\}$. Following this input sequence until $t=2$ we examine the problem $\mathcal{S}(2,0)$.

[TABLE]

For this sub-problem, there are two feasible inputs: $u(2)\in\{0,h\}$. Of these, the first is optimal (objective value [math] vs $h/2$). Thus, although $\mathbf{u}^{*}=\{h,-h,h\}$ and $\mathbf{x}^{*}=\{0,h,0,h\}$ solve $\mathcal{S}(0,0),\mathbf{v}=\{h\}$ and $\mathbf{h}=\{0,h\}$ do not solve $\mathcal{S}(2,0)$. ∎

III Converting Forward Separable DP To Additively Separable DP

In this section we define the class of forward separable objective functions. We will show that for dynamic programming problems with a forward separable objective function, augmenting the state variables allows us to use Bellman’s equation to obtain an optimal policy.

Forward separable functions were first defined in [25]. Intuitively, this is the class of functions that can be separated into a nested composition of maps ordered forward in time. In the next definition we build upon the concept of forward separability by introducing the notion of augmented dimension.

Definition 9**.**

The function $J:\mathbb{R}^{m\times(T-t_{0})}\times\mathbb{R}^{n\times(T+1-t_{0})}\to\mathbb{R}$ is said to be forward separable if there exist representation maps $\phi_{t_{0}}:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}^{d_{t_{0}}}$, $\phi_{T}:\mathbb{R}^{n}\times\mathbb{R}^{d_{T-1}}\to\mathbb{R}$, and $\phi_{i}:\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{R}^{d_{i-1}}\to\mathbb{R}^{d_{i}}$ for $i=t_{0}+1,\cdots T-1$ such that

[TABLE]

where $\mathbf{u}=(u(t_{0}),...,u(T-1))\in\mathbb{R}^{m\times(T-t_{0})}$ and $u(i)\in\mathbb{R}^{m}$ for $i\in\{t_{0},...,T-1\}$; $\mathbf{x}=(x(t_{0}),...,x(T))\in\mathbb{R}^{n\times(T+1-t_{0})}$ and $x(i)\in\mathbb{R}^{n}$ for $i\in\{t_{0},...,T\}$; $d_{i}\in\mathbb{N}$ for $i\in\{t_{0},...,T-1\}$.

Moreover we say $J(\mathbf{u},\mathbf{x})$ is forward separable and has a representation dimension of $l$ if there exists $\{\phi_{i}\}$ that satisfies (6) and $l=\max_{i\in\{t_{0},...,T-1\}}\{d_{i}\}$ where $d_{i}=\dim(Im\{\phi_{i}\})$.

Note: The representation dimension of a forward separable function is a property of the set $\{\phi_{i}\}$ chosen and not the function. The representation dimension of a forward separable function is not unique. Moreover, the forward separable property of an objective function is independent of the DP problem it is associated with; forward separability is solely a property of the function only.

Clearly, any additively separable objective function of the form $J(\mathbf{u},\mathbf{x})=\sum_{t=t_{0}}^{T-1}c_{t}(u(t),x(t))+c_{T}(x(T))$ is forward separable and has a representation dimension of 1 using,

[TABLE]

III-A How State Augmentation Solves Forward Separable DP Problems

We now define the class of forward separable problems $\mathcal{H}(t_{0},x_{0})$. Such problems are a special case of $\mathcal{G}(t_{0},x_{0})$ (1), but not of $\mathcal{P}(t_{0},x_{0})$ (3). Specifically, $\mathcal{H}(t_{0},x_{0})$ has the form:

[TABLE]

where $J_{t_{0}}$ is forward separable with associated representation maps $\phi_{i}$. For any forward separable DP problem $\mathcal{H}(t_{0},x_{0})$, we may associate a new augmented-state DP problem of form $\mathcal{A}(t_{0},x_{0})$, which is equivalent to $\mathcal{H}(t_{0},x_{0})$ and which satisfies the Principle of Optimality. $\mathcal{A}(t_{0},x_{0})$ is defined as

[TABLE]

where $f$ : $\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{N}\to\mathbb{R}^{n}$, $z_{1}(t)\in\mathbb{R}^{n}$, $z_{2}(t)\in\mathbb{R}^{d_{t}}$, $d_{t}=\dim(Im\{\phi_{t-1}\})$ and $u(t)\in\mathbb{R}^{m}$ for all $t$.

Lemma 10**.**

Suppose $J_{t_{0}}$ is the objective function for the DP problem $\mathcal{H}(t_{0},x_{0})$ (8) and is forward separable with associated representation maps $\phi_{i}$. Consider the augmented DP problem $\mathcal{A}(t_{0},x_{0})$ (9) and denote its objective function by $L_{t_{0}}$. Then $J^{*}_{t_{0}}=L^{*}_{t_{0}}$. Furthermore, suppose $\mathbf{u}$ and $\mathbf{x}=(x(t_{0}),...,x(T))$ solve $\mathcal{H}(t_{0},x_{0})$ and $\mathbf{w}$ and $\mathbf{z}=\left(\begin{bmatrix}z_{1}(t_{0})\\ z_{2}(t_{0})\end{bmatrix},...,\begin{bmatrix}z_{1}(T)\\ z_{2}(T)\end{bmatrix}\right)$ solve $\mathcal{A}(t_{0},x_{0})$. Then $\mathbf{u}=\mathbf{w}$ and $x(t)=z_{1}(t)$ for all $t$.

Proof.

Suppose $\mathbf{w}$ and $\mathbf{z}$ solve $\mathcal{A}(t_{0},x_{0})$. First we show that $\mathbf{w}$ and $\mathbf{z}_{1}:=(z_{1}(t_{0}),...,z_{1}(T))$ are feasible for $\mathcal{H}(t_{0},x_{0})$. Clearly $w(t)\in U$ for all $t$ and if we let $\mathbf{u}=\mathbf{w}$ then $x(0)=x_{0}$ and $x(t+1)=f(x(t),u(t),t)$ for all $t$. Likewise since $z_{1}(t_{0})=x_{0}$ and $z_{1}(t+1)=f(z_{1}(t),u(t),t)$ for all $t$, we have $x(t)=z_{1}(t)\in X_{t}$ for all $t$. Hence $\mathbf{u}$ and $\mathbf{x}=\mathbf{z}_{1}$ are feasible for $\mathcal{H}(t_{0},x_{0})$. On the other hand, if $\mathbf{u}$ and $\mathbf{x}$ solve $\mathcal{H}(t_{0},x_{0})$, then if we let $\mathbf{w}=\mathbf{u}$ and $\mathbf{z}_{1}=\mathbf{x}$ and define $z_{2}(t+1)=\phi_{t}(z_{1}(t),u(t),z_{2}(t))$, $z_{2}(t_{0}+1)=\phi_{0}(z_{1}(t_{0}),u(t_{0}))$, $z_{2}(t_{0})=0$, then $\mathbf{w}$ and $\mathbf{z}$ are feasible. Furthermore, since $\mathcal{H}(t_{0},x_{0})$ has a forward separable objective function we have,

[TABLE]

However, we now observe

[TABLE]

Hence we have,

[TABLE]

Thus if $\mathbf{w}$ and $\mathbf{z}$ solve $\mathcal{A}(t_{0},x_{0})$ with objective $L^{*}_{t_{0}}=\phi_{T}(z_{1}(T),z_{2}(T))$, then $\mathbf{w}$ and $\mathbf{z}_{1}$ solve $\mathcal{H}(t_{0},x_{0})$ with objective value $J^{*}_{t_{0}}$. ∎

Proposition 11**.**

The augmented DP problem $\mathcal{A}(t_{0},x_{0})$ in (9) satisfies the Principle of Optimality.

Proof.

$\mathcal{A}(t_{0},x_{0})$ is a special case of $\mathcal{P}(t_{0},x_{0})$ (3) where $c_{i}=0$ for $i\neq T$ and $c_{T}([z_{1},z_{2}]^{T})=\phi_{T}(z_{1},z_{2})$. Lemma 5 shows DP problems of the form $\mathcal{P}(t_{0},x_{0})$ satisfy the Principle of Optimality. ∎

Lemma 10 tells us that for any DP problem with forward separable objective, $\mathcal{H}(t_{0},x_{0})$ (8), there exists an equivalent DP problem of the form $\mathcal{A}(t_{0},x_{0})$ (9). Furthermore Proposition 11 shows that $\mathcal{A}(t_{0},x_{0})$ satisfies the Principle of Optimality. Therefore a solution for $\mathcal{H}(t_{0},x_{0})$ can be found by recursively solving Bellman’s equation (4) for $\mathcal{A}(t_{0},x_{0})$.

To understand the augmented approach intuitively, we note that DP breaks a multi-period planning problem into simpler sub-problems at each stage. However, for non-separable problems, to make the correct decision at each stage we need past information about the system. In this context, the augmented state contains the information from the trajectory history necessary to make the correct decision at the present time. However by adding augmented states we increase the state space dimension and the complexity of the DP problem.

Corollary 12**.**

Suppose the forward separable function, $J:\mathbb{R}^{m\times(T-t_{0})}\times\mathbb{R}^{n\times(T+1-t_{0})}\to\mathbb{R}$, is the objective function for DP problem $\mathcal{H}(t_{0},x_{0})$ (8) and has a representation dimension of $l$. Then the associated augmented DP problem with this representation, $\mathcal{A}(t_{0},x_{0})$ (9), has a state space of dimension $l+n$ and input space of dimension $m$.

Proof.

From the definition of $\mathcal{A}(t_{0},x_{0})$ (9), the state space dimension is $n+\max_{t_{0}\leq t\leq T}$, where $d_{t}=\dim(Im\{\phi_{t-1}\})$. From the definition of representation dimension, we have $\max_{t_{0}\leq t\leq T}d_{t}=l$ and hence it follows that the state space dimension is $n+\max_{t_{0}\leq t\leq T}d_{t}=n+l$. ∎

IV A Class Of DP For Which The Use Of State Augmentation Is Tractable

It is well known that discretization of the state space combined with a solution of Bellman’s equation become computationally intractable when the discretized dimension increases; this is often called “the curse of dimensionality”. In the previous Section, we proved that any non-separable DP of state space dimension $n$ can be converted to a separable augmented DP with state-space dimension $n+l$, where $l$ is the representation dimension of the objective function. However, for some representations, $l$ may increase as the time interval increases - thus triggering the curse of dimensionality. To address this problem, in this section, we define a class of forward separable objective functions, called Naturally Forward Separable (NFS) functions, with representation dimension, $l$, which is independent of the number of time steps and the dimension of the state and input space.

Before we define NFS functions we motivate this new class of functions by showing that it is possible to represent any function as a forward separable function. To do this we introduce some additional notation. Specifically, for a vector $v=(v_{1},...,v_{n})^{T}\in\mathbb{R}^{n}$ we define $[v]_{i}^{j}=(v_{i},...,v_{j})$ for $1\leq i<j\leq n$.

Lemma 13**.**

Any function $J:\mathbb{R}^{m\times(T-t_{0})}\times\mathbb{R}^{n\times(T+1-t_{0})}\to\mathbb{R}$ is forward separable with a representation of dimension $l(n,m,T-t_{0})=(T-t_{0})(n+m)$.

Proof.

Consider a function $J:\mathbb{R}^{m\times(T-t_{0})}\times\mathbb{R}^{n\times(T+1-t_{0})}\to\mathbb{R}$. To show $J$ is forward separable we define a forward separable representation $\{\phi_{i}\}_{i=t_{0}}^{T}$ which satisfy (6) as follows.

First, define $\phi_{t_{0}}:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}^{n+m}$ as

[TABLE]

For $i\in\{t_{0}+1,...T-1\}$ the define $\phi_{i}:\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{R}^{(i-t_{0})(n+m)}\to\mathbb{R}^{(i+1-t_{0})(n+m)}$ as

[TABLE]

Lastly, define $\phi_{T}:\mathbb{R}^{n}\times\mathbb{R}^{(T-t_{0})(n+m)}\to\mathbb{R}$ as

[TABLE]

Clearly, this definition of $\phi_{i}$ satisfies (6). Furthermore, it can be seen that the maximum dimension of the images of the maps $\{\phi_{i}\}_{i=t_{0}}^{T}$ is $(T-t_{0})(n+m)$ showing the dimension of this representation of $J$ is $l(n,m,T-t_{0})=(T-t_{0})(n+m)$. ∎

In the above approach to show that $J(\mathbf{u},\mathbf{x})$ is forward separable we naively took the strategy of using the functions $(\phi_{i})_{t_{0}\leq i\leq T}$ to act like memory functions; that is to store the entire historic state trajectory and input sequence used. If $J(\mathbf{u},\mathbf{x})$ is the objective function for some DP problem of form $\mathcal{H}(t_{0},x_{0})$ (8) then this approach would result in the associated augmented DP problem, $\mathcal{A}(t_{0},x_{0})$ (9), having a very large state space dimension. Specifically, Corollary 12 shows that $\mathcal{A}(t_{0},x_{0})$ has state space dimension $(T-t_{0})(n+m)+n$. Clearly, for a large number of time-steps, $T-t_{0}$, $\mathcal{A}(t_{0},x_{0})$ is intractable. For this reason we next define a special class of forward separable functions that have a representation with dimension independent of the number of time-steps.

Definition 14**.**

We say a function $J:\mathbb{R}^{m\times(T-t_{0})}\times\mathbb{R}^{n\times(T+1-t_{0})}\to\mathbb{R}$ is a Naturally Forward Separable (NFS) function if there exists maps, $\{\phi_{i}\}_{i=t_{0}}^{T}$, that satisfy (6) with representation dimension independent of $n$, $m$ and $T$.

IV-A An Algebra Of Naturally Forward Separable Functions

Given a function, $J:\mathbb{R}^{m\times(T-t_{0})}\times\mathbb{R}^{n\times(T+1-t_{0})}\to\mathbb{R}$, there is no obvious way to determine whether $J$ is NFS. Instead, in this section, we show that the set of NFS functions form an algebra closed under pointwise multiplication and which is preserved under nonlinear transformation - implying that simple NFS functions (‘building blocks’) can be combined to construct new, more complex, NFS functions. In this way, one might approach the problem of finding representation maps for a function, $J$, by combining known NFS “building blocks”. Several examples of such “building blocks” can be found in Subsection IV-B. We first prove closure under addition and pointwise multiplication.

Lemma 15**.**

Consider the function $U:\mathbb{R}\to\mathbb{R}$ and the NFS function, $J_{1}:\mathbb{R}^{m_{1}\times(T_{1}-t_{1})}\times\mathbb{R}^{n_{1}\times(T_{1}+1-t_{1})}\to\mathbb{R}$ and $J_{2}:\mathbb{R}^{m_{2}\times(T_{2}-t_{2})}\times\mathbb{R}^{n_{2}\times(T_{2}+1-t_{2})}\to\mathbb{R}$, with representation dimensions $l_{1}$ and $l_{2}$ respectively. The functions $G_{1}(\mathbf{u},\mathbf{x})=J_{1}(\mathbf{u},\mathbf{x})+J_{2}(\mathbf{u},\mathbf{x})$, $G_{2}(\mathbf{u},\mathbf{x})=J_{1}(\mathbf{u},\mathbf{x})\cdot J_{2}(\mathbf{u},\mathbf{x})$ and $G_{3}(\mathbf{u},\mathbf{x})=U\left(J_{1}(\mathbf{u},\mathbf{x})\right)$ are NFS functions with representation dimension less than or equal to $l_{1}+l_{2}$, $l_{1}+l_{2}$, and $l_{1}$, receptively.

Proof.

For simplicity let us consider the case where $t_{1}=t_{2}$ and $T_{1}=T_{2}$; other cases follow by the same argument. As $J_{1}$ and $J_{2}$ are forward separable functions there exist associated representations $\{g_{i}\}$ and $\{h_{i}\}$ such that $J_{1}$ and $J_{2}$ can be written in the form (6) and with associated representation dimensions $l_{1}$ and $l_{2}$, respectively. We now show that $G_{1}$ is forward separable by defining the associate representation $\{\phi_{i}\}$ such that $G_{1}$ can be written in the form (6). Specifically, let

[TABLE]

where $d_{i}=\dim(Im\{g_{i}\})$ and $s_{i}=\dim(Im\{h_{i}\})$ for $i\in\{t_{1},...,T_{1}-1\}$.

We conclude that $G_{1}$ has a representation dimension, denoted $l_{G_{1}}$, such that

[TABLE]

Furthermore, by a similar argument it can be shown that $G_{2}$ and $G_{3}$ are NFS with representation dimension less than or equal to $l_{1}+l_{2}$. We are able to show this using the same representation maps $\{\phi_{i}\}_{t_{1}\leq i\leq T_{1}-1}$ from (10) with the terminal representation map for $G_{2}$ given by

[TABLE]

and the terminal representation map for $G_{3}$ given by

[TABLE]

∎

IV-B Simple Examples Of NFS Functions

The first example of a NFS function is found in problems involving risk measures and certainty equivalents [12]. In this case, we have the function $U(x)=\frac{1}{\gamma}e^{\gamma x}$ and apply the following:

Example 16**.**

For any functions $U:\mathbb{R}\to\mathbb{R}$ and $c_{t}:\mathbb{R}^{n}\times\mathbb{R}^{m}\rightarrow\mathbb{R}$,

[TABLE]

is NFS with representation dimension 1.

Proof.

The additively separable function $\sum_{t=t_{0}}^{T-1}{c_{t}(x(t),u(t))}$ is NFS using the representation maps given in (7). It therefore follows $J$ is NFS by Lemma 15. ∎

Example 17**.**

The mixed p-norm function given by

[TABLE]

where $p_{j}>0$ for all $j\in\{1,...,N\}$, $c_{j,t}:\mathbb{R}^{n}\times\mathbb{R}^{m}\rightarrow\mathbb{R}^{+}$, and $N\in\mathbb{N}$, is NFS with representation dimension $N\in\mathbb{N}$.

Proof.

Follows as $J(\mathbf{u},\mathbf{x})$ that can be written in the form of (6) using maps $\phi_{t_{0}}(x,u)=[c_{1,t_{0}}(x(0),u(0))^{p_{1}},...,c_{N,t_{0}}(x(0),u(0))^{p_{N}}]^{T}$, $\phi_{i}(x,u,w)=[c_{1,i}(x(i),u(i))^{p_{1}}+w_{1},...,c_{N,i}(x(i),u(i))^{p_{N}}+w_{N}]^{T}$ for all $i\in\{t_{0}+1,...,T-1\}$, and $\phi_{T}(x,w)=\sum_{j=1}^{N}w_{j}^{\frac{1}{p_{j}}}$. ∎

We next give a NFS function that can be considered a discrete time version of the Green measure; when used as an objective function for a DP problem it measures the amount of time the state and input spend in some set.

Example 18**.**

Consider the function $J:\mathbb{R}^{m\times(T-t_{0})}\times\mathbb{R}^{n\times(T+1-t_{0})}\to\mathbb{R}$ defined as

[TABLE]

where $\mathbf{u}=(u(t_{0}),...,u(T-1))$, $u(t)\in\mathbb{R}^{m}$, $\mathbf{x}=(x(t_{0}),...,x(T))$, $x(t)\in\mathbb{R}^{n}$, $S\subset\mathbb{R}^{n}\times\mathbb{R}^{m}$ and for $B\subset\mathbb{N}$ we denote $|B|$ to be the cardinality of the set $B$. Then $J$ is NFS and has a representation of dimension 1.

Proof.

We present functions such that $J(\mathbf{u},\mathbf{x})$ that can be written in the form of (6).

Define $\phi_{t_{0}}:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}$ as

[TABLE]

Define $\phi_{t}:\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{R}\to\mathbb{R}$ for $1\leq t\leq T-1$ as

[TABLE]

Define the function $\phi_{T}:\mathbb{R}^{n}\times\mathbb{R}\to\mathbb{R}$ as

[TABLE]

This definition of $\phi_{i}$ satisfies (6). Moreover it can be seen that the maximum dimension of the images of the maps $\{\phi_{i}\}_{i=t_{0}}^{T}$ is $1$ implying that the dimension of this representation of $J$ is 1. ∎

Example 19**.**

Consider the variance type function, $J:\mathbb{R}^{m\times T}\times\mathbb{R}^{n\times(T+1)}\to\mathbb{R}$ defined as

[TABLE]

where $\mathbf{u}=(u(0),...,u(T-1))$, $u(t)\in\mathbb{R}^{m}$, $\mathbf{x}=(x(0),...,x(T))$, $x(t)\in\mathbb{R}^{n}$, and $a_{t}:\mathbb{R}^{n}\to\mathbb{R}$. Then $J$ is NFS and has a representation dimension of 2.

Proof.

Expanding the right hand side of (11) as in [14] we get,

[TABLE]

We now present functions $J(\mathbf{u},\mathbf{x})$ that can be written in the form of (6). We define $\phi_{t_{0}}:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}^{2}$ as

[TABLE]

We define $\phi_{i}:\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{R}^{2}\to\mathbb{R}^{2}$ as

[TABLE]

Finally, $\phi_{T}:\mathbb{R}^{n}\times\mathbb{R}^{2}\to\mathbb{R}$ is given by,

[TABLE]

This definition of $\phi_{i}$ satisfies (6). Moreover it can be seen that the maximum dimension of the images of the maps $\{\phi_{i}\}_{i=t_{0}}^{T}$ is $2$ showing the dimension of this representation of $J$ is 2. ∎

We now show that the maximum function, that appears in the objective function of the battery scheduling problem in Section IX, is NFS.

Example 20**.**

Consider the function $J:\mathbb{R}^{m\times T}\times\mathbb{R}^{n\times(T+1)}\to\mathbb{R}$ such that,

[TABLE]

where $\mathbf{u}=(u(0),...,u(T-1))$, $u(t)\in\mathbb{R}^{m}$, $\mathbf{x}=(x(0),...,x(T))$, $x(t)\in\mathbb{R}^{n}$, $c_{k}:\mathbb{R}^{m}\times\mathbb{R}^{n}\to\mathbb{R}$ for $0\leq k\leq T-1$ and $c_{T}:\mathbb{R}^{n}\to\mathbb{R}$. Then $J$ is NFS and has a representation dimension of 1.

Proof.

[TABLE]

It is now clear we can write $J$ in the form of (6) as follows. The function $\phi_{t_{0}}:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}$ is defined by,

[TABLE]

The function $\phi_{i}:\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{R}\to\mathbb{R}$ is defined by,

[TABLE]

The function $\phi_{T}:\mathbb{R}^{n}\times\mathbb{R}\to\mathbb{R}$ is defined by,

[TABLE]

This definition of $\phi_{i}$ satisfies (6). Moreover it can be seen that the maximum dimension of the images of the maps $\{\phi_{i}\}_{i=t_{0}}^{T}$ is $1$ showing the dimension of this representation of $J$ is 1. ∎

V Solving Deterministic Additively Separable DP Problems

In Section III we showed that all forward separable problems of the form $\mathcal{H}(t_{0},x_{0})$ have an equivalent DP problem of the form $\mathcal{A}(t_{0},x_{0})$. Problems of the form $\mathcal{A}(t_{0},x_{0})$ are special cases of problems of the form $\mathcal{P}(t_{0},x_{0})$. In this section we address the problem of implementation by numerically solving problems of the form $\mathcal{P}(t_{0},x_{0})$.

For implementation, we use an approximation scheme that maps our class of DP problems to a much simpler class of DP problems with finite state and control spaces. It is known for DP problems with finite state and control spaces that the infimum in Bellman’s equation (4) is attained and the optimal cost to go function, $F(x,t)$, can be computed by enumeration. Similar numerical schemes with convergence proofs can be found in [16] and [15].

V-A Construction Of Approximated Tractable DP Problems

Consider the DP problem $\mathcal{P}(t_{0},x_{0})$ (3) with compact state and control spaces of the form $X=[\underline{x},\bar{x}]^{n}$ and $U=[\underline{u},\bar{u}]^{m}$. For DP problems of this form it is not generally possible to solve Bellman’s Equation (4). We thus need to consider a sequence of “close” DP problems with countable state and control spaces. We define a sequence of approximated DP problems indexed by $k$ and denoted by $\mathcal{P}_{k}(t_{0},x_{0})$,

[TABLE]

where ${X}_{k}=\{x_{1},...,x_{k}\}^{n}$ such that $\underline{x}=x_{1}<x_{2}<...<x_{k}=\bar{x}$ and $||x_{i+1}-x_{i}||_{2}=\frac{\bar{x}-\underline{x}}{k}$ for $1\leq i\leq k-1$, and ${U}_{k}=\{u_{1},...,u_{k}\}^{m}$ such that $\underline{u}=u_{1}<u_{2}<...<u_{k}=\bar{u}$ and $||u_{i+1}-u_{i}||_{2}=\frac{\bar{u}-\underline{u}}{k}$ for $1\leq i\leq k-1$.

Note that our approximation scheme is based on the discretization of the state and input space and as such is subject to the “curse of dimensionality”. However, discretization-based schemes for solving Bellman’s equation have efficient parallel implementations - see [26]. Alternatively, other Approximate Dynamic Programming (ADP) schemes such as [27] use grid sampling and can be shown to converge with respect to expected cost.

V-B Constructing A Feasible Policy From The Solution Of The Approximated DP Problem

By iteratively solving Bellman’s equation (4) we can find an optimal solution to $\mathcal{P}_{k}(t_{0},x_{0})$ which we denote as $(\mathbf{x}^{*}_{k},\mathbf{u}^{*}_{k})$. Because the vector fields that define the underlying dynamics of $\mathcal{P}(t_{0},x_{0})$ and $\mathcal{P}_{k}(t_{0},x_{0})$ are different, the solution $(\mathbf{x}^{*}_{k},\mathbf{u}^{*}_{k})$ is not necessarily feasible for $\mathcal{P}(t_{0},x_{0})$. However using an optimal policy for $\mathcal{P}_{k}(t_{0},x_{0})$, $\pi^{*}_{k}$, we can construct a feasible policy for $\mathcal{P}(t_{0},x_{0})$ in the following way,

[TABLE]

where we recall $\Gamma_{t,x}$ is the set of feasible inputs such that if $u\in\Gamma_{t,x}$ then $u\in U$ and $f(x,u,t)\in X$ for the DP problem $\mathcal{P}(t_{0},x_{0})$ (3).

V-C Convergence Of Our Constructed Policy

Suppose $\theta_{k}(x,t)$, from (13), is a feasible policy for $\mathcal{P}(t_{0},x_{0})$ constructed from an optimal policy of $\mathcal{P}_{k}(t_{0},x_{0})$ using (13). Let $\mathbf{u}_{k}=(\theta_{k}(x_{0},t),...,\theta_{k}(x_{k}(T-1),T-1))$ and $\mathbf{x}_{k}=(x_{k}(t_{0}),...,x_{k}(T))$ where $x_{k}(t_{0})=x_{0}$, $x_{k}(t+1)=f(x_{k}(t),\theta_{k}(x_{k}(t),t),t)$ and $f$ is the vector field from $\mathcal{P}_{k}(t_{0},x_{0})$. From Theorem 2 in [16] if $\mathcal{P}(t_{0},x_{0})$ satisfies certain continuity assumptions then it is known that

[TABLE]

where $J_{t_{0}}(\mathbf{u}_{k},\mathbf{x}_{k})$ is the resulting value objective function of $\mathcal{P}(t_{0},x_{0})$ when the policy $\theta_{k}$ is used and $J_{t_{0}}^{*}$ is the optimal value of the objective function.

VI Stochastic DP And Optimal Policies

As shown in Counterexample 8 there exist non-separable deterministic DP problems that do not satisfy the Principle of Optimality as formulated in Definition 4. This Principle Of Optimality stated that the optimality of an input sequence for any instantiation of the sequence of DP problems is inherited by every subsequent instantiation - implying that recursive use of Bellman’s equation will eventually return an input which solves the DP problem for every instantiation.

In the stochastic case, however, additively separable DP problems may not satisfy this version of Principle of Optimality. Specifically, we show that even in the case of an additively separable objective function, stochastic perturbations can drive the system to a state wherein the original optimal input sequence is no longer optimal. This implies that for problems which include stochastic perturbations, a new criterion must be proposed for the use of Bellman’s equation. This reformulation of the Principle Of Optimality is non-trivial and requires us to re-work our definitions of DP. Specifically, we formulate the stochastic DP problem as a time-indexed sequence of DP problems, wherein the variable is not the input sequence, but is rather replaced by the policy. This policy is then used to generate a set of possible trajectories and input sequences, indexed by the set of possible instantiations of the random variables.

Using this new formulation of the stochastic DP problem, our definition of the Principle Of Optimality requires that optimality of the policy is inherited by every possible instantiation of the trajectory with probability one - i.e. the policy may be sub-optimal, but only on a set of instantiations of measure zero. We then show that the state-augmentation strategy we propose for DP problems with stochastic perturbations and NFS objective functions results in a stochastic DP problem for which this revised the Principle Of Optimality holds. Finally, we show how Bellman’s equation can be used in this case to recover an optimal policy.

We begin by defining the map from a chosen policy ($\pi$), initial condition ($x_{0}$) and instantiation of the random variables ($[\mathbf{v}]_{t_{0}}^{T-1}$) to the resulting trajectory, $x$; this will allow us to state precisely which random variables the expectation in the objective function is respect to. For simplicity we only consider random variables with Gaussian distributions.

Definition 21**.**

For a vector field $f$ : $\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{N}\times\mathbb{R}^{q}\to\mathbb{R}^{n}$, a set of polices $\Pi$ associated with some DP problem, a starting time $t_{0}\in\mathbb{N}$, and terminal time $T\in\mathbb{N}$, let us denote the state map by $\psi_{f,t_{0}}:\Pi\times\mathbb{R}^{n}\times\mathbb{N}\times\mathbb{R}^{q\times(T-t_{0})}\to\mathbb{R}^{n}$. We say that $x=\psi_{f,t_{0}}(\pi,x_{0},T,[\mathbf{v}]_{t_{0}}^{T-1})$ if $x=x(T)$ where $x(T)$ is a solution to the following recursion equations $x(t_{0})=x_{0}$, $x(t+1)=f(x(t),\pi(x(t),t),t,v(t))$ for $t\in\{t_{0},...,T-1\}$ and $[\mathbf{v}]_{t_{0}}^{T-1}=[v(t_{0}),..,v(T-1)]\in\mathbb{R}^{q\times(T-t_{0})}$. We denote the image of the state vector under a set of instantiations $Y\subset\mathbb{R}^{q\times(T-t_{0})}$ by $\psi_{f,t_{0}}(\pi,x_{0},T,Y)=\{\psi_{f,t_{0}}(\pi,x_{0},T,[\mathbf{v}]_{t_{0}}^{T-1})\in\mathbb{R}^{n}:[\mathbf{v}]_{t_{0}}^{T-1}\in Y\}$.

We also denote the trajectory map by $\Phi_{f,t_{0}}:\Pi\times\mathbb{R}^{n}\times\mathbb{N}\times\mathbb{R}^{q\times(T-t_{0})}\to\mathbb{R}^{m\times(T-t_{0})}\times\mathbb{R}^{n\times(T-t_{0}+1)}$. We say that $(\mathbf{u},\mathbf{x})=\Phi_{f,t_{0}}(\pi,x_{0},T,[\mathbf{v}]_{t_{0}}^{T-1})$ if $\mathbf{u}=(\pi(x(t_{0}),t_{0}),...,\pi(x(T-1),T-1))$, and $\mathbf{x}=(x(t_{0}),...,x(T))$ is such that $x(t)=\psi_{f,t_{0}}(\pi,x_{0},t,[\mathbf{v}]_{t_{0}}^{T-1})$ for $t\in\{t_{0},...,T-1\}$.

We define the class of stochastic DP problems with forward separable objective as $\mathcal{H}_{s}(t_{0},x_{0})$,

[TABLE]

where $J_{t_{0}}^{\mathcal{H}_{s}}:\mathbb{R}^{m\times(T-t_{0})}\times\mathbb{R}^{n\times(T-t_{0}+1)}\to\mathbb{R}$ is a forward separable function with associated representation $\{\phi_{i}\}_{i=t_{0}}^{T}$; $f$ : $\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{N}\times\mathbb{R}^{q}\to\mathbb{R}^{n}$; $\psi_{f,t_{0}}$ is measurable and $\Phi_{f,t_{0}}$ are the state and trajectory map respectively defined in Definition 21; $U_{i}$ is assumed to be some compact subset of $\mathbb{R}^{m\times(i-t_{0})}$; $X_{i}\subset\mathbb{R}^{n\times(i-t_{0}+1)}$; $[\mathbf{v}]_{t_{0}}^{T-1}=[v(t_{0}),..,v(t-1)]\in\mathbb{R}^{q\times(T-t_{0})}$; $\mathbb{E}_{\mathbf{v}}$ is the expectation with respect to the random variable $\mathbf{v}$. Define $J_{t_{0}}^{\mathcal{H}_{s}*}=\mathbb{E}_{[\mathbf{v}]_{t_{0}}^{T-1}}\left(J^{\mathcal{H}_{s}}_{t_{0}}(\Phi_{f,t_{0}}(\pi^{H_{s}*},x_{0},T,[\mathbf{v}]_{t_{0}}^{T-1}))\right)$ as the expected cost of using an optimal policy when applied to $\mathcal{H}_{s}(t_{0},x_{0})$.

Change in Variables for Stochastic Problems: Unlike in the deterministic case the solution to stochastic DP problems, such as (15), is now a policy $\pi\in\Pi$ and not a definite input and state sequence $\mathbf{u}^{*}\in\mathbb{R}^{m\times(T-t_{0})}$ and $\mathbf{x}^{*}\in\mathbb{R}^{n\times(T-t_{0}+1)}$, such as in (1). This is because the optimal sequence of inputs, $\mathbf{u}^{*}$, that results in an optimal trajectory, $\mathbf{x}^{*}$, will depend on the instantiation of the random variables. This change of notation demonstrates that the solution to DP problems involving stochastic dynamics no longer belongs to some finite dimensional space, $(\mathbf{u}^{*},\mathbf{x}^{*})\in\mathbb{R}^{m\times(T-t_{0})}\times\mathbb{R}^{n\times(T-t_{0}+1)}$, but rather an infinite dimensional functional space $\pi^{*}\in\Pi$.

VI-A Stochastic Additively Separable DP Problems

In the special case when the objective function of (15) is an additively separable function, as per Definition 1, given as

[TABLE]

we denote the DP problem (15) by $\mathcal{Q}(t_{0},x_{0})$.

stochastic DP problems of form $\mathcal{Q}(t_{0},x_{0})$ are often referred to as Markov Decision Processes (MDP), and are sometimes denoted by the tuple $\{\{X_{t}\}_{t_{0}}^{T},\{U_{t}\}_{t_{0}}^{T-1},\psi,\{Q_{t}\}_{t_{0}}^{T-1},\{c\}_{t_{0}}^{T}\}$; where $\psi(x,v,t)=\{u\in U_{t}:f(x,u,t,v)\in X_{t+1}\}$, $Q_{t}(B|x,u)=\int_{B}\mathds{1}_{B}(f(x,u,t,v))\phi(v)dv$, and $\phi(v)$ is the probability density function of the random variable $v$.

VI-B The Principle Of Optimality For Stochastic Problems

As discussed in [6] the extension of the Principle of Optimality to the stochastic case is non-trivial. We first give an example from [28] of a stochastic DP problem which shows that an optimal policy may not be optimal for every instantiation of the random variables at future time steps.

Let us consider the following stochastic DP problem $\mathcal{W}(0,x_{0})$,

[TABLE]

Here $J_{t}(\mathbf{u},\mathbf{x})=-\sum_{n=t}^{1}u(n)$, $f(x,u,t,v)=v$, and $\pi\in\Pi\iff\pi(x,t)\in\{0,1\}\forall x\in\mathbb{R},t=0,1$.

Counterexample 22**.**

The policy $\pi(x,t)=\begin{cases}1\text{ if }x\in[0,1)\\ 0\text{ if }x=1\end{cases}$ is optimal for the problem $\mathcal{W}(0,0)$ (17) but not optimal for the problem $\mathcal{W}(1,1)$.

Proof.

Clearly $J_{0}(\mathbf{u},\mathbf{x})\geq-2$ for all $(\mathbf{u},\mathbf{x})\in\{0,1\}^{2}\times\mathbb{R}^{3}$ and $J_{0}(\mathbf{u},\mathbf{x})=-2$ is attainable using the input $(u(0),u(1))=(1,1)$; therefore any solution of $\mathcal{W}(0,0)$ will minimize the objective function to a value of -2. Now using the law of total expectation we get,

[TABLE]

since the probability of a continuous random variable (such as a uniformly distributed random variable) taking a particular value is 0. Thus it follows the policy $\pi$ is optimal for $\mathcal{W}(0,0)$. Trivially $\pi$ is not optimal for $\mathcal{W}(1,1)$ as the value of the objective functions becomes 0 under $\pi$ whereas the input $u(1)=1$ produces a smaller objective function value of -1. ∎

Clearly, for the stochastic DP problems of form $\mathcal{H}_{s}(t_{0},x_{0})$ (15), such as $\mathcal{W}(0,0)$ (17), any optimal policy $\pi^{*}$ does not always result in the same trajectory $\mathbf{x}=(x(t_{0}),...,x(T))$; as this is dependent on the instantiations of the underlying random variables, $[\mathbf{v}]_{t_{0}}^{T-1}$. As Counterexample 22 has shown there exist stochastic DP problems, with additively separable objective functions, that have optimal policies that are no longer optimal for future timesteps if certain instantiations of the underlying random variables are realized. Therefore, it is too restrictive to extend Definition 4, the Principle of Optimality for the deterministic case, to the stochastic case by requiring stochastic DP problems satisfying the Principle of Optimality to be such that their optimal policies are also optimal for each instantiation at any future time step. With this in mind and motivated by the work of [7] we now give a probabilistic definition of the Principle of Optimality for stochastic DP problems.

Definition 23**.**

For a stochastic DP problem $\mathcal{H}_{s}(t_{0},x_{0})$ (15) with an optimal policy $\pi^{*}\in\Pi$ and associated state map $\psi_{f,t_{0}}$, defined in definition 21, let us denote the set indexed by $k\geq t_{0}$,

[TABLE]

where $[\mathbf{v}]_{t_{0}}^{k-1}=[v(t_{0}),...,v(k-1)]\in\mathbb{R}^{q\times(k-t_{0})}$. We say stochastic DP problems of the form $\mathcal{H}_{s}(t_{0},x_{0})$ (15) satisfy the Principle of Optimality if for any $k\geq t_{0}$ we have

[TABLE]

Here $\mathbb{P}_{[\mathbf{v}]_{t_{0}}^{k-1}}$ is the probability measure associated with the random variable $[\mathbf{v}]_{t_{0}}^{k-1}\in\mathbb{R}^{q\times(k-t_{0})}$, $v(t)\sim\mathcal{N}(\mathbf{0},I_{q\times q})$ for $t\in\{t_{0},...,k-1\}$.

We next show that stochastic DP problems with additively separable objective functions (MDP’s) satisfy the Principle of Optimality as formulated in Definition 23.

Lemma 24**.**

A stochastic DP problem of Form $\mathcal{Q}(t_{0},x_{0})$ (16) satisfies the Principle of Optimality as formulated in Definition 23.

Proof.

Suppose $\pi^{*}$ solves $\mathcal{Q}(t_{0},x_{0})$. For $k>t_{0}$ and the state map $\psi_{f,t_{0}}$ associated with $\mathcal{Q}(t_{0},x_{0})$ let us recall the set defined in Definition 23,

[TABLE]

Here $[\mathbf{v}]_{t_{0}}^{k-1}:=[v(t_{0}),...,v(k-1)]\in\mathbb{R}^{q\times(k-t_{0})}$, and we use the short-hand $x_{\pi^{*}}([\mathbf{v}]_{t_{0}}^{k-1}):=\psi_{f,t_{0}}(\pi^{*},x_{0},k,[\mathbf{v}]_{t_{0}}^{k-1})$.

Now for contradiction suppose there exists $k\in\{t_{0},...,T\}$ such that $\mathbb{P}_{[\mathbf{v}]_{t_{0}}^{k-1}}([\mathbf{v}]_{t_{0}}^{k-1}\in Y_{k})>0$; where $v(t)\sim\mathcal{N}(\mathbf{0},I_{q\times q})$ for $t\in\{t_{0},...,k-1\}$. For $[\mathbf{v}]_{t_{0}}^{k-1}\in Y_{k}$ we know the policy $\pi^{*}$ is not optimal for $\mathcal{Q}(k,x_{\pi^{*}}([\mathbf{v}]_{t_{0}}^{k-1}))$ and thus there exists a feasible policy $\theta\in\Pi$ such that,

[TABLE]

Now let us consider the map,

[TABLE]

Using Lemma 28, there exists a policy $\alpha\in\Pi$ such that (36) holds for $\{\hat{\pi}_{t}\}$ defined in (19). We will now show the policy $\alpha$ contradicts that $\pi^{*}$ be an optimal policy for $\mathcal{Q}(t_{0},x_{0})$. We first note using (36) and the law of total probabilities,

[TABLE]

We recall the additive structure of $J_{t_{0}}^{\mathcal{Q}}$

[TABLE]

where $\mathbf{u}=(u(t_{0}),...,u(T-1))\text{ and }\mathbf{x}=(x(t_{0}),...,x(T))$ and $c_{T}(x):\mathbb{R}^{n}\to\mathbb{R}$, $c_{t}(x,u):\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}$ for $t=t_{0},\cdots T-1$.

Now using the fact $\hat{\pi}_{t}([x(t_{0}),...,x(t)])=\pi^{*}(x(t),t)$ for all $t<k$, $\hat{\pi}_{t}([x(t_{0}),...,x(t)])=\theta(x(t),t)$ if $t\geq k$ and $x(k)\in\psi_{f,t_{0}}(\pi^{*},x_{0},k,Y_{k})$, linearity of the expectation and the inequality (18) we have,

[TABLE]

Therefore using (20); the fact $\hat{\pi}_{t}([x(t_{0}),...,x(t)])=\pi^{*}(x(t),t)$ if $x(k)\notin\psi_{f,t_{0}}(\pi^{*},x_{0},k,Y_{k,\pi^{*}})$; the total law of probability; the above inequality (21); and the assumption $\mathbb{P}_{[\mathbf{v}]_{t_{0}}^{k-1}}([\mathbf{v}]_{t_{0}}^{k-1}\in Y_{k})>0$ (so the inequality remains strict) we derive,

[TABLE]

This contradicts the fact $\pi^{*}$ is an optimal policy for $\mathcal{Q}(t,x)$. Therefore we conclude $P_{[\mathbf{v}]_{t_{0}}^{k-1}}([\mathbf{v}]_{t_{0}}^{k-1}\in Y_{k})=0$ showing DP problems of the form $\mathcal{Q}(t_{0},x_{0})$ satisfy Definition 23 and hence satisfy the Principle of Optimality. ∎

We will now state Bellman’s equation for stochastic DP problems of the form $\mathcal{Q}(t,x)$.

Proposition 25** ([24]).**

For stochastic DP problems of the form $\mathcal{Q}(t,x)$ in (16) with optimal objective values $J_{t}^{\mathcal{Q}*}$, define the function $F(x,t)=J_{t}^{\mathcal{Q}*}$. Then the following hold for all $x\in X_{t}$,

[TABLE]

We see in the next corollary that if we are able to solve the stochastic Bellman equation (23) then we are able to construct an optimal policy that solves (16).

Corollary 26** ([24]).**

Consider a stochastic DP problem of the form $\mathcal{Q}(t_{0},x_{0})$ in (16). Suppose $F(x,t)$ satisfies Equation (23) and suppose there exists a policy such that,

[TABLE]

Then the policy $\theta$ solves $\mathcal{Q}(t_{0},x_{0})$.

VI-C State Augmentation For Stochastic DP problems

Analogous to the deterministic case shown in Lemma 10, for a stochastic DP problem of the form $\mathcal{H}_{s}(t_{0},x_{0})$ (15) we can use the separable representation maps $\{\phi_{i}\}_{i=t_{0}}^{T}$ of the objective function $J^{\mathcal{H}_{s}}_{t_{0}}$ to construct an equivalent DP problem of form $\mathcal{Q}(t_{0},x_{0})$ (16) by using state augmentation.

VII Numerically Solving Additively Separable Stochastic DP Problems

In Section VI, we proposed a state-augmentation scheme for converting a stochastic forward separable DP problem, of form $\mathcal{H}_{s}(t_{0},x_{0})$ (15), to an equivalent additively separable DP problem, of form $\mathcal{Q}(t_{0},x_{0})$ (16). In this section we propose a scheme to numerically solve stochastic additively separable DP problems of the form $\mathcal{Q}(t_{0},x_{0})$ (16).

To numerically solve problems of the form $\mathcal{Q}(t_{0},x_{0})$, we propose a discretization scheme similar to the one detailed in Section V. However, unlike in the deterministic case, the presence of random variables, $v\sim\mathcal{N}(0,1)$, implies a non-compact state space. This requires the use of state projection onto appropriately constructed approximating compact sets.

VII-A Constructing An Approximated DP Problem With Compact State Space

Consider the stochastic DP problem $\mathcal{Q}(t_{0},x_{0})$ with compact control space $U=[\underline{u},\bar{u}]^{m}$ and underlying random variables $v\sim\mathcal{N}(\mathbf{0},I_{q\times q})$. As in [15], we assume $\forall\epsilon>0$ that there exists a compact set $H_{\varepsilon,t}=[\underline{x}_{\varepsilon,t},\bar{x}_{\varepsilon,t}]^{n}\subset X$ (that depends on $\varepsilon$ and $t$) such that $x_{0}\in H_{\varepsilon,0}$ and,

[TABLE]

We then construct the associated compact stochastic DP approximation to $\mathcal{Q}(t_{0},x_{0})$ denoted by $\mathcal{Q}_{\varepsilon,k}(t_{0},x_{0})$,

[TABLE]

where $\tilde{f}(x,u,t,v)=\arg\min_{y\in X_{\varepsilon,t+1,k}}\{||y-f(x,u,t,v)||_{2}\}$, $\tilde{X}_{\varepsilon,t,k}=\{x_{1,t},...,x_{k,t}\}^{n}$ such that $\underline{x}_{\varepsilon,t}=x_{1,t}<x_{2,t}<...<x_{k,t}=\bar{x}_{\varepsilon,t}$ and $||x_{i+1,t}-x_{i,t}||_{2}=\frac{\bar{x}_{\varepsilon,t}-\underline{x}_{\varepsilon,t}}{k}$ for $1\leq i\leq k-1$, $\tilde{U}_{k}=\{u_{1},...,u_{k}\}^{m}$ such that $\underline{u}=u_{1}<u_{2}<...<u_{k}=\bar{u}$ and $||u_{i+1}-u_{i}||_{2}=\frac{\bar{u}-\underline{u}}{k}$ for $1\leq i\leq k-1$, and $[\mathbf{v}]_{t_{0}}^{T-1}=[v(t_{0}),..,v(T-1)]\in\mathbb{R}^{q\times(T-t_{0})}$.

Analogous to the deterministic case, an optimal policy $\pi^{*}_{\varepsilon,k}$ for $\mathcal{Q}_{\varepsilon,k}(t_{0},x_{0})$ can be found exactly by iteratively solving Bellman’s equation (23). One can then construct a feasible policy for $\mathcal{Q}(t_{0},x_{0})$ using,

[TABLE]

where $\Gamma_{t,x}$ is the set of feasible controls at time $t\in\{0,...,T-1\}$ and state position $x\in\mathbb{R}^{n}$ for $\mathcal{Q}(t_{0},x_{0})$ (16) and $X_{\varepsilon,t,k}$ is the state grid constraint in the problem $\mathcal{Q}_{\varepsilon,k}(t_{0},x_{0})$ (25).

If $\mathcal{Q}(t_{0},x_{0})$ satisfies assumption (A1) to (A4) from Theorem 3.5 [15] then

[TABLE]

where $J_{t_{0}}^{\mathcal{Q}*}=\mathbb{E}_{[\mathbf{v}]_{t_{0}}^{T-1}}\left(J^{\mathcal{Q}}_{t_{0}}(\Phi_{f,t_{0}}(\pi^{\mathcal{Q}*},x_{0},T,[\mathbf{v}]_{t_{0}}^{T-1}))\right)$ is the expected cost of using an optimal policy when applied to $\mathcal{Q}(t_{0},x_{0})$.

VIII Summary: Solving NFS DP Problems Using Augmentation And Discretization

Given a DP problem with a NFS objective function, with known representation maps, we have shown in Section III and Section VI how to construct equivalent DP problems with additively separable objective functions. We have furthermore proposed discretization schemes in Section V, for the deterministic case, and Section VII-A, for the stochastic case, to solve DP problems with additively separable objective functions. We now summarize these results by proposing the following steps for solving a non-separable DP problem. Given a DP problem of the from $\mathcal{H}(t_{0},x_{0})$ (8), or $\mathcal{H}_{s}(t_{0},x_{0})$ (15) if stochastic, we do the following:

Find a NFS representation of the objective function (Eqn. (6)) with associated representation maps. One approach to this is to use Section IV-A which details how to combine known NFS functions, with known representation maps, in order to find potential representation maps for other NFS functions. 2. 2.

Construct the associated augmented DP problem of form $\mathcal{P}(t_{0},x_{0})$ (3), if deterministic, or $\mathcal{Q}(t_{0},x_{0})$ (16), if stochastic. 3. 3.

Use discretization to approximate the augmented DP problem using Form $\mathcal{P}_{k}(t_{0},x_{0})$ (12), if deterministic, or $\mathcal{Q}_{\varepsilon,k}(t_{0},x_{0})$ (25), if stochastic. 4. 4.

Numerically solve $\mathcal{P}_{k}(t_{0},x_{0})$ or $\mathcal{Q}_{\varepsilon,k}(t_{0},x_{0})$ for a sufficiently large $k\in\mathbb{N}$. 5. 5.

Construct a feasible policy for the original DP problem from an optimal policy of $\mathcal{P}_{k}(t_{0},x_{0})$ or $\mathcal{Q}_{\varepsilon,k}(t_{0},x_{0})$ using (13) or (26).

To illustrate how we use state augmentation and discretization methods we consider the following DP problem from [8].

[TABLE]

In [8] an analytic solution for (28) was found to be:

[TABLE]

The objective function $J$ in (28) is NFS and has a representation dimension of 2. This can be shown by writing $J$ in the form of (6) using the functions,

[TABLE]

The DP Problem (28) can now be written in the form of $\mathcal{A}(t_{0},x_{0})$ using state augmentation, as

[TABLE]

The DP Problem (29) is now a special case of $\mathcal{P}(t_{0},x_{0})$ and equivalent to the original DP Problem (28). The associated approximated DP problem of the form $\mathcal{P}_{k}(t_{0},x_{0})$ (12) can now be found by selecting appropriate compact state and control spaces; $X\subset\mathbb{R}^{3}$ and $U\subset\mathbb{R}$. A feasible policy for (28) is then constructed from an optimal policy of the associated $\mathcal{P}_{k}(t_{0},x_{0})$ using (13). Figure 2 shows the state trajectories by following different constructed policies for various values of $k$. It is seen that for $k=200$ the algorithm produces a solution within three significant figures of the analytic optimal objective function for (28).

IX Application To The Energy Storage Problem

We apply our augmented DP methodology to the scheduling of batteries in the presence of demand charges and show that our proposed algorithm outperforms existing heuristics, such as [22] (approximately $0.98 savings). To do this, we propose a simple model for the dynamics of battery storage. We then formulate the objective function using electricity pricing plans which include demand charges. We see that the system described becomes a DP problem of the form$\mathcal{S}(t_{0},x_{0})$ (5); which can be tractably solved as it has a NFS objective function. We will first solve the battery scheduling problem in the deterministic case based on real historical solar data. Later, we develop a stochastic Markov model that generates similar solar data to that seen in Tempe, Az.

IX-A Battery Dynamics

We model the energy stored in the battery using the difference equation:

[TABLE]

where $e(k)$ denotes the energy stored in the battery at time step $k$, $\alpha$ is the bleed rate of the battery, $\eta$ is the efficiency of the battery, $u({k})$ denotes the charging/discharging $(+/-)$ at time step $k$ and $\Delta t$ is the amount of time passed between each time step. Moreover we denote the maximum charge and discharge rate by $\bar{u}$ and $\underline{u}$ respectively. Thus we have the constraint that $u({k})\in[\underline{u},\bar{u}]:=U$ for all $k$. Similarly we also add the constraint $e(k)\in[\underline{e},\bar{e}]:=X$ for all $k$ where $\underline{e}$ and $\bar{e}$ are the capacity constraints of the battery (typically $\underline{e}=0$).

IX-B The Objective Function

Let us denote $q(k)$ as the power supplied by the grid at time step k.

[TABLE]

where $q_{a}(k)$ and $q_{s}(k)$ are the power consumed by HVAC/appliances and the power supplied by solar photovoltaics at time step $k$ respectively. For now, it is assumed that both are known a priori.

To define the cost of electricity we divide the day $t\in[0,T]$ into on-peak and off-peak periods. We define an off peak period starting from 12am till $t_{\text{on}}$ and $t_{\text{off}}$ till 12am. We define an on-peak period between $t_{\text{on}}$ till $t_{\text{off}}$. The Time-of-Use (TOU, $per kWh) electricity cost during on-peak and off-peak is denoted by$p_{\text{on}}$and$p_{\text{off}}$respectively. We further simplify this as$p_{k}=p_{on}$if$k\in T_{on}$and$p_{k}=p_{off}$if$k\in T_{off}$where$T_{on}$and$T_{off}$ are the on-peak and off-peak hours, respectively. These TOU charges define the first part of the objective function as:

[TABLE]

where the daily terminal timestep is $T=24/\Delta t$.

We also include a demand charge, which is a cost proportional to the maximum rate of power taken from the grid during on-peak times. This cost is determined by $p_{d}$ which is the price in $ per kW. Thus it follows the demand charge will be:

[TABLE]

IX-C 24 hr Optimal Residential Battery Storage Problem

We may now define the problem of optimal battery scheduling in the presence of demand and Time-of-Use charges, denoted $\mathcal{D}(0,e_{0})$.

[TABLE]

where recall $U:=[\underline{u},\bar{u}]$ and $X:=[\underline{e},\bar{e}]$.

Proposition 27**.**

Problem $\mathcal{D}(0,e_{0})$ is a special case of $\mathcal{S}(t_{0},x_{0})$ (5).

Proof.

Let $c_{i}=p_{i}(q_{a}(i)-q_{s}(i)+u({i}))\Delta t$

$d_{i}=\begin{cases}p_{d}(q_{a}(k)-q_{s}(k)+u_{k})&k\in T_{on}\\ 0&\text{otherwise.}\end{cases}$ ∎

We conclude that our approach to solving NFS DP problems can be applied to battery scheduling. That is, battery scheduling can be represented as an augmented DP problem of Form $\mathcal{A}(t_{0},x_{0})$.

IX-D Numerically Solving The Deterministic Battery Scheduling Problem

Our proposed approximation scheme can be applied to solve the battery scheduling problem, $\mathcal{D}(0,e_{0})$. This is done by creating an augmented state variable based on the maximum function in the objective function, as in Section III, and thus constructing an equivalent DP problem of the form $\mathcal{A}(0,x_{0})$ (9); which is a special case of $\mathcal{P}(t_{0},x_{0})$. Figure 3 shows how the monthly cost decreases when we use policies constructed from the associated discretized DP problems, $\mathcal{P}_{k}(t_{0},x_{0})$, as $k$ is increased. Although we do not get a monotonically decreasing sequence of costs, the error does decrease as $k\to\infty$. Figure 4 also shows that augmenting and then following our proposed discretization scheme for the battery scheduling problem results in a policy that reduces the consumption demand peak as $k$ is increased. Figure 5 shows how the computational time required to solve the discretized battery scheduling problem appears to be of exponential nature with respect to the number of grid points.

We used solar and usage data obtained by local utility Salt River Project (SRP) in Tempe, AZ, for power variables $q_{s}$ and $q_{a}$. We also use pricing data from SRP for the parameters $p_{\text{on}}$, $p_{\text{off}}$ and $p_{d}$. Battery data obtained for the Tesla Powerwall was used to determine the parameters $\alpha$, $\eta$, $\bar{u}$, $\underline{u}$ and $\bar{e}$. The results of the simulation are shown in Figure 6. The policy used for this simulation was created using our augmentation and approximation scheme with $k=20$. Interpolation was used to aid in solving Bellman’s equation (4) and decrease the approximation error. These results show an improvement in accuracy over results obtained for a similar problem in [22] (approximately $0.98 savings). As expected, we see the battery charges during off-peak and then discharges during on peak times to reduce ToU charges, while maintaining a reserve which it uses to keep consumption flat during on peak times, thereby minimizing the demand charge. As a result the power stabilizes during on peak times - becoming constant.

IX-E Solving The Stochastic Battery Scheduling Problem

To evaluate the effect of stochastic uncertainty on battery scheduling, we identify a Gauss-Markov model of solar generation based on SRP data. We construct the battery scheduling problem of the form $\mathcal{H}_{s}(t_{0},x_{0})$ (15) and then use our proposed state augmentation approach to construct an equivalent DP problem of form $\mathcal{Q}(t_{0},x_{0})$ (16). The problem of form $\mathcal{Q}(t_{0},x_{0})$ is then solved approximately using the methodology of Section VII-A.

IX-F Solar Generation Model

Our approach to modeling the dynamics of solar generation is based on [29]. Our Markov type model can be used to generate high resolution data over large time horizons. The Markov property of the model results in deviation from the mean being correlated time to time, helping represent the physical phenomena of clouds gradually passing overhead rather than instantaneously appearing.

Our model is a type of autoregressive-moving-average model (ARMAX) [30]. In [31] it is seen ARMAX models preform better than auto-regressive integrated moving average (ARIMA) and in [32] it is shown ARMAX models can produce data similar to real data for local sites in Califonia and Colorado.

Exogenous variables, temperature and humidity, are included as state variables in addition to the primary variable - solar radiance. Cross correlations between state variables are computed from data. Specifically, we take time-series data of these quantities, denoted $\mathbf{W}(t)$ and normalize this data as,

[TABLE]

where $\mu_{i}(t)$ is the average historic and clear-sky mean of the variable $W_{i}$ at time step $t$ and $\sigma_{i}(t)$ is the standard deviation of variable $W_{i}$ at time step $t$.

The generating process is then given by:

[TABLE]

where the matrices $A$ and $B$ are chosen to preserve the lag 0 and lag 1 cross-correlations seen in the collected data. Specifically, we can compute these matrices as ([29])

[TABLE]

where $M_{i}$ is the i-lag cross correlation matrix. So $(M_{i})_{m,n}=\rho_{i}(m,n)$ where $\rho_{i}(m,n)$ is the cross-correlation coefficient between variables m and n with variable n lagged by i time steps. Then, adding back in the mean and deviation, we obtain the power supplied by solar at time step $k$ as

[TABLE]

Figure 7 shows simulated irradiance data from our solar model when compared to actual recorded irradiance data. For this numerical implementation the mean and standard deviation, $(\mu_{i}(t))_{0\leq t\leq T}$ and $(\sigma_{i}(t))_{0\leq t\leq T}$, were calculated using data from Wunderground for a weather station in Tempe, AZ on October the 15th 2014 for each state variable. Cross correlations between the variables were also calculated from the same data set and (34) was solved giving the matrices $A$ and $B$ in (33). As seen in Figure 7 this solar generation model gives an output similar to what is observed in real data. Next we incorporate this model into our battery scheduling DP problems.

Stochastic Battery Scheduling We now modify Problem $\mathcal{D}(0,e_{0})$ (32) to give a stochastic version of the battery scheduling problem $\mathcal{D}_{s}(0,[e_{0},0])$,

[TABLE]

where $J_{E}$ is the ToU cost function and $J_{D}$ is the demand charge found in Section IX-A; $f([e,w],u,t,v)=\begin{bmatrix}\alpha(e+\eta u\Delta t)\\ Aw+Bv\end{bmatrix}$; $E_{t}=[\underline{e},\bar{e}]$ and $U_{t}=[\underline{u},\bar{u}]$ for all $t\in\{0,...,T\}$; $\psi_{f,t_{0}}$ and $\Phi_{f,t_{0}}$ are the state and trajectory map respectively from Definition 21; $[\mathbf{v}]_{0}^{T-1}=[v(0),..,v(T-1)]\in\mathbb{R}^{3\times T}$; matrices $A$ and $B$ are calculated from weather data using equations (34); and all constants are found in Table II.

IX-G Numerically Solving The Stochastic Battery Scheduling Problem

Using the state augmentation procedure in Section III on the stochastic battery scheduling problem $\mathcal{D}_{s}(0,[e_{0},0])$ (35), we may find a stochastic DP problem of the form $\mathcal{Q}(t_{0},x_{0})$ (16) such that an optimal policy for $\mathcal{D}_{s}(0,[e_{0},0])$ can be constructed from an optimal policy of $\mathcal{Q}(t_{0},x_{0})$. We may then construct the approximated stochastic DP problem $\mathcal{Q}_{\varepsilon,k}(t_{0},k)$ (25) and solve it using Bellman’s equation (23). From an optimal policy of $\mathcal{Q}_{\varepsilon,k}(t_{0},k)$ we then construct a feasible policy for $\mathcal{Q}(t_{0},x_{0})$ using (26). Figure 1 demonstrates a simulation of using the feasible policy obtained via augmenting and approximating the stochastic battery scheduling problem with a reasonably selected family of compact state spaces, $\{H_{\varepsilon,t}\}_{0\leq t\leq T}$, and discretization level $k=10$. To simplify computation we use a one state version of our solar model (33) and use interpolation while solving Bellman’s equation. As expected the battery charges during the on peak times and conservatively discharges during the off-peak times. The solar data generated from this run are then used as input to the deterministic algorithm in order to compare performance. As anticipated, the deterministic case performs better than the stochastic case.

X Conclusion

In this paper we propose a general formulation of the DP problem. We show that if the objective function is forward separable, DP problems may reformulated using state augmentation as an equivalent DP problem with additively separable objective function. Furthermore, we define a class of functions, called naturally forward separable (NFS) functions, such that DP problems with an objective function of this class can be tractably solved using state augmentation. Moreover, we show that the problem of optimal scheduling of battery storage in the presence of combined demand and time-of-use charges is a special case of this class of NFS DP problems. We further extend these results to stochastic DP problems with a NFS objective. The proposed algorithms are applied to a battery scheduling problem using first a deterministic and then Gauss-Markov model for solar generation and load.

Extensions of this work include the use of non-separable input constraints (such as those considered in [33]) and algorithms for finding the minimal dimension NFS representation of a given objective function.

XI Appendix

Lemma 28**.**

Consider a DP problem of the form $\mathcal{Q}(t_{0},x_{0})$ (16) with additively separable objective function $J_{t_{0}}^{\mathcal{Q}}$. For any family of functions of the form $\hat{\pi}:\mathbb{R}^{n\times(t-t_{0}+1)}\to\mathbb{R}^{m}$ are such $\hat{\pi}_{t}([(x(t_{0}),....,x(t))])\in U_{t}$ and $f(x(t),\hat{\pi}_{t}([(x(t_{0}),....,x(t))]),t,v(t))\in X_{t+1}$ for all $x(i)\in X_{i}$, $i\in\{t_{0},...,t\}$, $v(t)\in\mathbb{R}^{q}$ and $t\in\{t_{0},...,T-1\}$ there exists $\alpha\in\Pi$ such that

[TABLE]

where we make a small abuse of notation to extend the trajectory map $\Phi_{f,t_{0}}$ to policies that use the entire state space history.

Proof.

Proposition 8.1 [34] or Theorem 6.2 [7]. ∎

Acknowledgments

This research was supported by the NSF Grant CNS-1739990. We gratefully acknowledge data and expertise provided by Robert Hess of Salt River Project.