Duality and upper bounds in optimal stochastic control governed by partial differential equations

TL;DR

This paper introduces a dual control framework for stochastic PDE-governed systems, establishing duality theorems that help bound and identify the optimal control values.

Contribution

It develops a dual control problem for stochastic PDE systems and proves duality theorems linking the original and dual problems, aiding in optimal value estimation.

Findings

Duality theorems relate original and dual problem values.

Dual problem provides upper bounds for the original control problem.

The approach helps in estimating and achieving optimal control values.

Abstract

A dual control problem is presented for the optimal stochastic control of a system governed by partial differential equations. Relationships between the optimal values of the original and the dual problems are investigated and two duality theorems are proved. The dual problem serves to provide upper bounds for the optimal and maximum value of the original one or even to give the optimal value.

Equations

$$dX(t,x)$$

$$X(0,x)$$

$$X(t,x)$$

$$\hat{A}\varphi (x)=\sum _{i,j=1}^n{a}_{ij}(x)\frac{{\partial }^2\varphi (x)}{\partial x_i\partial x_j}+\sum _{i=1}^n{b}_i(x)\frac{\partial \varphi (x)}{\partial x_i},$$

$$\mathcal{A}=\{u(t,x)|u(t,x)\in \mathcal{U}\mathrm{is}{\mathcal{F}}_t-\mathrm{measurable}\mathrm{for}\mathrm{all}(t,x)\}.$$

$$\displaystyle{\mathcal{J}}(u)=\mathbb{E}\Big{[}\int_{0}^{T}\Big{(}\int_{V}\big{(}F(t,x,X(t,x))+G(t,x,u(t,x))\big{)}dx\Big{)}dt\Big{]},$$

$${\mathcal{J}}^{\ast }=\underset{u\in \mathcal{A}}{\sup }\mathcal{J}(u)=\mathcal{J}(u^{\ast }).$$

$$\underset{u\in \mathcal{A}}{\sup }\mathcal{J}(u).$$

$${\hat{A}}^{\ast }\varphi (x)=\sum _{i,j=1}^n\frac{{\partial }^2(a_{ij}(x)\varphi (x))}{\partial x_i\partial x_j}-\sum _{i=1}^n\frac{\partial (b_i(x)\varphi (x))}{\partial x_i}.$$

$$\displaystyle H(t,x,p)=\sup_{u\in\mathcal{U}}\,\big{(}G(t,x,u)+p\,C(t,x,u)\big{)}.$$

$$\displaystyle\mathcal{L}(X,p)=\mathbb{E}\Big{[}\int_{V}\int_{0}^{T}\Big{(}H(t,x,p(t,x))+F(t,x,X(t,x))-X(t,x)\frac{\partial F(t,x,X(t,x))}{\partial X}\Big{)}dtdx$$

$$\displaystyle-\int_{V}\int_{0}^{T}\Big{(}X(t,x)\hat{A}^{\ast}p(t,x)-p(t,x)\hat{A}X(t,x)\Big{)}dtdx$$

$$\displaystyle+\int_{V}\Big{(}p(0,x)\xi(x)+\int_{0}^{T}p(t,x)\sigma(t,x)dB(t)\Big{)}dx\Big{]},$$

$$-dp(t,x)/dt={\hat{A}}^{\ast }p(t,x)+\partial F(t,x,X(t,x))/\partial X;$$

$$p(T,x)=0\mathrm{for}x\in \bar{V},p(t,x)=0\mathrm{for}(t,x)\in (0,T)\times \Gamma ;$$

$$X(0,x)=\xi (x)\mathrm{for}x\in \bar{V},X(t,x)=\eta (t,x)\mathrm{for}(t,x)\in (0,T)\times \Gamma .$$

$$\underset{(X,p)\in \mathcal{B}}{\inf }\mathcal{L}(X,p).$$

$$\underset{u\in \mathcal{A}}{\sup }\mathcal{J}(u)\le \underset{(X,p)\in \mathcal{B}}{\inf }\mathcal{L}(X,p).$$

$$\displaystyle\mathcal{J}(u)=\mathbb{E}\Big{[}\int_{0}^{T}\Big{(}\int_{V}\big{(}F(t,x,\bar{X}(t,x))+G(t,x,u(t,x))\big{)}dx\Big{)}dt\Big{]}.$$

$$\displaystyle L(u;X^{\circ},p^{\circ})=\mathbb{E}\Big{[}\int_{V}\int_{0}^{T}\Big{(}F(t,x,X^{\circ}(t,x))+G(t,x,u(t,x))+p^{\circ}(t,x)C(t,x,u(t,x))$$

$$\displaystyle-X^{\circ}(t,x)\frac{\partial F(t,x,X^{\circ}(t,x))}{\partial X}\Big{)}dtdx-\int_{V}\int_{0}^{T}\Big{(}X^{\circ}(t,x)\hat{A}^{\ast}p^{\circ}(t,x)-p^{\circ}(t,x)\hat{A}X^{\circ}(t,x)\Big{)}dtdx$$

$$\displaystyle+\int_{V}\Big{(}p^{\circ}(0,x)\xi(x)+\int_{0}^{T}p^{\circ}(t,x)\sigma(t,x)dB(t)\Big{)}dx\Big{]},$$

$$-d{p}^{\circ }(t,x)/dt$$

$$p^{\circ }(T,x)$$

$$X^{\circ }(0,x)$$

$$\displaystyle~{}~{}~{}~{}~{}{\mathcal{J}}(u)-L(u;X^{\circ},p^{\circ})=\mathbb{E}\Big{[}\int_{V}\int_{0}^{T}\Big{(}F(t,x,\bar{X}(t,x))-F(t,x,X^{\circ}(t,x))-p^{\circ}(t,x)C(t,x,u(t,x))$$

$$\displaystyle+X^{\circ}(t,x)\frac{\partial F(t,x,X^{\circ}(t,x))}{\partial X}\Big{)}dtdx+\int_{V}\int_{0}^{T}\Big{(}X^{\circ}(t,x)\hat{A}^{\ast}p^{\circ}(t,x)-p^{\circ}(t,x)\hat{A}X^{\circ}(t,x)\Big{)}dtdx$$

$$\displaystyle-\int_{V}\Big{(}p^{\circ}(0,x)\xi(x)+\int_{0}^{T}p^{\circ}(t,x)\sigma(t,x)dB(t)\Big{)}dx\Big{]}.$$

$$F(t,x,\bar{X}(t,x))-F(t,x,X^{\circ }(t,x))\le \frac{\partial F(t,x,X^{\circ }(t,x))}{\partial X}(\bar{X}(t,x)-X^{\circ }(t,x)),$$

$$\displaystyle{\mathcal{J}}(u)-L(u;X^{\circ},p^{\circ})\leq\mathbb{E}\Big{[}\int_{V}\int_{0}^{T}\Big{(}\frac{\partial F(t,x,X^{\circ}(t,x))}{\partial X}\bar{X}(t,x)-p^{\circ}(t,x)C(t,x,u(t,x))\Big{)}dtdx$$

$$\displaystyle+\int_{V}\int_{0}^{T}\Big{(}X^{\circ}(t,x)\hat{A}^{\ast}p^{\circ}(t,x)-p^{\circ}(t,x)\hat{A}X^{\circ}(t,x)\Big{)}dtdx~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}$$

$$\displaystyle-\int_{V}\Big{(}p^{\circ}(0,x)\xi(x)+\int_{0}^{T}p^{\circ}(t,x)\sigma(t,x)dB(t)\Big{)}dx\Big{]}.$$

$$\partial F(t,x,X^{\circ }(t,x))/\partial X=-d{p}^{\circ }(t,x)/dt-{\hat{A}}^{\ast }{p}^{\circ }(t,x),$$

$$\displaystyle{\mathcal{J}}(u)-L(u;X^{\circ},p^{\circ})\leq-\mathbb{E}\Big{[}\int_{V}\int_{0}^{T}\Big{(}\frac{dp^{\circ}(t,x)}{dt}\bar{X}(t,x)+\bar{X}(t,x)\hat{A}^{\ast}p^{\circ}(t,x)~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}$$

$$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}+p^{\circ}(t,x)C(t,x,u(t,x))\Big{)}dtdx-\int_{V}\int_{0}^{T}\Big{(}X^{\circ}(t,x)\hat{A}^{\ast}p^{\circ}(t,x)-p^{\circ}(t,x)\hat{A}X^{\circ}(t,x)\Big{)}dtdx~{}~{}~{}~{}~{}~{}$$

$$\displaystyle+\int_{V}\Big{(}p^{\circ}(0,x)\xi(x)+\int_{0}^{T}p^{\circ}(t,x)\sigma(t,x)dB(t)\Big{)}dx\Big{]}.~{}~{}~{}~{}~{}$$

$$\displaystyle d\bar{X}(t,x)=\big{(}\hat{A}\bar{X}(t,x)+C(t,x,u(t,x)\big{)}dt+\sigma(t,x)dB(t),$$

$${\int }_0^T\frac{d{p}^{\circ }(t,x)}{dt}\bar{X}(t,x)dt=p^{\circ }(T,x)\bar{X}(T,x)-p^{\circ }(0,x)\bar{X}(0,x)$$

Full text

1. Introduction

The original problem (or primal problem) considered is the optimal control of a system governed by a stochastic heat equation that is described in [4], which is a maximization problem. In this paper, to the problem we associate another, called its dual problem, which is in turn a minimization problem. We prove two types of duality theorem.

First we show that solutions of the dual problem provide upper bounds for the maximum of the primal problem. We call this assertion a weak duality theorem. Next, under some conditions related to the maximum principle of control theory, the maximum can be attained by solving the dual problem. Such a property is called a strong duality theorem.

Let $T>0$ and $V$ be a bounded and open domain in $\mathbb{R}^{n}$ with $C^{1}$ boundary $\partial V=\Gamma$. On $[0,T]\times V$ we consider the following stochastically controlled system. The one-dimensional Brownian motion $B(t)=B(t,\omega)$ is defined on a filtered probability space $(\Omega,\mathcal{F},\{\mathcal{F}_{t}\}_{t\geq 0},P)$. The state of the system is denoted by $X(t,x)\in\mathbb{R}$, which is controlled by $u(t,x)\in\mathbb{R}$ for $t\in[0,T]$ and at $x\in\bar{V}=V\cup\Gamma$. The control process $u(t,x)=u(t,x,\omega)$ satisfies $u(t,x)\in\mathcal{U}$, where $\mathcal{U}$ is a bounded set of $\mathbb{R}^{k}$, and it is $\mathcal{F}_{t}$-measurable for all $(t,x)\in(0,T)\times V$. The state $X(t,x)$ is described by a stochastic heat equation of the form

[TABLE]

The boundary value functions $\xi$ on $\bar{V}$, and $\eta$ on $(0,T)\times\Gamma$ are $C^{1}$ real-valued and deterministic. $\hat{A}$ is a second order partial differential operator acting on smooth functions of $x$:

[TABLE]

where $(a_{ij}(x))$ is a symmetric nonnegative definite $n\times n$ matrix with entries $a_{ij}(x)\in C^{2}(V)\cap C(\bar{V})$ and $b_{i}(x)\in C^{2}(V)\cap C(\bar{V})$ for $1\leq i\leq n$. A control process $u(t,x)$ is called admissible if the corresponding solution $X_{u}(t,x)$ of Eqs.(1)-(3) is unique and belongs to $L^{2}(\Lambda\times P)$, where $\Lambda$ is the Lebesgue measure on $[0,T]\times\bar{V}$. The set of all admissible controls is denoted by $\mathcal{A}$;

[TABLE]

The $C^{1}$ functions $C$ and $\sigma$ in (1) are, respectively, $C:[0,T]\times V\times\mathcal{U}\to\mathbb{R}$ and $\sigma:[0,T]\times V\to\mathbb{R}$. The expected performance (or payoff) is given by, for each $u\in\mathcal{A}$,

[TABLE]

where $X(t,x)=X_{u}(t,x)$. Throughout this paper we impose the following;

(Assumption) $F:[0,T]\times V\times\mathbb{R}\to\mathbb{R}$ is a $C^{1}$ function that is concave with respect to $X$.

$G:[0,T]\times V\times\mathcal{U}\to\mathbb{R}$ is a bounded continuous function, and $\mathbb{E}$ denotes the expectation with respect to the probability measure $P$. The aim of the primal problem is to find a maximizing control $u^{\ast}\in\mathcal{A}$ and ${\mathcal{J}}^{\ast}\in\mathbb{R}$ such that

[TABLE]

Thus the primal problem is formulated as

[TABLE]

In the next section a dual problem to (5) is proposed. Similar dual control problems were constructed for max-min control problems in [5], for non-well-posed distributed systems in [6] and for optimal stochastic control in [7]. When a primal problem is a minimization problem, its dual problem serves to provide lower bounds for the minimum value of the primal one. Here the primal problem is a maximization problem, its dual problem provides upper bounds for the maximum value. Under some conditions related to the maximum principle of control theory it is also able to attain the maximum.

2. Dual Problem

The adjoint of the differential operator $\hat{A}$ is defined by

[TABLE]

In order to present the dual problem, for each real number $p$, we define a function

[TABLE]

Note that the control variable $u$ of the primal problem disappears at this stage.

The dual control problem is the system with performance functional that is to be minimized:

[TABLE]

over all variables $X$ and $p$ that satisfy:

[TABLE]

The variable $X(t,x)$ plays a role of control process of the dual problem that is a continuous process belonging to $L^{2}(\Lambda\times P)$. As indicated by the strong duality theorem (Section 4), $X(t,x)$ may be a solution of Eqs.(1)–(3), which indeed becomes a continuous process. Or it can be even a deterministic and continuous variable. Hence the dual problem is more manageable than the primal one. The variable $p(t,x)$, in turn, represents the state of the dual problem. We denote by $\mathcal{B}$ the set of all pairs $(X,p)$ that satisfy Eqs.(8)–(10). So the dual problem is formulated as

[TABLE]

3. Weak Duality Theorem

In this section we show that solutions of the dual problem provide upper bounds for the maximum of problem (5). We call this property a weak duality theorem.

Theorem 1. Under the concavity of the function $F$ it follows that

[TABLE]

Proof. Let $u\in\mathcal{A}$ be an admissible control and let us fix it for the moment. Let $\bar{X}$ be the solution of Eqs.(1)-(3) for $u$ and put (see Eq.(4));

[TABLE]

On the other hand, for the same $u$ we consider the following expectation, using an arbitrary $(X^{\circ},p^{\circ})\in\mathcal{B}$:

[TABLE]

where $(X^{\circ},p^{\circ})$ is a solution of Eqs.(8)-(10):

[TABLE]

Making use of these fixed $u\in\mathcal{A}$ and $(X^{\circ},p^{\circ})\in\mathcal{B}$, the difference between ${\mathcal{J}}(u)$ and $L(u;X^{\circ},p^{\circ})$ is

[TABLE]

By the concavity of $F$ with respect to $X$ we have

[TABLE]

from which we have the inequality

[TABLE]

We show that the right-hand side of (13) is equal to zero. From Eq.(8) it follows that

[TABLE]

and that

[TABLE]

On the other hand, since $\bar{X}(t,x)$ satisfies

[TABLE]

we get by integration of parts ([2])

[TABLE]

where we used $p^{\circ}(T,x)=0$ and $\bar{X}(0,x)=\xi(x)$. Since $\bar{X}-X^{\circ}=0$ (see Eqs.(3) and (10)) and $p^{\circ}=0$ on $\Gamma$, the surface of $V$, the first Green formula ([8, p.258]) implies

[TABLE]

From this equality it follows that

[TABLE]

Upon substituting Eqs.(15), (16) into (14), we see that the right-hand side of Eq.(14) (and (13)) is equal to zero. Hence we can conclude that for each $u\in\mathcal{A}$ it follows that

[TABLE]

Since $(X^{\circ},p^{\circ})\in{\mathcal{B}}$ is arbitrary, we have

[TABLE]

The optimal value for the primal problem is $\sup_{u\in\mathcal{A}}{\mathcal{J}}(u)$ and it satisfies

[TABLE]

By a well-known inequality of game theory [3], we have

[TABLE]

In view of (6) we see that for each fixed $(X^{\circ},p^{\circ})\in\mathcal{B}$ the value $\sup_{u\in\mathcal{A}}~{}L(u;X^{\circ},p^{\circ})$ is identical to Eq.(7) of the dual problem, that is, ${\mathcal{L}}(X^{\circ},p^{\circ})$, which is to be minimized. Therefore, we obtain

[TABLE]

This proves the weak duality theorem.

The last inequality shows that each $(X,~{}p)\in{\mathcal{B}}$ provides an upper bound for the primal problem.

4. Strong Duality Theorem

In this sction we assume that a control process $\bar{u}$ satisfies a sort of the maximum principle of optimality such as in [4, Theorem 2.1]. Under the concavity of the function $F$ in Eq.(4), it entails the strong duality theorem. More precisely, the corresponding solution $\bar{X}=X_{\bar{u}}$ of Eqs.(1)-(3) provides an optimal control for the dual problem and there is no duality gap; both extreme values (5) and (11) are exactly equal.

Theorem 2. Suppose $\bar{X}$ is a solution of Eqs.(1)-(3)* for an admissible control $\bar{u}\in\mathcal{A}$, and that $\bar{p}$, together with this $\bar{X}$, is a solution of Eqs.(8)-(10). If $\bar{u}\in\mathcal{A}$ satisfies*

[TABLE]

the function $H$ being defined by (6), then $\bar{u}$ is an optimal control of the primal problem and $\bar{X}$ is that of the dual one. Moreover, there is no duality gap;

[TABLE]

Proof. The proof is similar to that of Theorem 1. Let us put

[TABLE]

On the other hand, using (7) and (18), we have

[TABLE]

We evaluate the difference

[TABLE]

Now it is easy to prove that the difference is equal to zero, using a similar calculation to the right-hand side of (13); $\mathcal{J}(\bar{u})={\mathcal{L}}(\bar{X},\bar{p})$. Using Theorem 1 (weak duality), it follows that $\bar{u}$ is an optimal control for the primal problem and that $(\bar{X},\bar{p})$ is an optimal pair for the dual one. This completes the proof.

Although our system is simpler than that of [4] and the approach is different from it, Eq.(18) turns out a sufficient optimality condition for the primal problem.

5. Partial Observation Control

In partially observable systems as in [1], it is necessary to consider controls that do not depend on the space variable $x$. We denote the subset of such controls by $\mathcal{A}_{1}$;

[TABLE]

The primal problem is to maximize the functional

[TABLE]

over $u\in\mathcal{A}_{1}$ together with $X(t,x)$ satisfying

[TABLE]

The dual system is governed by Eqs.(8)-(10) as before. In order to formulate the dual problem, let us put

[TABLE]

for functions $p(t,x)$ that are solutions of Eqs.(8)-(10). The dual problem is to minimize the functional

[TABLE]

over all $p(t,x)$ and $X(t,x)$ satisfying Eqs.(8)-(10). Note that this type of dual problem takes a more similar form to the one dealt with in [7].

We prove two duality theorems. To do this, let us take an arbitrarily chosen control $u\in\mathcal{A}_{1}$, and introduce the corresponding functional $L(u;X,p)$ similar to Eq.(12), while $X,p$ satisfy Eqs.(8)-(10), i.e., $(X,p)\in\mathcal{B}$. Then we can derive the inequality analogous to (17);

[TABLE]

for all $(X,p)\in\mathcal{B}$. Among the terms of $L(u;X,p)$, those relevant to $u(t)$ are $G(t,x,u(t))$ and $p(t,x)C(t,x,u(t))$. Hence we divide $\sup_{u\in\mathcal{A}_{1}}L(u;X,p)$ into two parts: one is

[TABLE]

and the other is

[TABLE]

[TABLE]

Using a measurable selection theorem, Fubini’s theorem and Eq.(19), we see that the expectation (20) can be written as

[TABLE]

This together with (21) yields the functional $\mathcal{L}_{1}(X,p)$ for which the weak duality theorem holds;

[TABLE]

Next suppose that $\bar{X}$ is a solution of Eqs.(1)-(3) for an admissible control $\bar{u}\in\mathcal{A}_{1}$, and that $(\bar{X},\bar{p})\in\mathcal{B}$ satisfies (averaged maximum condition in [4])

[TABLE]

Then we obtain the equality ${\mathcal{J}}(\bar{u})=\mathcal{L}_{1}(\bar{X},\bar{p})$ and hence the strong duality theorem as in Section 4, implying no duality gap

[TABLE]

Moreover, from the weak duality theorem it follows that $\bar{u}$ provides an optimal control for the primal problem, and so does the pair $(\bar{X},\bar{p})$ for the dual problem.

References

• [1]

A. Bensoussan, Stochastic Control of Partially Observable Systems, Cambridge Univ. Press, 1992.

• [2]

K. L. Chung and R. J. Williams, Introduction to Stochastic Integration, Second Edition, Birkhäuser, 1990.

• [3]

S. Karlin, Mathematical Methods and Theory in Games, Programming and Economics, Vol. I, Addison-Wesley, 1959.

• [4]

B. Øksendal, Optimal control of stochastic partial differential equations, Stochastic Analysis and Applications, 23, No. 1, 165–179, 2005.

• [5]

S. Tanimoto, A duality theorem for max-min control problems, IEEE Transactions on Automatic Control, AC-27, No. 5, 1129–1131, 1982.

• [6]

S. Tanimoto, Duality in the optimal control of non-well-posed distributed systems, Journal of Mathematical Analysis and Applications, 171, 277–287, 1992.

• [7]

S. Tanimoto, Duality and lower bounds in optimal stochastic control, International Journal of Systems Science, 25, 1365–1372, 1994.

• [8]

J. Wloka, Partial Differential Equations, Cambridge Univ. Press, 1987.