A Multilevel Approach for Stochastic Nonlinear Optimal Control
Ajay Jasra, Jeremy Heng, Yaxian Xu, Adrian N. Bishop
TL;DR
This paper introduces a novel multilevel Monte Carlo method for efficiently solving stochastic nonlinear optimal control problems, significantly reducing computational costs compared to existing approaches.
Contribution
It proposes a multilevel Monte Carlo approach combined with smoothing algorithms to improve the efficiency of path integral computations in stochastic control.
Findings
Achieves $ ext{O}( rac{1}{ ext{Ξ΅}^2})$ mean squared error with reduced computational cost
Reduces computational complexity from $ ext{O}( rac{1}{ ext{Ξ΅}^3})$ to $ ext{O}( rac{1}{ ext{Ξ΅}^2} ext{log}( ext{Ξ΅})^2)
Validated effectiveness through numerical examples
Abstract
We consider a class of finite time horizon nonlinear stochastic optimal control problem, where the control acts additively on the dynamics and the control cost is quadratic. This framework is flexible and has found applications in many domains. Although the optimal control admits a path integral representation for this class of control problems, efficient computation of the associated path integrals remains a challenging Monte Carlo task. The focus of this article is to propose a new Monte Carlo approach that significantly improves upon existing methodology. Our proposed methodology first tackles the issue of exponential growth in variance with the time horizon by casting optimal control estimation as a smoothing problem for a state space model associated with the control problem, and applying smoothing algorithms based on particle Markov chain Monte Carlo. To further reduceβ¦
Click any figure to enlarge with its caption.
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Abstract
We consider a class of finite time horizon nonlinear stochastic optimal control problem, where the control acts additively on the dynamics and the control cost is quadratic. This framework is flexible and has found applications in many domains. Although the optimal control admits a path integral representation for this class of control problems, efficient computation of the associated path integrals remains a challenging Monte Carlo task. The focus of this article is to propose a new Monte Carlo approach that significantly improves upon existing methodology. Our proposed methodology first tackles the issue of exponential growth in variance with the time horizon by casting optimal control estimation as a smoothing problem for a state space model associated with the control problem, and applying smoothing algorithms based on particle Markov chain Monte Carlo. To further reduce computational cost, we then develop a multilevel Monte Carlo method which allows us to obtain an estimator of the optimal control with mean squared error with a computational cost of . In contrast, a computational cost of is required for existing methodology to achieve the same mean squared error. Our approach is illustrated on two numerical examples, which validate our theory.
Key words: Optimal Control; Multilevel Monte Carlo; Markov chain Monte Carlo, Sequential Monte Carlo.
A Multilevel Approach for Stochastic Nonlinear Optimal Control
BY AJAY JASRA1, JEREMY HENG2, YAXIAN XU3 & ADRIAN N. BISHOP4
1,3Department of Statistics & Applied Probability, National University of Singapore, Singapore, 117546, SG. E-Mail:β [email protected], [email protected]
2ESSEC Business School, 5 Nepal Park, Singapore 139408, SG. E-Mail:β [email protected]
4University of Technology Sydney, AUS. E-Mail:β [email protected]
1 Introduction
We consider a class of finite time horizon nonlinear stochastic optimal control problem, where the control acts additively on the dynamics and the control cost is quadratic [14, 15]. This framework is flexible and has applications in domains such as robotics [23], epidemiology [26, 28], reinforcement learning [23], and nonlinear particle smoothing [17, 22]. For this class of control problems, the nonlinear Hamilton-Jacobi-Bellman equation can be reduced to a linear equation by applying a suitable logarithmic transformation. Although this allows the optimal control to admit a closed form expression via the Feynman-Kac formula, efficient computation of the associated path integrals remains a challenging Monte Carlo task. Simple approaches based on simulating the uncontrolled dynamics and performing normalized importance sampling [4] often suffer from exponential growth in variance with the time horizon. Hence to accurately estimate the optimal control, one might require an exponentially commensurate number of Monte Carlo simulations. As optimal importance sampling, i.e. zero variance estimation of the optimal control, is achieved when one simulates from the optimally controlled dynamics [25, Theorem 2], this prompts an iterative procedure [24, 25, 21] to estimate optimal control. Although these iterative importance control methods can often give substantial variance reduction, they typically require parameterizing the form of the control. The focus of this article is to propose a new Monte Carlo approach to the above path integral control problem. Our proposed methodology first tackles the issue of exponential growth in variance with the time horizon by casting optimal control estimation as a smoothing problem for a state space model associated with the control problem, and applying state-of-the-art smoothing algorithms based on particle Markov chain Monte Carlo (MCMC) [1].
To further reduce computational cost, we then consider the multilevel Monte Carlo (MLMC) method [8, 9, 10] which is particularly well-suited to the problem at hand as path integrals are expectations w.r.t.Β a continuum model, defined by the probability law of the uncontrolled stochastic differential equation (SDE). For numerical implementation to be tractable, one must typically resort to discretizing the continuum model (for instance using Euler discretizations of SDEs) and considering expectations w.r.t.Β the discretized model. As the time discretization becomes more precise, the approximation becomes more accurate, but simultaneously, the associated cost of computation increases. The MLMC method considers a seemingly trivial telescoping sum representation of the expectation w.r.t.Β the most precise discretization, where the summands are differences of expectations of increasingly coarsely discretized approximations. The key idea is to approximate the differences by sampling dependent couplings of the two probability measures in the difference, independently for each difference. In some contexts, if the couplings are appropriately constructed (see e.g.Β [8]) then relative to Monte Carlo estimation for the most precise expectation, the computational cost can be significantly reduced. In the case of optimal control estimation, one cannot adopt the MLMC method in [8] directly, as exact sampling from smoothing distributions and their couplings is intractable. To circumvent this difficulty, we adapt the ideas in [13] (see also [7]) to our context to develop a MLMC method that is based on MCMC sampling methods. We then establish that, for some given , for our proposed estimator of the optimal control to have mean squared error (MSE), the computational cost required is . In contrast, a computational cost of is required for existing methodology to achieve the same MSE. In the case where one assumes a parametric form of the control, we note that previous work in [21] have considered applying the standard MLMC method in [8] to reduce the computational cost within an iterative importance control scheme to update the control parameters.
This article is structured as follows. In Section 2, we begin by detailing the stochastic optimal control problem of interest and its path integral formulation. We then describe our proposed methodology to compute the optimal control in Section 3, and state some theoretical results on its complexity in Section 4. In Section 5, we validate our theory on two examples, including a nonlinear stochastic compartmental model for an epidemic with cost-controlled vaccination. The appendix features the assumptions and proofs for our complexity theorem in Section 4.
2 Nonlinear stochastic optimal control
We consider a nonlinear controlled process in , defined as the solution of the following SDE
[TABLE]
with initial condition . The above is to be understood in the Ito sense [2] and is a standard Brownian motion in . We assume that , and are twice differentiable and there exists a constant such that
[TABLE]
for all . Without any loss of generality, we suppose that and (which may be non-square) have full rank. Note that the latter implies existence and uniqueness of left-inverses, i.e. functions and such that and for all . Lastly, we assume that is uniformly positive definite over . For a fixed time interval , the set of admissible controls we shall consider are Borel measurable functions satisfying
[TABLE]
for all and , where denotes conditional expectations w.r.t. the law of (1) on the event . These conditions are sufficient for the existence of a unique, continuous, (strong) solution to (1); see for e.g. [2, 27].
For each and , we associate the process defined by (1) with the following cost functional
[TABLE]
where are continuous terminal and running cost functions, respectively, and is a positive-definite symmetric. We then define the value function as
[TABLE]
If there exists a unique minimizer of (3) across the entire time horizon , we will refer to it as the optimal control and denote it by .
For any suitably smooth function , we will denote its partial derivative w.r.t. the time by , its gradient w.r.t. the spatial variable by and its Hessian by . For any , we write its trace as . Under appropriate conditions, the value function (3) can be associated with the following Hamilton-Jacobi-Bellman (HJB) equation
[TABLE]
defined for , with a boundary condition at the terminal time . This association is made precise in the following.
Assumption 2.1**.**
Suppose that the value function defined in (3) is once continuously differentiable in the time variable, twice continuously differentiable the spatial variable, and is a (classical) solution to the HJB equation (4).
Sufficient conditions for this assumption to hold, in addition to the modeling hypotheses introduced thus far, can be found for e.g. in [6]. These conditions typically take the form of further regularity or boundedness assumptions on the system (1) and cost functions in (2) and/or their derivatives and are rather standard111It is noteworthy that while a classical solution to the HJB equation arising in deterministic optimal control is not typical, it is well-known [18, 6, 19, 27] that the stochastic optimal control problem is quite generally βmore regularβ..
Under Assumption 2.1, the value-to-go satisfies the HJB equation
[TABLE]
for , with a boundary condition . From (4), we find that the corresponding optimal control is given by
[TABLE]
for . Substituting the form of the optimal control back into the HJB equation (5) gives
[TABLE]
which is a nonlinear partial differential equation defined on . However, the latter can be simplified by considering a logarithmic transformation of the value function
[TABLE]
for and some satisfying the following assumption.
Assumption 2.2**.**
Suppose that there exists such that for all .
This assumption222The interpretation of this relationship is that along directions where the noise variance is small, the control is deemed more expensive while, conversely, in those directions in which the noise has larger variance the control is cheap [14]. Indeed, this may be desirable in practice since it forces control energy to be spent mostly in those directions in which the noise level may be problematic [23]. is standard in the path integral formulation of optimal control [14], but it also appears more generally in the stochastic optimal control literature [6]. Assumption 2.2 allows us to write
[TABLE]
which is a linear partial differential equation on , with boundary condition . By the Feynman-Kac formula, the solution is given by
[TABLE]
where denotes conditional expectations w.r.t. the law of the uncontrolled process defined by
[TABLE]
with initial condition . The following result uses the relationships in (6) and (7) to deduce an expression of the optimal control.
Proposition 2.1**.**
Suppose Assumptions 2.1 and 2.2, and the modeling hypotheses hold. We have for
[TABLE]
where expectations are path integrals defined by the uncontrolled SDE (8) with initial condition .
Proof.
This result appears in [25] with and it is straightforward to generalize. β
The controller form in Proposition 2.1 (and variations of such) is often referred to as the path integral formulation of optimal control [14].
3 Computation of optimal control
To simplify notation, throughout this section, we will set the terminal time as and consider estimating the optimal control at time .
3.1 Standard approach
From Proposition 2.1, the objective is to compute for small
[TABLE]
We note that (10) neglects the additional bias incurred by truncating and refer the reader to [4] for a discussion on the impact of this parameter. To numerically approximate (10), the standard approach to path integral control would rely on a sufficiently precise time discretization of the model. For a sufficiently large , we consider the Euler-Maruyama discretization of (8) with step size
[TABLE]
with initial condition , where denote independent Brownian increments that are distributed according to a Gaussian distribution mean zero and covariance . By taking for , this prompts the following time discretization of (10)
[TABLE]
where denotes conditional expectations w.r.t. the law of (11) with initial condition .
To approximate the expectations in (12), we can simulate trajectories and consider the standard Monte Carlo approximation
[TABLE]
Noting that we can write with normalized weights
[TABLE]
and test function
[TABLE]
it follows that (13) can be seen as the normalized importance sampling estimator of , where denotes expectation w.r.t. the distribution
[TABLE]
defined on , equipped with the Borel -algebra . The notation denotes the law of (11) with initial condition ,
[TABLE]
and is the normalizing constant of (14).
Although the estimator (13) is straightforward to implement and amenable to parallel computation, its variance will often grow exponentially with the time horizon (taken as in our notation), and particularly so when is not sufficiently large [11]. The following section presents an alternative Monte Carlo approach that circumvents this difficulty.
3.2 Smoothing approach
Our proposed methodology is based on the observation that (14) can be seen as the smoothing distribution of a state space model with (11) as the latent Markov process and (15) as the observation densities. This connection between optimal control and smoothing has been previously noted in [5, 16]. This perspective explains why the variance of (13) is often large, and allows us to proposed better optimal control estimators by exploiting state-of the-art smoothing algorithms based on sequential Monte Carlo (SMC) methods.
A basic SMC method known as the bootstrap particle filter is detailed in Algorithm 1, where refers to a resampling operation based on a vector of nonnegative unnormalized weights . For example, this is the categorical distribution on with probabilities , when multinomial resampling is employed; other lower variance and adaptive resampling schemes can also be considered. The algorithm requires specifying the number of particles as input, which determines the accuracy and cost of the approximation, and outputs an approximate sample from (14) and an unbiased estimator of its normalizing constant.
We now consider a particular smoothing algorithm known as the particle independent Metropolis-Hastings (PIMH) method, which uses Algorithm 1 as a building block to design a MCMC method to sample from (14); see [1] for additional details and other alternatives. An algorithmic description of PIMH is given in Algorithm 2. From the Markov chain generated by PIMH, one obtains a consistent estimator of the optimal control for the time discretized model (12)
[TABLE]
as the number of iterations , for any number of particles . The latter also impacts the quality of the approximation: since the normalizing constant estimator given by Algorithm 1 is consistent as , the acceptance probability in Step 2(b) would be close to one if is large.
3.3 Multilevel approach
To further improve the computational efficiency of (16), we now leverage upon the MLMC method [8, 9, 10] by considering a hierarchy of time discretizations with time steps for . The multilevel approach is based on the following telescopic sum
[TABLE]
where refers to the distribution defined in (14) with time step . The first term in the sum can be approximated using the methodology described in Section 3.2, i.e. we employ Algorithm 2 with iterations to obtain a Markov chain , and return the estimator
[TABLE]
In standard MLMC [8], the summands
[TABLE]
are then estimated independently, for , by sampling from appropriately constructed couplings of and that induce sufficient correlation to reduce the computational cost of just approximating in isolation. However, in our context, exact sampling from these smoothing distributions and their couplings is not feasible. In the following, we will adapt the ideas in [13] to our setup to approximate (19).
We now construct a smoothing distribution
[TABLE]
defined on the product space , equipped with the product -algebra , that would allow us to couple our approximation of the smoothing distributions at time discretization levels and . This corresponds to a new state space model: denotes the law of a latent process that evolves according to
[TABLE]
with initial condition ,
[TABLE]
where the observation densities given by
[TABLE]
and is the normalization constant. In (21), the Brownian increments and at levels and , respectively, are coupled by independently sampling for and setting for . To approximate (19), we will rely on the identity
[TABLE]
[TABLE]
where denotes expectation w.r.t. and
[TABLE]
The choice of observation densities in (22) is such that the Radon-Nikodym derivatives (24) are upper-bounded by a finite constant that does not depend on the level . This ensures that the change of measure approach in (23) would not introduce too much variance relative to exact sampling from a dependent coupling of and .
Since (20) is just another smoothing distribution on an extended state space, we can also employ the methodology of Section 3.2 to construct a MCMC method to sample from it. The resulting algorithm based on the PIMH method is detailed in Algorithm 4. Like before, this uses a SMC method on model (20), described in Algorithm 3, as a building block. Given the Markov chain generated by Algorithm 4, we approximate (23) using the estimator
[TABLE]
[TABLE]
Using the above approach independently for levels and (18) for level gives the following multilevel estimator of the optimal control (10)
[TABLE]
It follows under mild assumptions that (26) is a consistent estimator. In the next section, we will establish, under appropriate assumptions, that the multilevel estimator has a reduced computational cost compared to the single level estimator (16).
4 Theoretical results
In our context, to sample for (resp.Β ) we will generate a Markov chain on a potentially enlarged space with the -algebra (resp.Β ). The purpose of this construction is to facilitate the application of advanced Markov chain samplers such as in [1]. We denote the associated Markov kernel as , .
We will consider studying (component-wise)
[TABLE]
where denotes an expectation w.r.t.Β the Markov chains that have been simulated to estimate . Our objective is to verify that the MSE (27) can be made, for given, of , with a cost that is smaller than using a single MCMC algorithm to approximate .
The assumptions for the following result are in Appendix A.1 and the proof is in Appendix A.2.
Theorem 4.1**.**
Assume (A(A1)-(A2)). Then there exists a such that for any , :
[TABLE]
where , is the indicator function.
To understand the significance of this result, suppose that one iteration of each Markov chain costs . The latter is the cost of sampling Euler approximations (note Remark A.1 in the appendix implies that the cost per iteration is possible). Let be given and set , so that the square bias is (see (36) in the appendix, but this is the term on the R.H.S.Β of (28)). Now choosing as in [8] yields
[TABLE]
So, we have achieved a MSE of at most . The cost to achieve this MSE is
[TABLE]
If one simply used a Markov chain simulation for or using the original approach in [4] the cost to achieve this same MSE would be ; a significant increase.
5 Numerical results
5.1 Linear quadratic Gaussian control
We consider a linear quadratic Gaussian (LQG) control problem where the underlying continuous-time linear controlled process is given by
[TABLE]
with , , and denoting a standard Brownian motion in . The estimation of the optimal control that minimizes the following quadratic cost function is considered
[TABLE]
In our experiments, we set , , , , , and initialize the process at . The estimation of the optimal control at time using the different methods considered is compared: the standard Monte Carlo (MC) approach described in Section 3.1 (see also [3, 15]), single level PIMH introduced Section 3.2, and multilevel Monte Carlo (MLMC) approach based on PIMH as detailed in Section 3.3. For the MLMC method, we set the number of MCMC samples according to the multilevel analysis, i.e.Β and , with the number of particles in SMC to be fixed at .
For the single level PIMH method, the number of MCMC samples is . The single level PIMH algorithm and the multilevel PIMH algorithm are compared for a time horizon of length , with the MSE v.s. cost plot presented in Figure 1. For the same level of MSE, the cost reduction in the multilevel approach is clear. All three approaches are then compared for a longer time horizon of length . The number of samples is set so that the computational cost of the standard MC approach and that of the PIMH method are the same. From Figure 2, we see a marked reduction in computational cost when employing single level PIMH as compared to the standard MC approach, and a further reduction in cost with the multilevel PIMH approach.
5.2 Nonlinear compartmental model
5.2.1 Model specification
We consider the optimal control of a stochastic compartmental model for an epidemic with cost-controlled vaccination. The state variables are corresponding to susceptible (S), infected (I), vaccinated (V), and removed (R) individuals in a population with the constraint
[TABLE]
for all . The controlled model considered herein respects this constraint for all , whenever it is enforced to hold at time [math]; see [26]. We consider a modification of [26] suitable for our purposes,
[TABLE]
where denotes a standard Brownian motion in . In this model, the birth and death rate are given by , and the infection rate is controlled by , known as the contact rate. The parameter controls the curing rate, controls the loss of vaccine effectiveness, and controls the efficacy of the vaccination protocol, i.e. letting would imply the vaccine is perfectly effective, while implies the vaccination has no effect. The parameter is necessary for our model to be well-defined and taken small enough so that it has no effect qualitatively.
The control input specifies the fraction of the susceptible class being vaccinated at any moment. Note that although we would like , values outside this constraint cause no mathematical difficulty and do not pose a problem for satisfaction of the constraint (31), i.e. . Given some fixed terminal time , the cost function we aim to minimize is given by
[TABLE]
for , where are weighting parameters. Note that the running cost is linear in the state.
The basic reproduction rate is and, with , it is shown in [28] that for the stochastic system (32) is almost surely exponentially stable with any constant to the equilibrium
[TABLE]
Hence, we take parameters such that going forward. In particular, we take , , and . We also take , and . The initial condition is . To check our assumptions stated in Section 2, we note that
[TABLE]
for some . Indeed, we require in order to find uniquely. Given our prior parameters this implies and then . We only need the left inverse of . Note in the control computation we can act as if the system is three-dimensional and ignore the dynamics of . The left inverse in this case is
[TABLE]
which in general exists since the inverse on the right hand side in general exists.
5.2.2 Numerical results
In our experiments, we set , , and . In the following, we will compare the single level PIMH algorithm to the multilevel PIMH algorithm for the task of optimal control estimation. As the standard MC approach did not perform well in the simple LQG model, it is not considered for this application. A trajectory of the controlled process generated by the two algorithms are shown in Figure 3. It is clear from Figure 3 that the infected and the susceptible compartments are decreasing over time, while the vaccinated and the removed compartments are increasing over time.
In Figure 4, we compare the two algorithms at a fixed computational cost by reporting the sample average (left panel) and sample variance (right panel) of the value function, computed using independent repetitions of each algorithm. We observe that the multilevel approach achieves lower values on average in terms of the objective (33) with much smaller variance.
Lastly, for the task of estimating the optimal control at time , we present a MSE v.s.Β cost plot in Figure 5. For the MLMC method, we set the number of MCMC samples according to the multilevel analysis i.e.Β and , with the number of particles in SMC to be fixed at . For the singe level PIMH method, the number of MCMC samples is taken as . The true value is computed by running the latter algorithm at one plus the most precise level (i.e.Β ) and the MSE is computed using independent repetitions of each algorithm. The results illustrate that the multilevel approach offers significant reduction in computational cost.
Acknowledgements
AJ & YX were supported by an AcRF tier 2 grant: R-155-000-161-112. AJ is affiliated with the Risk Management Institute, the Center for Quantitative Finance and the OR & Analytics cluster at NUS. AJ was supported by a KAUST CRG4 grant ref: 2584.
Appendix A Technical results
Throughout our proofs is a finite constant that does not depend upon and whose value may change on each appearance.
A.1 Assumptions
In the context of Theorem 4.1, to shorten our proofs, we will suppose that the Markov chain(s) are started in stationarity. This latter assumption can be removed with some work, but is unnecessary in order to convey the point of our approach. Note also that we are assuming that is fixed throughout and there is an additional bias which is not addressed. In order to derive our theoretical results, in addition to the assumptions that have already been made, we will make the following assumptions. Below denotes the collection of probability measures on .
- (A1)
, , , , are all bounded and measurable. In addition , are Lipschitz. Set , then we assume each element of is bounded and Lipschitz.
- (A2)
is reversible. There exists a such that for each there exists a such that for any bounded, measurable and Lipschitz,
[TABLE]
Remark A.1**.**
In (A(A2)) we have assumed a mixing rate that will be independent of . At first glance, it may seem that this is not possible in practice. However, using Lemma A.1 below, one can easily establish that (for example) the PIMH algorithm in [1] has such a property.
A.2 Proof of Theorem 4.1
The proof is constructed by using several results, which are first quoted and proved later on in the appendix.
Theorem A.1**.**
Assume (A(A1)-(A2)). Then there exists a such that for any , :
[TABLE]
Proof.
This essentially the same as [13, Theorem 3.1] except that one needs Proposition A.2 and Lemma A.1 in Appendix A.3. β
The proof of the following result is in Appendix A.4.
Proposition A.1**.**
Assume (A(A1)-(A2)). Then there exists a such that for any , :
[TABLE]
Remark A.2**.**
We note that, via [13, Proposition A.1.], a simple decomposition:
[TABLE]
for any , , and Lemma A.1 one can show that
[TABLE]
for independent of . In addtion, in the proof of Proposition A.2 we have established that
[TABLE]
where does not depend on . (36) can be obtained using the bound for (39) (), (34) and Lemma A.1.
Proof of Theorem 4.1.
Using the inequality,
[TABLE]
[TABLE]
First, apply the inequality to the variance term (the bias is the L.H.S.Β of (36) and hence the variance is the left term on the R.H.S.Β of (37)), splitting and the other terms. Second, apply Theorem A.1, Proposition A.1 and (35) to the variance terms and (36) to the bias term. This allows one to complete the proof. β
A.3 Proofs for Theorem A.1
Below for we write the element of vector as . For , we write the element of as . For ease of notation set
[TABLE]
[TABLE]
Proposition A.2**.**
Assume (A(A1)). Then for any , , there exist a such that for any
[TABLE]
Proof.
We give the proof for , the proof for the case follows by Jensenβs inequality.
Let denote the normalizing constant of . By Lemma A.1 it easily follows that
[TABLE]
Thus it follows that
[TABLE]
[TABLE]
Define
[TABLE]
where we are denoting expectations w.r.t.Β the law of the diffusion (8) as and is the element of the Brownian motion in (8). Then it is clear that the integral on the R.H.S.Β of (38) is upper-bounded by
[TABLE]
Hence we focus upon to conclude our result.
We have
[TABLE]
[TABLE]
We deal with the two expectations on the R.H.S.Β of (39) individually. For the first term on the R.H.S.Β we have that
[TABLE]
[TABLE]
[TABLE]
Application of Lemmata A.2-A.3 yields the upper-bound of . For the second term on the R.H.S.Β of (39), one can use Lemma A.4. Hence the proof is completed. β
Lemma A.1**.**
Assume (A(A1)). Then there exists a such that for any ,
[TABLE]
and for any ,
[TABLE]
Proof.
Throughout are finite constants that do not depend upon and whose value may change on each appearance. We note that for any , any and any
[TABLE]
Clearly for any , hence it follows that for any ,
[TABLE]
The second result is established using the relationship between and . β
Recall we are denoting expectations w.r.t.Β the law of the diffusion (8) as and is the element of the Brownian motion in (8).
Lemma A.2**.**
Assume (A(A1)). Then for any there exist a such that for any
[TABLE]
Proof.
By Lemma A.1 we need only deal with
[TABLE]
This latter term, via the inequality is upper-bounded by
[TABLE]
[TABLE]
We treat these two terms independently, calling them and respectively.
Term: . By repeated use of the inequality, is upper-bounded by
[TABLE]
[TABLE]
[TABLE]
where we have used the Lipschitz property of to go-to the last line. Splitting the expectation of the summand using Cauchy Schwarz we have
[TABLE]
Then using standard results from Euler-discretization of diffusions (see e.g.Β [20]) and Gaussian distributions:
[TABLE]
hence
[TABLE]
Term: . By repeated use of the inequality, is upper-bounded by
[TABLE]
[TABLE]
Using the Ito isometry formula, clearly
[TABLE]
Then using the Lipschitz property of and then standard results for diffusion processes
[TABLE]
The proof is hence completed by the above arguments. β
Lemma A.3**.**
Assume (A(A1)). Then for any there exist a such that for any
[TABLE]
Proof.
We begin by splitting the expectation via Cauchy Schwarz to yield the upper-bound
[TABLE]
By (a classic variation of) the Burkholder-Gundy-Davis inequality, it follows that
[TABLE]
hence we consider the right-hand expectation in (40). Using the Lipschitz property of for bounded , it follows that
[TABLE]
[TABLE]
Using the Lipschitz property of and it clearly follows that
[TABLE]
Hence using standard results for Euler approximations of diffusion processes
[TABLE]
and one can conclude the proof by the above arguments. β
Lemma A.4**.**
Assume (A(A1)). Then for any there exist a such that for any
[TABLE]
[TABLE]
Proof.
As in the proof of Lemma A.3 split the expectation via Cauchy Schwarz to yield the upper-bound
[TABLE]
Again (as in the proof of Lemma A.3) by the Burkholder-Gundy-Davis inequality the first expectation is so we concentrate upon the second expectation (call it ). Using the Lipschitz property of for bounded , it follows that
[TABLE]
Then via the Minkowski inequality
[TABLE]
where we have used the fact that is bounded. Then applying Jensen with standard results in diffusions, yields β
A.4 Proof of Proposition A.1
Proof of Proposition A.1.
We make the defintions:
[TABLE]
Then component-wise (from here, our calculations should be considered component-by-component):
[TABLE]
By [12, Lemma D.5]
[TABLE]
[TABLE]
[TABLE]
The first two terms on the R.H.S.Β and the last four terms on the R.H.S.Β can be treated using similar calculations, so we only consider
[TABLE]
Term: . We have
[TABLE]
On taking expectations w.r.t.Β the law of the simulated Markov chain, we have
[TABLE]
as the chain is started in stationarity. Applying the Cauchy-Schwarz inequality:
[TABLE]
By using a similar result to [13, Proposition A.1] and by Proposition A.2
[TABLE]
By standard results for uniformly ergodic Markov chains (e.g.Β the result in [13, Proposition A.1]), along with Lemma A.1
[TABLE]
and hence
[TABLE]
Term: . We have
[TABLE]
and taking expectations as for
[TABLE]
Hence, by Cauchy-Schwarz
[TABLE]
By Proposition A.2 and again, standard results for uniformly ergodic Markov chains, along with Lemma A.1
[TABLE]
and hence
[TABLE]
β
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] Andrieu , C., Doucet , A. & Holenstein , R. (2010). Particle Markov chain Monte Carlo methods (with discussion). J. R. Statist. Soc. Ser. B , 72 , 269β342.
- 2[2] Arnold , L. (1974). Stochastic Differential Equations: Theory and Applications . Wiley: New York.
- 3[3] Bertoli , F. & Bishop , A. N. (2018). Nonlinear stochastic receding horizon control: Stability, robustness and Monte Carlo methods for control approximation. Intβl J. Cont. , 91 , 2387β2402.
- 4[4] Bertoli , F. & Bishop , A. N. (2018). An error analysis in the limit approximation in path integral control. Technical Report.
- 5[5] Fleming , W. H. & Mitter , S. K. (1982). Optimal control and nonlinear filtering for nondegenerate diffusion processes. Stochastics , 8 (1), 63β77.
- 6[6] Fleming , W. H. & Soner , H. M. (2006). Controlled Markov Processes and Viscosity Solutions . 2nd edition, Springer: New York.
- 7[7] Franks , J., Jasra , A., Law , K. J. H. & Vihola , M. (2018). Unbiased inference for discretely observed hidden Markov model diffusions. ar Xiv:1807.10259.
- 8[8] Giles , M. B. (2008). Multilevel Monte Carlo path simulation. Op. Res. , 56 , 607β617.