Batch Sample-wise Stochastic Optimal Control via Stochastic Maximum Principle

TL;DR

This paper introduces an improved stochastic optimal control method using sample-wise stochastic maximum principle, higher order schemes, and a damped contraction algorithm, enhancing convergence and accuracy in control estimation.

Contribution

It proposes a novel batch sample-wise approach with higher order schemes and a damped contraction algorithm for stochastic optimal control, improving convergence rates and control accuracy.

Findings

Enhanced convergence rate from (rac{N}{K} + rac{1}{N}) to (rac{1}{K} + rac{1}{N^2})

Numerical validation of first order convergence rate

Potential for efficient control in practical problems with neural networks

Abstract

In this work, we study the stochastic optimal control problem (SOC) mainly from the probabilistic view point, i.e. via the Stochastic Maximum principle (SMP) \cite{Peng4}. We adopt the sample-wise backpropagation scheme proposed in \cite{Hui1} to solve the SOC problem under the strong convexity assumption. Importantly, in the Stochastic Gradient Descent (SGD) procedure, we use batch samples with higher order scheme in the forward SDE to improve the convergence rate in \cite{Hui1} from $\sim \mathcal{O}(\sqrt{\frac{N}{K} + \frac{1}{N}})$ to $\sim \mathcal{O}(\sqrt{\frac{1}{K} + \frac{1}{N^2}})$ and note that the main source of uncertainty originates from the scheme for the simulation of $Z$ term in the BSDE. In the meantime, we note the SGD procedure uses only the necessary condition of the SMP, while the batch simulation of the approximating solution of BSDEs allows one to obtain a more accurate estimate of the control $u$ that minimizes the Hamiltonian. We then propose a damped contraction algorithm to solve the SOC problem whose proof of convergence for a special case is attained under some appropriate assumption. We then show numerical results to check the first order convergence rate of the projection algorithm and analyze the convergence behavior of the damped contraction algorithm. Lastly, we briefly discuss how to incorporate the proposed scheme in solving practical problems especially when the Randomized Neural Networks are used. We note that in this special case, the error backward propagation can be avoided and parameter update can be achieved via purely algebraic computation (vector algebra) which will potentially improve the efficiency of the whole training procedure. Such idea will require further exploration and we will leave it as our future work.