A Convergence-Guaranteed Algorithm for Stochastic Optimal Control Problems

TL;DR

This paper introduces a new algorithm for stochastic optimal control that avoids complex backward equations by using Malliavin calculus, reducing dimensionality issues and maintaining competitive performance.

Contribution

It presents a novel convergence-guaranteed iterative algorithm based on Malliavin calculus, eliminating the need for adjoint processes in stochastic control.

Findings

Reduces complexity compared to traditional methods

Alleviates the curse of dimensionality

Maintains competitive performance in empirical tests

Abstract

Stochastic Optimal Control Problems (SOCPs) plays a major role in the sequential decision-making challenges. There exist various iterative algorithms, under framework of stochastic maximum principle, that sequentially find the optimal control decision. However, they are based on the adjoint sensitivity analysis that necessitates simulation of an adjoint process, typically a backward stochastic differential equation (SDE) that must simultaneously be adapted to a forward filtration and satisfy a terminal condition, which substantially increases complexity and exacerbates the curse of dimensionality. We instead develop a stochastic maximum principle based on the Malliavin calculus, which enables us to devise an iterative algorithm without need of an adjoint process. Our algorithm however needs the Malliavin derivative that can be efficiently computed based on a forward simulator. Empirical comparisons against standard iterative algorithms demonstrate that our approach alleviates the dimensionality bottleneck while delivering competitive performance on the considered SOCPs.