Learning Optimal Control via Forward and Backward Stochastic Differential Equations

TL;DR

This paper introduces a sampling-based numerical method leveraging forward and backward stochastic differential equations to solve nonlinear stochastic optimal control problems efficiently, without needing an initial guess.

Contribution

The paper develops a novel iterative scheme using Girsanov's theorem and importance sampling to improve the efficiency of solving nonlinear stochastic control problems via FBSDEs.

Findings

The proposed method accurately solves complex nonlinear control problems.

The iterative scheme improves convergence and efficiency.

Simulations validate the effectiveness of the approach.

Abstract

In this paper we present a novel sampling-based numerical scheme designed to solve a certain class of stochastic optimal control problems, utilizing forward and backward stochastic differential equations (FBSDEs). By means of a nonlinear version of the Feynman-Kac lemma, we obtain a probabilistic representation of the solution to the nonlinear Hamilton-Jacobi-Bellman equation, expressed in the form of a decoupled system of FBSDEs. This system of FBSDEs can then be simulated by employing linear regression techniques. To enhance the efficiency of the proposed scheme when treating more complex nonlinear systems, we then derive an iterative modification based on Girsanov's theorem on the change of measure, which features importance sampling. The modified scheme is capable of learning the optimal control without requiring an initial guess. We present simulations that validate the algorithm and demonstrate its efficiency in treating nonlinear dynamics.