On the numerical integration of the Fokker-Planck equation driven by a mechanical force and the Bismut-Elworthy-Li formula

TL;DR

This paper develops numerical methods for solving the Fokker-Planck and Hamilton-Jacobi-Bellman equations in optimal control, utilizing Girsanov theorem and the Bismut-Elworthy-Li formula, with an application to machine learning-based control.

Contribution

It introduces numerical integration techniques for key PDEs in optimal control, employing probabilistic formulas and avoiding spatial discretization.

Findings

Effective numerical schemes for Fokker-Planck and HJB equations.

Application of machine learning to optimal control without spatial discretization.

Validation of methods through an example problem.

Abstract

Optimal control theory aims to find an optimal protocol to steer a system between assigned boundary conditions while minimizing a given cost functional in finite time. Equations arising from these types of problems are often non-linear and difficult to solve numerically. In this note, we describe numerical methods of integration for two partial differential equations that commonly arise in optimal control theory: the Fokker-Planck equation driven by a mechanical potential for which we use Girsanov theorem; and the Hamilton-Jacobi-Bellman, or dynamic programming, equation for which we find the gradient of its solution using the Bismut-Elworthy-Li formula. The computation of the gradient is necessary to specify the optimal protocol. Finally, we give an example application of the numerical techniques to solving an optimal control problem without spacial discretization using machine learning.