Actor-Critic Method for High Dimensional Static Hamilton--Jacobi--Bellman Partial Differential Equations based on Neural Networks

TL;DR

This paper introduces a neural network-based actor-critic method for solving high-dimensional Hamilton-Jacobi-Bellman PDEs, demonstrating effectiveness on problems up to 20 dimensions.

Contribution

It develops a novel neural network framework combining policy gradient and variance-reduced value function estimation for high-dimensional HJB PDEs.

Findings

Successfully solves PDEs in up to 20 dimensions

Achieves accurate results for linear and nonlinear PDEs

Employs adaptive step size for improved boundary accuracy

Abstract

We propose a novel numerical method for high dimensional Hamilton--Jacobi--Bellman (HJB) type elliptic partial differential equations (PDEs). The HJB PDEs, reformulated as optimal control problems, are tackled by the actor-critic framework inspired by reinforcement learning, based on neural network parametrization of the value and control functions. Within the actor-critic framework, we employ a policy gradient approach to improve the control, while for the value function, we derive a variance reduced least-squares temporal difference method using stochastic calculus. To numerically discretize the stochastic control problem, we employ an adaptive step size scheme to improve the accuracy near the domain boundary. Numerical examples up to $20$ spatial dimensions including the linear quadratic regulators, the stochastic Van der Pol oscillators, the diffusive Eikonal equations, and fully nonlinear elliptic PDEs derived from a regulator problem are presented to validate the effectiveness of our proposed method.