Physics-informed approach for exploratory Hamilton--Jacobi--Bellman equations via policy iterations

TL;DR

This paper introduces a mesh-free, physics-informed neural network framework for solving complex stochastic control problems, demonstrating high accuracy and scalability in high-dimensional and nonlinear systems without spatial discretization.

Contribution

It develops a novel mesh-free policy iteration method using PINNs for entropy-regularized stochastic control, with detailed error analysis and validation on high-dimensional and nonlinear benchmarks.

Findings

Effective in high-dimensional control tasks (5D and 10D).

Achieves high accuracy and robustness across benchmarks.

No spatial discretization needed, enabling scalability.

Abstract

We propose a mesh-free policy iteration framework based on physics-informed neural networks (PINNs) for solving entropy-regularized stochastic control problems. The method iteratively alternates between soft policy evaluation and improvement using automatic differentiation and neural approximation, without relying on spatial discretization. We present a detailed $L^2$ error analysis that decomposes the total approximation error into three sources: iteration error, policy network error, and PDE residual error. The proposed algorithm is validated with a range of challenging control tasks, including high-dimensional linear-quadratic regulation in 5D and 10D, as well as nonlinear systems such as pendulum and cartpole problems. Numerical results confirm the scalability, accuracy, and robustness of our approach across both linear and nonlinear benchmarks.