Neural Policy Iteration for Stochastic Optimal Control: A Physics-Informed Approach

TL;DR

This paper introduces a physics-informed neural network framework for solving stochastic optimal control problems, providing error control, interpretability, and convergence guarantees, demonstrated on various benchmark problems including high-dimensional LQR.

Contribution

It extends PINN-based methods to stochastic control, offering a systematic policy iteration approach with theoretical error bounds and convergence guarantees.

Findings

Effective on stochastic cartpole and pendulum problems

Achieves high accuracy in up to 10D LQR problems

Provides explicit bounds on policy evaluation errors

Abstract

We propose a physics-informed neural network policy iteration (PINN-PI) framework for solving stochastic optimal control problems governed by second-order Hamilton--Jacobi--Bellman (HJB) equations. At each iteration, a neural network is trained to approximate the value function by minimizing the residual of a linear PDE induced by a fixed policy. This linear structure enables systematic $L^2$ error control at each policy evaluation step, and allows us to derive explicit Lipschitz-type bounds that quantify how value gradient errors propagate to the policy updates. This interpretability provides a theoretical basis for evaluating policy quality during training. Our method extends recent deterministic PINN-based approaches to stochastic settings, inheriting the global exponential convergence guarantees of classical policy iteration under mild conditions. We demonstrate the effectiveness of our method on several benchmark problems, including stochastic cartpole, pendulum problems and high-dimensional linear quadratic regulation (LQR) problems in up to 10D.