SOC-MartNet: A Martingale Neural Network for the Hamilton-Jacobi-Bellman Equation without Explicit inf H in Stochastic Optimal Controls

TL;DR

This paper introduces SOC-MartNet, a neural network approach that efficiently solves high-dimensional Hamilton-Jacobi-Bellman equations and stochastic optimal control problems without needing explicit Hamiltonian infimum, using a martingale formulation.

Contribution

The paper presents a novel martingale-based neural network framework for high-dimensional HJB equations that avoids explicit infimum calculation and incorporates an adversarial network for martingale enforcement.

Findings

Effective for dimensions up to 10,000

Requires fewer than 6000 training iterations

Accurately solves HJB equations and SOCPs

Abstract

In this paper, we propose a martingale-based neural network, SOC-MartNet, for solving high-dimensional Hamilton-Jacobi-Bellman (HJB) equations where no explicit expression is needed for the infimum of the Hamiltonian, $\inf_{u \in U} H(t,x,u, z,p)$, and stochastic optimal control problems (SOCPs) with controls on both drift and volatility. We reformulate the HJB equations for the value function by training two neural networks, one for the value function and one for the optimal control with the help of two stochastic processes - a Hamiltonian process and a cost process. The control and value networks are trained such that the associated Hamiltonian process is minimized to satisfy the minimum principle of a feedback SOCP, and the cost process becomes a martingale, thus, ensuring the value function network as the solution to the corresponding HJB equation. Moreover, to enforce the martingale property for the cost process, we employ an adversarial network and construct a loss function characterizing the projection property of the conditional expectation condition of the martingale. Numerical results show that the proposed SOC-MartNet is effective and efficient for solving HJB-type equations and SOCPs with a dimension up to 10,000 in a small number of iteration steps (less than 6000) of training.