Approximating the Stationary Bellman Equation by Hierarchical Tensor Products

TL;DR

This paper introduces a hierarchical tensor product approach to numerically solve high-dimensional stationary Hamilton-Jacobi-Bellman equations arising from optimal control of discretized nonlinear PDEs, overcoming the curse of dimensionality.

Contribution

It develops a novel method combining tensor train approximations and minimal residual techniques to efficiently solve high-dimensional linearized HJB equations.

Findings

Successfully applied to viscous Burgers and diffusion equations with unstable reactions.

Demonstrates the effectiveness of tensor formats in high-dimensional PDEs.

Provides numerical evidence of overcoming computational challenges in high dimensions.

Abstract

We treat infinite horizon optimal control problems by solving the associated stationary Hamilton-Jacobi-Bellman (HJB) equation numerically to compute the value function and an optimal feedback law. The dynamical systems under consideration are spatial discretizations of non linear parabolic partial differential equations (PDE), which means that the HJB is non linear and suffers from the curse of dimensionality. Its non linearity is handled by the Policy Iteration algorithm, where the problem is reduced to a sequence of linear, hyperbolic PDEs. These equations remain the computational bottleneck due to their high dimensions. By the method of characteristics these linearized HJB equations can be reformulated via the Koopman operator in the spirit of dynamic programming. The resulting operator equations are solved using a minimal residual method. To overcome numerical infeasability we use low rank hierarchical tensor product approximation/tree-based tensor formats, in particular tensor trains (TT tensors), and multi-polynomials, together with high dimensional quadrature, e.g. Monte-Carlo. By controlling a destabilized version of viscous Burgers and a diffusion equation with unstable reaction term numerical evidences are given.