Path integral formulation of stochastic optimal control with generalized costs

TL;DR

This paper extends path integral control to handle more general cost functions by introducing an augmented cost state, enabling efficient Monte Carlo solutions for high-dimensional stochastic control problems with applications in engineering and finance.

Contribution

It introduces a new formulation with an augmented cost state allowing for stochastic integral costs and linear control costs in path integral control.

Findings

Efficient grid-free Monte Carlo implementation demonstrated.

Applicable to high-dimensional control problems where classical methods fail.

Shows usefulness in electric load management examples.

Abstract

Path integral control solves a class of stochastic optimal control problems with a Monte Carlo (MC) method for an associated Hamilton-Jacobi-Bellman (HJB) equation. The MC approach avoids the need for a global grid of the domain of the HJB equation and, therefore, path integral control is in principle applicable to control problems of moderate to large dimension. The class of problems path integral control can solve, however, is defined by requirements on the cost function, the noise covariance matrix and the control input matrix. We relax the requirements on the cost function by introducing a new state that represents an augmented running cost. In our new formulation the cost function can contain stochastic integral terms and linear control costs, which are important in applications in engineering, economics and finance. We find an efficient numerical implementation of our grid-free MC approach and demonstrate its performance and usefulness in examples from hierarchical electric load management. The dimension of one of our examples is large enough to make classical grid-based HJB solvers impractical.