Risk Sensitive Path Integral Control for Infinite Horizon Problem Formulations

TL;DR

This paper investigates whether the linearity property of the Hamilton-Jacobi-Bellman equation in path integral control can be extended to infinite horizon stochastic optimal control problems, with implications for stability and reinforcement learning.

Contribution

It demonstrates that the HJB equation can be linearized for certain infinite horizon formulations, providing new insights into control stability and RL applications.

Findings

Discounted cost formulation is intractable.

Average cost formulation is tractable.

Linear HJB may facilitate control solutions and RL methods.

Abstract

Path Integral Control methods were developed for stochastic optimal control covering a wide class of finite horizon formulations with control affine nonlinear dynamics. Characteristic for this class is that the HJB equation is linear and consequently the value function can be expressed as a conditional expectation of the exponentially weighted cost-to-go evaluated over trajectories with uncontrolled system dynamics, hence the name. Subsequently it was shown that under the same assumptions Path Integral Control generalises to finite horizon risk sensitive stochastic optimal control problems. Here we study whether the HJB of infinite horizon formulations can be made linear as well. Our interest in infinite horizon formulations is motivated by the stationarity of the associated value function and their inherent dynamic stability seeking nature. Technically a stationary value function may ease the solution of the associated linear HJB. Second we argue this may offer an interesting starting point for off-linear Reinforcement Learning applications. We show formally that the discounted and average cost formulations are respectively intractable and tractable.