Hamilton-Jacobi-Bellman Equations for Maximum Entropy Optimal Control

TL;DR

This paper extends maximum entropy reinforcement learning to continuous-time deterministic control problems by deriving novel Hamilton-Jacobi-Bellman equations, providing computationally tractable solutions and characterizing optimal controls explicitly.

Contribution

It introduces a new class of HJB equations for maximum entropy control in continuous time, proving their properties and linking to explicit solutions in special cases.

Findings

HJB equations correspond to unique viscosity solutions.

Maximum entropy formulation improves solution regularity.

Explicit solutions for linear-quadratic problems and Gaussian controls.

Abstract

Maximum entropy reinforcement learning (RL) methods have been successfully applied to a range of challenging sequential decision-making and control tasks. However, most of existing techniques are designed for discrete-time systems. As a first step toward their extension to continuous-time systems, this paper considers continuous-time deterministic optimal control problems with entropy regularization. Applying the dynamic programming principle, we derive a novel class of Hamilton-Jacobi-Bellman (HJB) equations and prove that the optimal value function of the maximum entropy control problem corresponds to the unique viscosity solution of the HJB equation. Our maximum entropy formulation is shown to enhance the regularity of the viscosity solution and to be asymptotically consistent as the effect of entropy regularization diminishes. A salient feature of the HJB equations is computational tractability. Generalized Hopf-Lax formulas can be used to solve the HJB equations in a tractable grid-free manner without the need for numerically optimizing the Hamiltonian. We further show that the optimal control is uniquely characterized as Gaussian in the case of control affine systems and that, for linear-quadratic problems, the HJB equation is reduced to a Riccati equation, which can be used to obtain an explicit expression of the optimal control. Lastly, we discuss how to extend our results to continuous-time model-free RL by taking an adaptive dynamic programming approach. To our knowledge, the resulting algorithms are the first data-driven control methods that use an information theoretic exploration mechanism in continuous time.