On Entropy Regularized Path Integral Control for Trajectory Optimization

TL;DR

This paper extends Path Integral Control to a broader class of stochastic optimal control problems by introducing entropy regularization, analyzing convergence, and deriving explicit updates for improved trajectory optimization.

Contribution

It formulates Entropy Regularized Trajectory Optimization, broadening PIC applicability, and connects it with RL and stochastic search methods, providing theoretical insights and explicit algorithms.

Findings

Proposed a less restrictive class of SOC problems with entropy regularization.

Analyzed convergence behavior of the state trajectory distribution sequence.

Derived explicit update rules for the entropy regularized PIC.

Abstract

In this article we present a generalised view on Path Integral Control (PIC) methods. PIC refers to a particular class of policy search methods that are closely tied to the setting of Linearly Solvable Optimal Control (LSOC), a restricted subclass of nonlinear Stochastic Optimal Control (SOC) problems. This class is unique in the sense that it can be solved explicitly to yield a formal optimal state trajectory distribution. In this contribution we first review the PIC theory and discuss related algorithms tailored to policy search in general. We are able to identify a generic design strategy that relies on the existence of an optimal state trajectory distribution and finds a parametric policy by minimizing the cross entropy between the optimal and a state trajectory distribution parametrized through its policy. Inspired by this observation we then aim to formulate a SOC problem that shares traits with the LSOC setting yet that covers a less restrictive class of problem formulations. We refer to this SOC problem as Entropy Regularized Trajectory Optimization. The problem is closely related to the Entropy Regularized Stochastic Optimal Control setting which is lately often addressed by the Reinforcement Learning (RL) community. We analyse the theoretical convergence behaviour of the theoretical state trajectory distribution sequence and draw connections with stochastic search methods tailored to classic optimization problems. Finally we derive explicit updates and compare the implied Entropy Regularized PIC with earlier work in the context of both PIC and RL for derivative-free trajectory optimization.