Acceleration of Gradient-based Path Integral Method for Efficient Optimal and Inverse Optimal Control

TL;DR

This paper introduces an accelerated path integral method for optimal and inverse optimal control, leveraging optimization techniques like momentum to improve convergence and efficiency in control tasks and neural network training.

Contribution

It applies momentum-based optimization methods to the path integral framework, significantly enhancing convergence speed and training efficiency for control and inverse control problems.

Findings

Accelerated path integral methods improve convergence rates.

Momentum-based methods like Nesterov and Adam enhance control search efficiency.

Accelerated PI-Net trains more efficiently with less memory.

Abstract

This paper deals with a new accelerated path integral method, which iteratively searches optimal controls with a small number of iterations. This study is based on the recent observations that a path integral method for reinforcement learning can be interpreted as gradient descent. This observation also applies to an iterative path integral method for optimal control, which sets a convincing argument for utilizing various optimization methods for gradient descent, such as momentum-based acceleration, step-size adaptation and their combination. We introduce these types of methods to the path integral and demonstrate that momentum-based methods, like Nesterov Accelerated Gradient and Adam, can significantly improve the convergence rate to search for optimal controls in simulated control systems. We also demonstrate that the accelerated path integral could improve the performance on model predictive control for various vehicle navigation tasks. Finally, we represent this accelerated path integral method as a recurrent network, which is the accelerated version of the previously proposed path integral networks (PI-Net). We can train the accelerated PI-Net more efficiently for inverse optimal control with less RAM than the original PI-Net.