Pontryagin Optimal Control via Neural Networks

TL;DR

This paper introduces a neural network-based framework integrating Pontryagin's Maximum Principle to efficiently solve complex, high-dimensional optimal control problems with unknown dynamics, outperforming traditional RL methods.

Contribution

It proposes a novel NN-PMP-Gradient framework that combines neural networks with PMP for sample-efficient optimal control in unknown systems.

Findings

Demonstrates effectiveness on LQR, energy arbitrage, pendulum, and MuJoCo tasks.

Achieves higher sample-efficiency than RL algorithms.

Provides a versatile tool for optimal control with complex dynamics.

Abstract

Solving real-world optimal control problems are challenging tasks, as the complex, high-dimensional system dynamics are usually unrevealed to the decision maker. It is thus hard to find the optimal control actions numerically. To deal with such modeling and computation challenges, in this paper, we integrate Neural Networks with the Pontryagin's Maximum Principle (PMP), and propose a sample efficient framework NN-PMP-Gradient. The resulting controller can be implemented for systems with unknown and complex dynamics. By taking an iterative approach, the proposed framework not only utilizes the accurate surrogate models parameterized by neural networks, it also efficiently recovers the optimality conditions along with the optimal action sequences via PMP conditions. Numerical simulations on Linear Quadratic Regulator, energy arbitrage of grid-connected lossy battery, control of single pendulum, and two MuJoCo locomotion tasks demonstrate our proposed NN-PMP-Gradient is a general and versatile computation tool for finding optimal solutions. And compared with the widely applied model-free and model-based reinforcement learning (RL) algorithms, our NN-PMP-Gradient achieves higher sample-efficiency and performance in terms of control objectives.