Neural Optimal Control using Learned System Dynamics

TL;DR

This paper introduces a neural network-based optimal control method that learns system dynamics from data, enabling efficient, near-optimal control across large state spaces without prior system models.

Contribution

The paper presents a novel approach combining neural networks and HJB equations to control systems with unknown dynamics, improving sample efficiency and training speed over existing methods.

Findings

Achieves near-optimal control for large state spaces.

More sample-efficient and faster training than recent RL algorithms.

Successfully applied to quadrotor control with 12 states.

Abstract

We study the problem of generating control laws for systems with unknown dynamics. Our approach is to represent the controller and the value function with neural networks, and to train them using loss functions adapted from the Hamilton-Jacobi-Bellman (HJB) equations. In the absence of a known dynamics model, our method first learns the state transitions from data collected by interacting with the system in an offline process. The learned transition function is then integrated to the HJB equations and used to forward simulate the control signals produced by our controller in a feedback loop. In contrast to trajectory optimization methods that optimize the controller for a single initial state, our controller can generate near-optimal control signals for initial states from a large portion of the state space. Compared to recent model-based reinforcement learning algorithms, we show that our method is more sample efficient and trains faster by an order of magnitude. We demonstrate our method in a number of tasks, including the control of a quadrotor with 12 state variables.