Autonomous Vehicle Controllers From End-to-End Differentiable Simulation

TL;DR

This paper introduces an end-to-end differentiable simulation framework with analytic policy gradients for training autonomous vehicle controllers, resulting in more robust, accurate, and human-like policies compared to traditional behavioural cloning.

Contribution

It presents a novel APG approach that integrates a differentiable simulator into the training loop, improving policy robustness and efficiency using large-scale real-world data.

Findings

Significant performance improvements over behavioural cloning.

Enhanced robustness to dynamic noise.

More human-like vehicle handling behaviors.

Abstract

Current methods to learn controllers for autonomous vehicles (AVs) focus on behavioural cloning. Being trained only on exact historic data, the resulting agents often generalize poorly to novel scenarios. Simulators provide the opportunity to go beyond offline datasets, but they are still treated as complicated black boxes, only used to update the global simulation state. As a result, these RL algorithms are slow, sample-inefficient, and prior-agnostic. In this work, we leverage a differentiable simulator and design an analytic policy gradients (APG) approach to training AV controllers on the large-scale Waymo Open Motion Dataset. Our proposed framework brings the differentiable simulator into an end-to-end training loop, where gradients of the environment dynamics serve as a useful prior to help the agent learn a more grounded policy. We combine this setup with a recurrent architecture that can efficiently propagate temporal information across long simulated trajectories. This APG method allows us to learn robust, accurate, and fast policies, while only requiring widely-available expert trajectories, instead of scarce expert actions. We compare to behavioural cloning and find significant improvements in performance and robustness to noise in the dynamics, as well as overall more intuitive human-like handling.