DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles

TL;DR

DiffStack introduces a differentiable, modular control stack for autonomous vehicles that enables joint optimization of components like prediction, planning, and control, improving performance while maintaining interpretability.

Contribution

It presents a novel differentiable control stack that combines modularity with end-to-end training, bridging the gap between traditional analytical components and neural network approaches.

Findings

Significant improvements in planning metrics on nuScenes dataset.

Enhanced prediction accuracy reduces downstream planning errors.

Enables joint optimization of AV components via backpropagation.

Abstract

Autonomous vehicle (AV) stacks are typically built in a modular fashion, with explicit components performing detection, tracking, prediction, planning, control, etc. While modularity improves reusability, interpretability, and generalizability, it also suffers from compounding errors, information bottlenecks, and integration challenges. To overcome these challenges, a prominent approach is to convert the AV stack into an end-to-end neural network and train it with data. While such approaches have achieved impressive results, they typically lack interpretability and reusability, and they eschew principled analytical components, such as planning and control, in favor of deep neural networks. To enable the joint optimization of AV stacks while retaining modularity, we present DiffStack, a differentiable and modular stack for prediction, planning, and control. Crucially, our model-based planning and control algorithms leverage recent advancements in differentiable optimization to produce gradients, enabling optimization of upstream components, such as prediction, via backpropagation through planning and control. Our results on the nuScenes dataset indicate that end-to-end training with DiffStack yields substantial improvements in open-loop and closed-loop planning metrics by, e.g., learning to make fewer prediction errors that would affect planning. Beyond these immediate benefits, DiffStack opens up new opportunities for fully data-driven yet modular and interpretable AV architectures. Project website: https://sites.google.com/view/diffstack