Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models

TL;DR

This paper introduces a multimodal foundation model-based end-to-end autonomous driving system that is robust to open-set environments, providing explainability and improved training through text-based data augmentation.

Contribution

It presents a novel approach leveraging multimodal foundation models for open-set autonomous driving, extracting spatial and semantic features from transformers for robustness and explainability.

Findings

Achieves superior performance in diverse, out-of-distribution tests.

Enables data augmentation and debugging using text-based latent space simulation.

Demonstrates enhanced robustness and adaptability in autonomous driving scenarios.

Abstract

As autonomous driving technology matures, end-to-end methodologies have emerged as a leading strategy, promising seamless integration from perception to control via deep learning. However, existing systems grapple with challenges such as unexpected open set environments and the complexity of black-box models. At the same time, the evolution of deep learning introduces larger, multimodal foundational models, offering multi-modal visual and textual understanding. In this paper, we harness these multimodal foundation models to enhance the robustness and adaptability of autonomous driving systems, enabling out-of-distribution, end-to-end, multimodal, and more explainable autonomy. Specifically, we present an approach to apply end-to-end open-set (any environment/scene) autonomous driving that is capable of providing driving decisions from representations queryable by image and text. To do so, we introduce a method to extract nuanced spatial (pixel/patch-aligned) features from transformers to enable the encapsulation of both spatial and semantic features. Our approach (i) demonstrates unparalleled results in diverse tests while achieving significantly greater robustness in out-of-distribution situations, and (ii) allows the incorporation of latent space simulation (via text) for improved training (data augmentation via text) and policy debugging. We encourage the reader to check our explainer video at https://www.youtube.com/watch?v=4n-DJf8vXxo&feature=youtu.be and to view the code and demos on our project webpage at https://drive-anywhere.github.io/.