Spatial Retrieval Augmented Autonomous Driving

TL;DR

This paper introduces a spatial retrieval paradigm for autonomous driving that incorporates offline geographic images to improve perception and decision-making, especially under poor visibility conditions.

Contribution

The paper proposes a novel spatial retrieval approach using offline geographic images, extending existing autonomous driving datasets and establishing benchmarks across multiple core tasks.

Findings

Enhanced performance in object detection and mapping tasks

Improved robustness under occlusion and poor visibility

Open-source dataset and benchmarks for future research

Abstract

Existing autonomous driving systems rely on onboard sensors (cameras, LiDAR, IMU, etc) for environmental perception. However, this paradigm is limited by the drive-time perception horizon and often fails under limited view scope, occlusion or extreme conditions such as darkness and rain. In contrast, human drivers are able to recall road structure even under poor visibility. To endow models with this ``recall" ability, we propose the spatial retrieval paradigm, introducing offline retrieved geographic images as an additional input. These images are easy to obtain from offline caches (e.g, Google Maps or stored autonomous driving datasets) without requiring additional sensors, making it a plug-and-play extension for existing AD tasks. For experiments, we first extend the nuScenes dataset with geographic images retrieved via Google Maps APIs and align the new data with ego-vehicle trajectories. We establish baselines across five core autonomous driving tasks: object detection, online mapping, occupancy prediction, end-to-end planning, and generative world modeling. Extensive experiments show that the extended modality could enhance the performance of certain tasks. We will open-source dataset curation code, data, and benchmarks for further study of this new autonomous driving paradigm.