RefAV: Towards Planning-Centric Scenario Mining

TL;DR

RefAV introduces a vision-language model-based approach for precise, scalable scenario mining in autonomous vehicle logs, supported by a large dataset and empirical analysis of baseline methods.

Contribution

The paper presents RefAV, a novel dataset and methodology leveraging vision-language models for accurate scenario localization in AV logs, addressing limitations of traditional techniques.

Findings

Naive use of off-the-shelf VLMs performs poorly on scenario mining.

RefAV dataset contains 10,000 natural language queries for complex scenarios.

Empirical analysis highlights challenges and future directions in vision-language based scenario detection.

Abstract

Autonomous Vehicles (AVs) collect and pseudo-label terabytes of multi-modal data localized to HD maps during normal fleet testing. However, identifying interesting and safety-critical scenarios from uncurated driving logs remains a significant challenge. Traditional scenario mining techniques are error-prone and prohibitively time-consuming, often relying on hand-crafted structured queries. In this work, we revisit spatio-temporal scenario mining through the lens of recent vision-language models (VLMs) to detect whether a described scenario occurs in a driving log and, if so, precisely localize it in both time and space. To address this problem, we introduce RefAV, a large-scale dataset of 10,000 diverse natural language queries that describe complex multi-agent interactions relevant to motion planning derived from 1000 driving logs in the Argoverse 2 Sensor dataset. We evaluate several referential multi-object trackers and present an empirical analysis of our baselines. Notably, we find that naively repurposing off-the-shelf VLMs yields poor performance, suggesting that scenario mining presents unique challenges. Lastly, we discuss our recently held competition and share insights from the community. Our code and dataset are available at https://github.com/CainanD/RefAV/ and https://argoverse.github.io/user-guide/tasks/scenario_mining.html