SMc2f: Robust Scenario Mining for Robotic Autonomy from Coarse to Fine

TL;DR

This paper introduces SMc2f, a robust, multi-stage scenario mining pipeline for autonomous robots that improves the retrieval of safety-critical scenarios from large datasets by combining vision-language models, few-shot learning, and contrastive learning.

Contribution

The paper presents a novel coarse-to-fine scenario mining approach that enhances retrieval accuracy and robustness by integrating vision-language models and contrastive learning techniques.

Findings

Significant improvements in retrieval quality over baseline methods.

Enhanced efficiency in scenario mining from large datasets.

Robustness against inaccuracies in trajectory data.

Abstract

The safety validation of autonomous robotic vehicles hinges on systematically testing their planning and control stacks against rare, safety-critical scenarios. Mining these long-tail events from massive real-world driving logs is therefore a critical step in the robotic development lifecycle. The goal of the Scenario Mining task is to retrieve useful information to enable targeted re-simulation, regression testing, and failure analysis of the robot's decision-making algorithms. RefAV, introduced by the Argoverse team, is an end-to-end framework that uses large language models (LLMs) to spatially and temporally localize scenarios described in natural language. However, this process performs retrieval on trajectory labels, ignoring the direct connection between natural language and raw RGB images, which runs counter to the intuition of video retrieval; it also depends on the quality of upstream 3D object detection and tracking. Further, inaccuracies in trajectory data lead to inaccuracies in downstream spatial and temporal localization. To address these issues, we propose Robust Scenario Mining for Robotic Autonomy from Coarse to Fine (SMc2f), a coarse-to-fine pipeline that employs vision-language models (VLMs) for coarse image-text filtering, builds a database of successful mining cases on top of RefAV and automatically retrieves exemplars to few-shot condition the LLM for more robust retrieval, and introduces text-trajectory contrastive learning to pull matched pairs together and push mismatched pairs apart in a shared embedding space, yielding a fine-grained matcher that refines the LLM's candidate trajectories. Experiments on public datasets demonstrate substantial gains in both retrieval quality and efficiency.