Why Braking? Scenario Extraction and Reasoning Utilizing LLM

TL;DR

This paper introduces a novel LLM-based framework for understanding and reasoning about driving scenarios, especially braking events, to improve safety-critical case retrieval in complex urban environments.

Contribution

It presents a new approach leveraging LLMs for scenario understanding, combining category and embedding-based retrieval, and curates a dataset for evaluation.

Findings

Outperforms rule-based methods in scenario retrieval

Generalizes effectively to out-of-distribution scenarios

Bridges numerical signals and natural language for better interpretation

Abstract

The growing number of ADAS-equipped vehicles has led to a dramatic increase in driving data, yet most of them capture routine driving behavior. Identifying and understanding safety-critical corner cases within this vast dataset remains a significant challenge. Braking events are particularly indicative of potentially hazardous situations, motivating the central question of our research: Why does a vehicle brake? Existing approaches primarily rely on rule-based heuristics to retrieve target scenarios using predefined condition filters. While effective in simple environments such as highways, these methods lack generalization in complex urban settings. In this paper, we propose a novel framework that leverages Large Language Model (LLM) for scenario understanding and reasoning. Our method bridges the gap between low-level numerical signals and natural language descriptions, enabling LLM to interpret and classify driving scenarios. We propose a dual-path scenario retrieval that supports both category-based search for known scenarios and embedding-based retrieval for unknown Out-of-Distribution (OOD) scenarios. To facilitate evaluation, we curate scenario annotations on the Argoverse 2 Sensor Dataset. Experimental results show that our method outperforms rule-based baselines and generalizes well to OOD scenarios.