DriveSafe: A Hierarchical Risk Taxonomy for Safety-Critical LLM-Based Driving Assistants

TL;DR

DriveSafe introduces a detailed hierarchical risk taxonomy for LLM-based driving assistants, highlighting the domain-specific safety challenges and evaluating model refusal behavior to unsafe queries.

Contribution

The paper presents a novel four-level risk taxonomy with 129 categories tailored for driving assistants, grounded in real-world regulations and validated through model evaluations.

Findings

Models often fail to refuse unsafe driving queries

The taxonomy captures diverse safety-critical failure modes

Evaluation reveals limitations of current safety alignments

Abstract

Large Language Models (LLMs) are increasingly integrated into vehicle-based digital assistants, where unsafe, ambiguous, or legally incorrect responses can lead to serious safety, ethical, and regulatory consequences. Despite growing interest in LLM safety, existing taxonomies and evaluation frameworks remain largely general-purpose and fail to capture the domain-specific risks inherent to real-world driving scenarios. In this paper, we introduce DriveSafe, a hierarchical, four-level risk taxonomy designed to systematically characterize safety-critical failure modes of LLM-based driving assistants. The taxonomy comprises 129 fine-grained atomic risk categories spanning technical, legal, societal, and ethical dimensions, grounded in real-world driving regulations and safety principles and reviewed by domain experts. To validate the safety relevance and realism of the constructed prompts, we evaluate their refusal behavior across six widely deployed LLMs. Our analysis shows that the evaluated models often fail to appropriately refuse unsafe or non-compliant driving-related queries, underscoring the limitations of general-purpose safety alignment in driving contexts.