DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios

TL;DR

DriveSafe introduces a risk-aware scene understanding framework for autonomous driving that uses structured natural language descriptions and multimodal cues to improve risk assessment and safety suggestions.

Contribution

It presents a novel method that combines spatially grounded captions with domain-specific fine-tuning to outperform existing zero-shot and baseline approaches.

Findings

DriveSafe achieves state-of-the-art results on the DRAMA benchmark.

Using caption-risk pairings enhances risk assessment accuracy.

The framework provides explicit hazard localization and safety suggestions.

Abstract

Comprehensive situational awareness is essential for autonomous vehicles operating in safety-critical environments, as it enables the identification and mitigation of potential risks. Although recent Multimodal Large Language Models (MLLMs) have shown promise on general vision-language tasks, our findings indicate that zero-shot MLLMs still underperform compared to domain-specific methods in fine-grained, spatially grounded risk assessment. To address this gap, we propose DriveSafe, a framework for risk-aware scene understanding that leverages structured natural language descriptions. Specifically, our method first generates spatially grounded captions enriched with multimodal context, including motion, spatial, and depth cues. These captions are then used for downstream risk assessment, explicitly identifying hazardous objects, their locations, and the unsafe behaviors they imply, followed by actionable safety suggestions. To further improve performance, we employ caption-risk pairings to fine-tune a lightweight adapter module, efficiently injecting domain-specific knowledge into the base LLM. By conditioning risk assessment on explicit language-based scene representations, DriveSafe achieves significant gains over both zero-shot MLLMs and prior domain-specific baselines. Exhaustive experiments on the DRAMA benchmark demonstrate state-of-the-art performance, while ablation studies validate the effectiveness of our key design choices. Project page: https://cvit.iiit.ac.in/ research/projects/cvit-projects/drivesafe