Scene-Aware Conversational ADAS with Generative AI for Real-Time Driver Assistance

TL;DR

This paper introduces SC-ADAS, a modular, real-time driver assistance system that uses generative AI to interpret scenes and engage drivers in natural language dialogue, enhancing flexibility and contextual understanding.

Contribution

The paper presents a novel framework integrating generative AI, vision interpretation, and structured commands for adaptive, scene-aware driver assistance without model fine-tuning.

Findings

Supports multi-turn, scene-aware dialogue in simulated environments

Demonstrates real-time execution of driver commands via structured AI components

Highlights trade-offs like increased latency and dialogue token growth

Abstract

While autonomous driving technologies continue to advance, current Advanced Driver Assistance Systems (ADAS) remain limited in their ability to interpret scene context or engage with drivers through natural language. These systems typically rely on predefined logic and lack support for dialogue-based interaction, making them inflexible in dynamic environments or when adapting to driver intent. This paper presents Scene-Aware Conversational ADAS (SC-ADAS), a modular framework that integrates Generative AI components including large language models, vision-to-text interpretation, and structured function calling to enable real-time, interpretable, and adaptive driver assistance. SC-ADAS supports multi-turn dialogue grounded in visual and sensor context, allowing natural language recommendations and driver-confirmed ADAS control. Implemented in the CARLA simulator with cloud-based Generative AI, the system executes confirmed user intents as structured ADAS commands without requiring model fine-tuning. We evaluate SC-ADAS across scene-aware, conversational, and revisited multi-turn interactions, highlighting trade-offs such as increased latency from vision-based context retrieval and token growth from accumulated dialogue history. These results demonstrate the feasibility of combining conversational reasoning, scene perception, and modular ADAS control to support the next generation of intelligent driver assistance.