Vision-Language Models for Autonomous Driving: CLIP-Based Dynamic Scene Understanding

TL;DR

This paper presents a CLIP-based dynamic scene understanding system for autonomous driving that achieves high accuracy and real-time performance, enhancing safety and decision-making in complex driving scenarios.

Contribution

It introduces a novel CLIP-based framework optimized for real-time edge deployment, outperforming existing in-context learning methods in complex driving environments.

Findings

Achieved 91.1% F1 score in scene classification.

Demonstrated robustness across diverse weather and road conditions.

Outperformed state-of-the-art zero-shot methods like GPT-4o.

Abstract

Scene understanding is essential for enhancing driver safety, generating human-centric explanations for Automated Vehicle (AV) decisions, and leveraging Artificial Intelligence (AI) for retrospective driving video analysis. This study developed a dynamic scene retrieval system using Contrastive Language-Image Pretraining (CLIP) models, which can be optimized for real-time deployment on edge devices. The proposed system outperforms state-of-the-art in-context learning methods, including the zero-shot capabilities of GPT-4o, particularly in complex scenarios. By conducting frame-level analysis on the Honda Scenes Dataset, which contains a collection of about 80 hours of annotated driving videos capturing diverse real-world road and weather conditions, our study highlights the robustness of CLIP models in learning visual concepts from natural language supervision. Results also showed that fine-tuning the CLIP models, such as ViT-L/14 and ViT-B/32, significantly improved scene classification, achieving a top F1 score of 91.1%. These results demonstrate the ability of the system to deliver rapid and precise scene recognition, which can be used to meet the critical requirements of Advanced Driver Assistance Systems (ADAS). This study shows the potential of CLIP models to provide scalable and efficient frameworks for dynamic scene understanding and classification. Furthermore, this work lays the groundwork for advanced autonomous vehicle technologies by fostering a deeper understanding of driver behavior, road conditions, and safety-critical scenarios, marking a significant step toward smarter, safer, and more context-aware autonomous driving systems.