Research on Driving Scenario Technology Based on Multimodal Large Lauguage Model Optimization

TL;DR

This paper presents a comprehensive optimization framework for multimodal large language models tailored to complex driving scenarios, enhancing accuracy and efficiency in autonomous driving applications.

Contribution

It introduces dynamic prompt optimization, dataset construction with real and synthetic data, and advanced training techniques like knowledge distillation for driving scenario models.

Findings

Improved model accuracy in key driving tasks

Enhanced resource efficiency through quantization and distillation

Effective handling of complex driving environments

Abstract

With the advancement of autonomous and assisted driving technologies, higher demands are placed on the ability to understand complex driving scenarios. Multimodal general large models have emerged as a solution for this challenge. However, applying these models in vertical domains involves difficulties such as data collection, model training, and deployment optimization. This paper proposes a comprehensive method for optimizing multimodal models in driving scenarios, including cone detection, traffic light recognition, speed limit recommendation, and intersection alerts. The method covers key aspects such as dynamic prompt optimization, dataset construction, model training, and deployment. Specifically, the dynamic prompt optimization adjusts the prompts based on the input image content to focus on objects affecting the ego vehicle, enhancing the model's task-specific focus and judgment capabilities. The dataset is constructed by combining real and synthetic data to create a high-quality and diverse multimodal training dataset, improving the model's generalization in complex driving environments. In model training, advanced techniques like knowledge distillation, dynamic fine-tuning, and quantization are integrated to reduce storage and computational costs while boosting performance. Experimental results show that this systematic optimization method not only significantly improves the model's accuracy in key tasks but also achieves efficient resource utilization, providing strong support for the practical application of driving scenario perception technologies.