Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UI

TL;DR

This paper presents a vision-language framework and dataset for understanding and interacting with automotive UIs, demonstrating strong cross-domain performance and cost-effective deployment of fine-tuned models.

Contribution

Introduces AutomotiveUI-Bench-4K dataset and a fine-tuned ELAM model for automotive UI understanding using vision-language techniques.

Findings

Achieved 80.8% accuracy on ScreenSpot benchmark.

Demonstrated +5.6% improvement over baseline on ScreenSpot.

Model and dataset are publicly available on Hugging Face.

Abstract

Modern automotive infotainment systems necessitate intelligent and adaptive solutions to manage frequent User Interface (UI) updates and diverse design variations. This work introduces a vision-language framework to facilitate the understanding of and interaction with automotive UIs, enabling seamless adaptation across different UI designs. To support research in this field, AutomotiveUI-Bench-4K, an open-source dataset comprising 998 images with 4,208 annotations, is also released. Additionally, a data pipeline for generating training data is presented. A Molmo-7B-based model is fine-tuned using Low-Rank Adaptation (LoRa), incorporating generated reasoning along with visual grounding and evaluation capabilities. The fine-tuned Evaluative Large Action Model (ELAM) achieves strong performance on AutomotiveUI-Bench-4K (model and dataset are available on Hugging Face). The approach demonstrates strong cross-domain generalization, including a +5.6% improvement on ScreenSpot over the baseline model. An average accuracy of 80.8% is achieved on ScreenSpot, closely matching or surpassing specialized models for desktop, mobile, and web, despite being trained primarily on the automotive domain. This research investigates how data collection and subsequent fine-tuning can lead to AI-driven advancements in automotive UI understanding and interaction. The applied method is cost-efficient, and fine-tuned models can be deployed on consumer-grade GPUs.