Evaluation and Comparison of Visual Language Models for Transportation Engineering Problems

TL;DR

This paper evaluates state-of-the-art vision language models for transportation engineering tasks like congestion detection, crack identification, and helmet violation detection, using zero-shot prompting to assess their performance without task-specific training.

Contribution

It provides a comprehensive comparison of VLM models for transportation tasks, highlighting their strengths and limitations for future development.

Findings

VLM models perform comparably to CNNs in image classification.

Object localization with VLMs still requires improvement.

Zero-shot prompting enables task execution without annotated datasets.

Abstract

Recent developments in vision language models (VLM) have shown great potential for diverse applications related to image understanding. In this study, we have explored state-of-the-art VLM models for vision-based transportation engineering tasks such as image classification and object detection. The image classification task involves congestion detection and crack identification, whereas, for object detection, helmet violations were identified. We have applied open-source models such as CLIP, BLIP, OWL-ViT, Llava-Next, and closed-source GPT-4o to evaluate the performance of these state-of-the-art VLM models to harness the capabilities of language understanding for vision-based transportation tasks. These tasks were performed by applying zero-shot prompting to the VLM models, as zero-shot prompting involves performing tasks without any training on those tasks. It eliminates the need for annotated datasets or fine-tuning for specific tasks. Though these models gave comparative results with benchmark Convolutional Neural Networks (CNN) models in the image classification tasks, for object localization tasks, it still needs improvement. Therefore, this study provides a comprehensive evaluation of the state-of-the-art VLM models highlighting the advantages and limitations of the models, which can be taken as the baseline for future improvement and wide-scale implementation.