Unleashing the Capabilities of Large Vision-Language Models for Intelligent Perception of Roadside Infrastructure

TL;DR

This paper presents a domain-adapted framework that enhances large vision-language models for accurate, standards-compliant perception of roadside infrastructure in smart city applications, combining fine-tuning, reasoning, and retrieval techniques.

Contribution

It introduces a novel domain-specific adaptation of VLMs using data-efficient fine-tuning, knowledge-grounded reasoning, and retrieval-augmented generation for infrastructure analysis.

Findings

Achieved 58.9 mAP in asset detection

Attained 95.5% accuracy in attribute recognition

Demonstrated robustness on a new urban roadside dataset

Abstract

Automated perception of urban roadside infrastructure is crucial for smart city management, yet general-purpose models often struggle to capture the necessary fine-grained attributes and domain rules. While Large Vision Language Models (VLMs) excel at open-world recognition, they often struggle to accurately interpret complex facility states in compliance with engineering standards, leading to unreliable performance in real-world applications. To address this, we propose a domain-adapted framework that transforms VLMs into specialized agents for intelligent infrastructure analysis. Our approach integrates a data-efficient fine-tuning strategy with a knowledge-grounded reasoning mechanism. Specifically, we leverage open-vocabulary fine-tuning on Grounding DINO to robustly localize diverse assets with minimal supervision, followed by LoRA-based adaptation on Qwen-VL for deep semantic attribute reasoning. To mitigate hallucinations and enforce professional compliance, we introduce a dual-modality Retrieval-Augmented Generation (RAG) module that dynamically retrieves authoritative industry standards and visual exemplars during inference. Evaluated on a comprehensive new dataset of urban roadside scenes, our framework achieves a detection performance of 58.9 mAP and an attribute recognition accuracy of 95.5%, demonstrating a robust solution for intelligent infrastructure monitoring.