WildRoadBench: A Wild Aerial Road-Damage Grounding Benchmark for Vision-Language Models and Autonomous Agents

TL;DR

WildRoadBench introduces a comprehensive UAV benchmark for evaluating vision-language models and autonomous agents in wild aerial road-damage detection, highlighting current limitations and fostering future research.

Contribution

It presents a novel benchmark coupling visual grounding and autonomous agent tasks on a UAV dataset, with detailed evaluation protocols and baseline model assessments.

Findings

Closed-source models outperform open-source ones but still leave significant room for improvement.

Open-source models struggle with small targets and reasoning tasks.

Autonomous agents underperform compared to vision-language models, often failing to submit valid predictions within constraints.

Abstract

We introduce WildRoadBench, a wild aerial road-damage grounding benchmark that couples direct visual grounding by vision-language models with autonomous research-and-engineering by LLM-driven agents on a single professionally annotated UAV corpus. The same image set and the same per-class AP_50 metric are evaluated under two protocols. The VLM Track measures whether a fixed VLM can localise domain-specific damage from one image and one short prompt under a unified prompting, decoding and parsing pipeline. The Agent Track measures whether an autonomous agent, given only a written task brief, a small exploratory slice and a fixed interaction budget, can search the public web, adapt pretrained components, write training and inference code, and submit predictions through a scalar-feedback oracle on a hidden holdout. We benchmark a broad pool of closed-source frontier models and open-source VLMs together with several frontier LLM-driven agents. Both routes remain far from reliable performance in this wild setting: closed-source frontier models lead the VLM leaderboard but still leave more than half of the metric on the table; open-source grounders plateau well below them, and newer generations or reasoning-style variants do not consistently improve grounding; small targets collapse for every open-source model; agents lag the strongest VLM despite richer affordances, and several fail to land a valid submission within the budget. We release the code and data at https://anonymous.4open.science/r/wildroadbench-0607 to support reproducible follow-up research.