What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models

TL;DR

This paper introduces HAERAE-Vision, a benchmark of real-world under-specified visual questions, revealing that current vision-language models struggle with natural, informal queries and that explicit rewriting significantly improves performance.

Contribution

The paper presents HAERAE-Vision, a new benchmark with real-world queries and their explicit rewrites, highlighting the impact of query under-specification on model performance.

Findings

State-of-the-art models achieve under 50% accuracy on original queries.

Explicit query rewriting improves accuracy by 8 to 22 points.

Web search does not fully compensate for under-specified queries.

Abstract

Current vision-language benchmarks predominantly feature well-structured questions with clear, explicit prompts. However, real user queries are often informal and underspecified. Users naturally leave much unsaid, relying on images to convey context. We introduce HAERAE-Vision, a benchmark of 653 real-world visual questions from Korean online communities (0.76% survival from 86K candidates), each paired with an explicit rewrite, yielding 1,306 query variants in total. Evaluating 39 VLMs, we find that even state-of-the-art models (GPT-5, Gemini 2.5 Pro) achieve under 50% on the original queries. Crucially, query explicitation alone yields 8 to 22 point improvements, with smaller models benefiting most. We further show that even with web search, under-specified queries underperform explicit queries without search, revealing that current retrieval cannot compensate for what users leave unsaid. Our findings demonstrate that a substantial portion of VLM difficulty stem from natural query under-specification instead of model capability, highlighting a critical gap between benchmark evaluation and real-world deployment.