Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Models

TL;DR

This paper introduces VDR-Bench, a new benchmark for evaluating multimodal large language models' visual and textual search abilities under realistic conditions, and proposes a multi-round search workflow to enhance retrieval performance.

Contribution

The paper presents VDR-Bench, a carefully curated benchmark for realistic visual-textual search evaluation, and introduces a multi-round cropped-search method to improve model retrieval capabilities.

Findings

VDR-Bench effectively challenges current MLLMs in realistic scenarios.

Multi-round cropped-search improves visual retrieval performance.

Benchmark and method guide future multimodal system development.

Abstract

Multimodal Large Language Models (MLLMs) have advanced VQA and now support Vision-DeepResearch systems that use search engines for complex visual-textual fact-finding. However, evaluating these visual and textual search abilities is still difficult, and existing benchmarks have two major limitations. First, existing benchmarks are not visual search-centric: answers that should require visual search are often leaked through cross-textual cues in the text questions or can be inferred from the prior world knowledge in current MLLMs. Second, overly idealized evaluation scenario: On the image-search side, the required information can often be obtained via near-exact matching against the full image, while the text-search side is overly direct and insufficiently challenging. To address these issues, we construct the Vision-DeepResearch benchmark (VDR-Bench) comprising 2,000 VQA instances. All questions are created via a careful, multi-stage curation pipeline and rigorous expert review, designed to assess the behavior of Vision-DeepResearch systems under realistic real-world conditions. Moreover, to address the insufficient visual retrieval capabilities of current MLLMs, we propose a simple multi-round cropped-search workflow. This strategy is shown to effectively improve model performance in realistic visual retrieval scenarios. Overall, our results provide practical guidance for the design of future multimodal deep-research systems. The code will be released in https://github.com/Osilly/Vision-DeepResearch.