ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge

TL;DR

ARK introduces a comprehensive multimodal retrieval benchmark that evaluates knowledge and reasoning capabilities across diverse visual data types, revealing significant challenges and room for improvement in current retrieval systems.

Contribution

The paper presents ARK, a novel benchmark for multimodal retrieval that assesses knowledge domains and reasoning skills, addressing limitations of existing benchmarks.

Findings

Significant performance gap between knowledge and reasoning retrieval.

Fine-grained visual and spatial reasoning are persistent challenges.

Simple re-ranking and rewriting improve retrieval results.

Abstract

Existing multimodal retrieval benchmarks largely emphasize semantic matching on daily-life images and offer limited diagnostics of professional knowledge and complex reasoning. To address this gap, we introduce ARK, a benchmark designed to analyze multimodal retrieval from two complementary perspectives: (i) knowledge domains (five domains with 17 subtypes), which characterize the content and expertise retrieval relies on, and (ii) reasoning skills (six categories), which characterize the type of inference over multimodal evidence required to identify the correct candidate. Specifically, ARK evaluates retrieval with both unimodal and multimodal queries and candidates, covering 16 heterogeneous visual data types. To avoid shortcut matching during evaluation, most queries are paired with targeted hard negatives that require multi-step reasoning. We evaluate 23 representative text-based and multimodal retrievers on ARK and observe a pronounced gap between knowledge-intensive and reasoning-intensive retrieval, with fine-grained visual and spatial reasoning emerging as persistent bottlenecks. We further show that simple enhancements such as re-ranking and rewriting yield consistent improvements, but substantial headroom remains.