DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories

TL;DR

DeepImageSearch introduces an agent-based approach for context-aware image retrieval in visual histories, emphasizing multi-step reasoning over temporal visual data, and provides a challenging benchmark with a scalable human-model collaborative pipeline.

Contribution

The paper presents a novel agentic paradigm for image retrieval in visual histories, along with DISBench benchmark and a scalable pipeline leveraging vision-language models.

Findings

DISBench challenges current models significantly.

Agent-based reasoning improves retrieval in visual histories.

The pipeline effectively mines spatiotemporal associations.

Abstract

Existing multimodal retrieval systems excel at semantic matching but implicitly assume that query-image relevance can be measured in isolation. This paradigm overlooks the rich dependencies inherent in realistic visual streams, where information is distributed across temporal sequences rather than confined to single snapshots. To bridge this gap, we introduce DeepImageSearch, a novel agentic paradigm that reformulates image retrieval as an autonomous exploration task. Models must plan and perform multi-step reasoning over raw visual histories to locate targets based on implicit contextual cues. We construct DISBench, a challenging benchmark built on interconnected visual data. To address the scalability challenge of creating context-dependent queries, we propose a human-model collaborative pipeline that employs vision-language models to mine latent spatiotemporal associations, effectively offloading intensive context discovery before human verification. Furthermore, we build a robust baseline using a modular agent framework equipped with fine-grained tools and a dual-memory system for long-horizon navigation. Extensive experiments demonstrate that DISBench poses significant challenges to state-of-the-art models, highlighting the necessity of incorporating agentic reasoning into next-generation retrieval systems.