ReSeDis: A Dataset for Referring-based Object Search across Large-Scale Image Collections

TL;DR

ReSeDis introduces a novel unified task combining large-scale image retrieval with fine-grained object localization based on natural language descriptions, addressing limitations of existing methods.

Contribution

It presents the first benchmark and task for joint corpus-level retrieval and pixel-level grounding, along with a zero-shot baseline using frozen vision-language models.

Findings

Benchmark dataset with unique description-to-object mappings

Proposed metric combining retrieval recall and localization precision

Baseline results indicating significant room for improvement

Abstract

Large-scale visual search engines are expected to solve a dual problem at once: (i) locate every image that truly contains the object described by a sentence and (ii) identify the object's bounding box or exact pixels within each hit. Existing techniques address only one side of this challenge. Visual grounding yields tight boxes and masks but rests on the unrealistic assumption that the object is present in every test image, producing a flood of false alarms when applied to web-scale collections. Text-to-image retrieval excels at sifting through massive databases to rank relevant images, yet it stops at whole-image matches and offers no fine-grained localization. We introduce Referring Search and Discovery (ReSeDis), the first task that unifies corpus-level retrieval with pixel-level grounding. Given a free-form description, a ReSeDis model must decide whether the queried object appears in each image and, if so, where it is, returning bounding boxes or segmentation masks. To enable rigorous study, we curate a benchmark in which every description maps uniquely to object instances scattered across a large, diverse corpus, eliminating unintended matches. We further design a task-specific metric that jointly scores retrieval recall and localization precision. Finally, we provide a straightforward zero-shot baseline using a frozen vision-language model, revealing significant headroom for future study. ReSeDis offers a realistic, end-to-end testbed for building the next generation of robust and scalable multimodal search systems.