KnowDR-REC: A Benchmark for Referring Expression Comprehension with Real-World Knowledge

TL;DR

KnowDR-REC is a new benchmark designed to evaluate multimodal models' reasoning with real-world knowledge, incorporating fine-grained annotations, negative samples, and novel metrics to assess robustness and interpretability.

Contribution

The paper introduces KnowDR-REC, a comprehensive benchmark with real-world knowledge, negative samples, and new evaluation metrics for assessing multimodal models' reasoning capabilities.

Findings

Existing models struggle with knowledge-driven visual grounding.

Many models rely on shortcut correlations rather than genuine reasoning.

The benchmark reveals a gap between textual understanding and visual grounding.

Abstract

Referring Expression Comprehension (REC) is a popular multimodal task that aims to accurately detect target objects within a single image based on a given textual expression. However, due to the limitations of earlier models, traditional REC benchmarks either rely solely on intra-image cues or lack sufficiently fine-grained instance annotations, making them inadequate for evaluating the reasoning capabilities of Multi-modal Large Language Models (MLLMs). To address this gap, we propose a new benchmark, KnowDR-REC, characterized by three key features: Firstly, it is built upon real-world knowledge, requiring fine-grained multimodal reasoning across text and image. Secondly, the dataset includes elaborately constructed negative samples via fine-grained expression editing, designed to evaluate a model's robustness and anti-hallucination ability. Lastly, we introduce three novel evaluation metrics to systematically explore the model's internal reasoning process. We evaluate 16 state-of-the-art multimodal models on KnowDR-REC, with experimental results showing that existing MLLMs still struggle with knowledge-driven visual grounding tasks. Furthermore, we observe a decoupling between textual understanding and visual grounding in MLLMs, where many models are significantly influenced by memorized shortcut correlations, which severely affect their behavior on our benchmark and hinder genuine multimodal reasoning. We anticipate that the proposed benchmark will inspire future research towards developing more robust, interpretable, and knowledge-intensive visual grounding frameworks, driving the development of more reliable and robust multimodal systems for complex real-world scenarios.