Interpretable Zero-shot Referring Expression Comprehension with Query-driven Scene Graphs

TL;DR

This paper introduces SGREC, a novel zero-shot referring expression comprehension method that uses query-driven scene graphs and large language models to improve interpretability and accuracy in locating objects in images.

Contribution

SGREC leverages scene graphs as structured intermediaries to enhance visual understanding and interpretability in zero-shot REC tasks, bridging vision and language models effectively.

Findings

Achieves top-1 accuracy of 66.78% on RefCOCO val

Outperforms existing zero-shot REC methods on multiple benchmarks

Provides interpretable explanations for object localization decisions

Abstract

Zero-shot referring expression comprehension (REC) aims to locate target objects in images given natural language queries without relying on task-specific training data, demanding strong visual understanding capabilities. Existing Vision-Language Models~(VLMs), such as CLIP, commonly address zero-shot REC by directly measuring feature similarities between textual queries and image regions. However, these methods struggle to capture fine-grained visual details and understand complex object relationships. Meanwhile, Large Language Models~(LLMs) excel at high-level semantic reasoning, their inability to directly abstract visual features into textual semantics limits their application in REC tasks. To overcome these limitations, we propose \textbf{SGREC}, an interpretable zero-shot REC method leveraging query-driven scene graphs as structured intermediaries. Specifically, we first employ a VLM to construct a query-driven scene graph that explicitly encodes spatial relationships, descriptive captions, and object interactions relevant to the given query. By leveraging this scene graph, we bridge the gap between low-level image regions and higher-level semantic understanding required by LLMs. Finally, an LLM infers the target object from the structured textual representation provided by the scene graph, responding with detailed explanations for its decisions that ensure interpretability in the inference process. Extensive experiments show that SGREC achieves top-1 accuracy on most zero-shot REC benchmarks, including RefCOCO val (66.78\%), RefCOCO+ testB (53.43\%), and RefCOCOg val (73.28\%), highlighting its strong visual scene understanding.