FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension

TL;DR

This paper introduces FineCops-Ref, a challenging new dataset for Referring Expression Comprehension that includes multi-level reasoning and negative samples, aiming to improve multi-modal understanding and grounding in AI models.

Contribution

The paper presents a novel dataset with controllable difficulty levels and negative samples, specifically designed to evaluate and enhance fine-grained multi-modal reasoning in REC tasks.

Findings

Significant performance gap in current models' grounding abilities

Dataset enables testing of multi-hop and attribute-based reasoning

Negative samples challenge models to reject incorrect references

Abstract

Referring Expression Comprehension (REC) is a crucial cross-modal task that objectively evaluates the capabilities of language understanding, image comprehension, and language-to-image grounding. Consequently, it serves as an ideal testing ground for Multi-modal Large Language Models (MLLMs). In pursuit of this goal, we have established a new REC dataset characterized by two key features: Firstly, it is designed with controllable varying levels of difficulty, necessitating multi-level fine-grained reasoning across object categories, attributes, and multi-hop relationships. Secondly, it includes negative text and images created through fine-grained editing and generation based on existing data, thereby testing the model's ability to correctly reject scenarios where the target object is not visible in the image--an essential aspect often overlooked in existing datasets and approaches. Utilizing this high-quality dataset, we conducted comprehensive evaluations of both state-of-the-art specialist models and MLLMs. Our findings indicate that there remains a significant gap in achieving satisfactory grounding performance. We anticipate that our dataset will inspire new approaches to enhance visual reasoning and develop more advanced cross-modal interaction strategies, ultimately unlocking the full potential of MLLMs. Our code and the datasets are available at https://github.com/liujunzhuo/FineCops-Ref.