Beyond Referring Expressions: Scenario Comprehension Visual Grounding

TL;DR

This paper introduces a new benchmark, RSC, for scenario-based visual grounding that requires understanding roles, intentions, and context, revealing limitations of current models.

Contribution

The paper presents RSC, a challenging scenario comprehension benchmark with detailed annotations, and proposes ScenGround, a curriculum reasoning method for improved performance.

Findings

Current models struggle with scenario-based queries that require deep understanding.

Curriculum training enhances model performance on difficult cases.

ScenGround improves transferability to standard benchmarks.

Abstract

Existing visual grounding benchmarks primarily evaluate alignment between image regions and literal referring expressions, where models can often succeed by matching a prominent named category. We explore a complementary and more challenging setting of scenario-based visual grounding, where the target must be inferred from roles, intentions, and relational context rather than explicit naming. We introduce Referring Scenario Comprehension (RSC), a benchmark designed for this setting. The queries in this benchmark are paragraph-length texts that describe object roles, user goals, and contextual cues, including deliberate references to distractor objects that often require deep understanding to resolve. Each instance is annotated with interpretable difficulty tags for uniqueness, clutter, size, overlap, and position which expose distinct failure modes and support fine-grained analysis. RSC contains approximately 31k training examples, 4k in-domain test examples, and a 3k out-of-distribution split with unseen object categories. We further propose ScenGround, a curriculum reasoning method serving as a reference point for this setting, combining supervised warm-starting with difficulty-aware reinforcement learning. Experiments show that scenario-based queries expose systematic failures in current models that standard benchmarks do not reveal, and that curriculum training improves performance on challenging slices and transfers to standard benchmarks.