GREC: Generalized Referring Expression Comprehension

TL;DR

This paper introduces GREC, a new benchmark and dataset for referring expression comprehension that handles multiple targets and non-target expressions, extending the scope of classic REC for more practical applications.

Contribution

It presents the first large-scale GREC dataset, gRefCOCO, enabling the study of expressions referring to multiple or no specific objects, and offers a compatible evaluation framework.

Findings

First large-scale GREC dataset gRefCOCO created

Supports expressions for multiple targets and no specific target

Provides code for GREC methods and evaluation

Abstract

The objective of Classic Referring Expression Comprehension (REC) is to produce a bounding box corresponding to the object mentioned in a given textual description. Commonly, existing datasets and techniques in classic REC are tailored for expressions that pertain to a single target, meaning a sole expression is linked to one specific object. Expressions that refer to multiple targets or involve no specific target have not been taken into account. This constraint hinders the practical applicability of REC. This study introduces a new benchmark termed as Generalized Referring Expression Comprehension (GREC). This benchmark extends the classic REC by permitting expressions to describe any number of target objects. To achieve this goal, we have built the first large-scale GREC dataset named gRefCOCO. This dataset encompasses a range of expressions: those referring to multiple targets, expressions with no specific target, and the single-target expressions. The design of GREC and gRefCOCO ensures smooth compatibility with classic REC. The proposed gRefCOCO dataset, a GREC method implementation code, and GREC evaluation code are available at https://github.com/henghuiding/gRefCOCO.