GREx: Generalized Referring Expression Segmentation, Comprehension, and Generation

TL;DR

This paper introduces GREx, a set of new benchmarks and a large-scale dataset that extend referring expression tasks to multi-target, no-target, and single-target scenarios, enabling more realistic applications.

Contribution

The paper presents the first large-scale GREx dataset gRefCOCO supporting complex expressions and proposes ReLA, a baseline model that improves performance on these tasks.

Findings

ReLA achieves state-of-the-art results on GRES and GREC tasks.

gRefCOCO dataset includes multi-target, no-target, and single-target expressions.

GREx benchmarks reveal performance gaps in existing REx methods.

Abstract

Referring Expression Segmentation (RES) and Comprehension (REC) respectively segment and detect the object described by an expression, while Referring Expression Generation (REG) generates an expression for the selected object. Existing datasets and methods commonly support single-target expressions only, i.e., one expression refers to one object, not considering multi-target and no-target expressions. This greatly limits the real applications of REx (RES/REC/REG). This paper introduces three new benchmarks called Generalized Referring Expression Segmentation (GRES), Comprehension (GREC), and Generation (GREG), collectively denoted as GREx, which extend the classic REx to allow expressions to identify an arbitrary number of objects. We construct the first large-scale GREx dataset gRefCOCO that contains multi-target, no-target, and single-target expressions and their corresponding images with labeled targets. GREx and gRefCOCO are designed to be backward-compatible with REx, facilitating extensive experiments to study the performance gap of the existing REx methods on GREx tasks. One of the challenges of GRES/GREC is complex relationship modeling, for which we propose a baseline ReLA that adaptively divides the image into regions with sub-instance clues and explicitly models the region-region and region-language dependencies. The proposed ReLA achieves the state-of-the-art results on the both GRES and GREC tasks. The proposed gRefCOCO dataset and method are available at https://henghuiding.github.io/GREx.