Referring Expression Comprehension: A Survey of Methods and Datasets

TL;DR

This survey reviews recent methods and datasets for referring expression comprehension, highlighting the challenges, architectures, and future directions in localizing objects in images based on natural language descriptions.

Contribution

It provides a comprehensive classification of REC methods, compares state-of-the-art approaches, and discusses future research directions including compositional reasoning.

Findings

Joint embedding of images and expressions is common in REC methods.

Graph-based models effectively utilize structured representations.

Datasets vary in size and complexity, impacting model evaluation.

Abstract

Referring expression comprehension (REC) aims to localize a target object in an image described by a referring expression phrased in natural language. Different from the object detection task that queried object labels have been pre-defined, the REC problem only can observe the queries during the test. It thus more challenging than a conventional computer vision problem. This task has attracted a lot of attention from both computer vision and natural language processing community, and several lines of work have been proposed, from CNN-RNN model, modular network to complex graph-based model. In this survey, we first examine the state of the art by comparing modern approaches to the problem. We classify methods by their mechanism to encode the visual and textual modalities. In particular, we examine the common approach of joint embedding images and expressions to a common feature space. We also discuss modular architectures and graph-based models that interface with structured graph representation. In the second part of this survey, we review the datasets available for training and evaluating REC systems. We then group results according to the datasets, backbone models, settings so that they can be fairly compared. Finally, we discuss promising future directions for the field, in particular the compositional referring expression comprehension that requires longer reasoning chain to address.