Dynamic Graph Attention for Referring Expression Comprehension

TL;DR

This paper introduces a dynamic graph attention network that models object relationships and linguistic structure for improved referring expression comprehension, enabling multi-step reasoning and interpretable visual evidence generation.

Contribution

It proposes a novel language-guided visual reasoning approach using a dynamic graph attention network for complex referring expressions.

Findings

Significantly outperforms state-of-the-art methods on benchmark datasets.

Enables interpretable stepwise reasoning with visual evidence.

Effectively models relationships and linguistic structure for complex expressions.

Abstract

Referring expression comprehension aims to locate the object instance described by a natural language referring expression in an image. This task is compositional and inherently requires visual reasoning on top of the relationships among the objects in the image. Meanwhile, the visual reasoning process is guided by the linguistic structure of the referring expression. However, existing approaches treat the objects in isolation or only explore the first-order relationships between objects without being aligned with the potential complexity of the expression. Thus it is hard for them to adapt to the grounding of complex referring expressions. In this paper, we explore the problem of referring expression comprehension from the perspective of language-driven visual reasoning, and propose a dynamic graph attention network to perform multi-step reasoning by modeling both the relationships among the objects in the image and the linguistic structure of the expression. In particular, we construct a graph for the image with the nodes and edges corresponding to the objects and their relationships respectively, propose a differential analyzer to predict a language-guided visual reasoning process, and perform stepwise reasoning on top of the graph to update the compound object representation at every node. Experimental results demonstrate that the proposed method can not only significantly surpass all existing state-of-the-art algorithms across three common benchmark datasets, but also generate interpretable visual evidences for stepwisely locating the objects referred to in complex language descriptions.