One for All: One-stage Referring Expression Comprehension with Dynamic Reasoning

TL;DR

This paper introduces a dynamic multi-step reasoning network for referring expression comprehension that adjusts reasoning steps on-the-fly, improving performance across various datasets with expressions of differing complexity.

Contribution

It proposes a novel Transformer-based model with reinforcement learning to dynamically determine reasoning steps, addressing fixed-step limitations in prior models.

Findings

Achieves state-of-the-art results on multiple REC datasets.

Improves handling of complex, long, and compositional expressions.

Demonstrates significant performance gains over fixed-step models.

Abstract

Referring Expression Comprehension (REC) is one of the most important tasks in visual reasoning that requires a model to detect the target object referred by a natural language expression. Among the proposed pipelines, the one-stage Referring Expression Comprehension (OSREC) has become the dominant trend since it merges the region proposal and selection stages. Many state-of-the-art OSREC models adopt a multi-hop reasoning strategy because a sequence of objects is frequently mentioned in a single expression which needs multi-hop reasoning to analyze the semantic relation. However, one unsolved issue of these models is that the number of reasoning steps needs to be pre-defined and fixed before inference, ignoring the varying complexity of expressions. In this paper, we propose a Dynamic Multi-step Reasoning Network, which allows the reasoning steps to be dynamically adjusted based on the reasoning state and expression complexity. Specifically, we adopt a Transformer module to memorize & process the reasoning state and a Reinforcement Learning strategy to dynamically infer the reasoning steps. The work achieves the state-of-the-art performance or significant improvements on several REC datasets, ranging from RefCOCO (+, g) with short expressions, to Ref-Reasoning, a dataset with long and complex compositional expressions.