TransVG: End-to-End Visual Grounding with Transformers

TL;DR

TransVG introduces a transformer-based framework for visual grounding that simplifies existing complex fusion modules and directly regresses coordinates, achieving state-of-the-art results across multiple datasets.

Contribution

It replaces complex multi-modal fusion modules with a simple transformer encoder stack and reformulates grounding as a coordinate regression task, improving performance.

Findings

Achieves state-of-the-art results on five datasets.

Simplifies the fusion process with transformer encoders.

Reformulates grounding as direct coordinate regression.

Abstract

In this paper, we present a neat yet effective transformer-based framework for visual grounding, namely TransVG, to address the task of grounding a language query to the corresponding region onto an image. The state-of-the-art methods, including two-stage or one-stage ones, rely on a complex module with manually-designed mechanisms to perform the query reasoning and multi-modal fusion. However, the involvement of certain mechanisms in fusion module design, such as query decomposition and image scene graph, makes the models easily overfit to datasets with specific scenarios, and limits the plenitudinous interaction between the visual-linguistic context. To avoid this caveat, we propose to establish the multi-modal correspondence by leveraging transformers, and empirically show that the complex fusion modules e.g., modular attention network, dynamic graph, and multi-modal tree) can be replaced by a simple stack of transformer encoder layers with higher performance. Moreover, we re-formulate the visual grounding as a direct coordinates regression problem and avoid making predictions out of a set of candidates i.e., region proposals or anchor boxes). Extensive experiments are conducted on five widely used datasets, and a series of state-of-the-art records are set by our TransVG. We build the benchmark of transformer-based visual grounding framework and make the code available at \url{https://github.com/djiajunustc/TransVG}.