Referencing Where to Focus: Improving VisualGrounding with Referential Query

TL;DR

This paper introduces RefFormer, a novel visual grounding method that improves target localization by using referential queries generated through a query adaption module, enhancing learning efficiency and accuracy.

Contribution

The paper proposes a query adaption module integrated with CLIP to generate referential queries, reducing learning difficulty and leveraging multi-level image features for better grounding performance.

Findings

Outperforms state-of-the-art on five benchmarks.

Effectively mitigates learning difficulty in query generation.

Preserves CLIP knowledge without backbone tuning.

Abstract

Visual Grounding aims to localize the referring object in an image given a natural language expression. Recent advancements in DETR-based visual grounding methods have attracted considerable attention, as they directly predict the coordinates of the target object without relying on additional efforts, such as pre-generated proposal candidates or pre-defined anchor boxes. However, existing research primarily focuses on designing stronger multi-modal decoder, which typically generates learnable queries by random initialization or by using linguistic embeddings. This vanilla query generation approach inevitably increases the learning difficulty for the model, as it does not involve any target-related information at the beginning of decoding. Furthermore, they only use the deepest image feature during the query learning process, overlooking the importance of features from other levels. To address these issues, we propose a novel approach, called RefFormer. It consists of the query adaption module that can be seamlessly integrated into CLIP and generate the referential query to provide the prior context for decoder, along with a task-specific decoder. By incorporating the referential query into the decoder, we can effectively mitigate the learning difficulty of the decoder, and accurately concentrate on the target object. Additionally, our proposed query adaption module can also act as an adapter, preserving the rich knowledge within CLIP without the need to tune the parameters of the backbone network. Extensive experiments demonstrate the effectiveness and efficiency of our proposed method, outperforming state-of-the-art approaches on five visual grounding benchmarks.