Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching

TL;DR

This paper introduces a bidirectional cross-modal matching framework for unpaired referring expression grounding, combining top-down and bottom-up strategies with pretrained models to improve performance without paired annotations.

Contribution

The paper proposes a novel BiCM framework with query-aware attention, cross-modal object matching using CLIP, and knowledge adaptation, advancing unpaired referring grounding methods.

Findings

Outperforms previous methods by 6.55% and 9.94% on two datasets.

Introduces a query-aware attention map for top-down guidance.

Leverages CLIP for bottom-up object matching.

Abstract

Referring expression grounding is an important and challenging task in computer vision. To avoid the laborious annotation in conventional referring grounding, unpaired referring grounding is introduced, where the training data only contains a number of images and queries without correspondences. The few existing solutions to unpaired referring grounding are still preliminary, due to the challenges of learning image-text matching and lack of the top-down guidance with unpaired data. In this paper, we propose a novel bidirectional cross-modal matching (BiCM) framework to address these challenges. Particularly, we design a query-aware attention map (QAM) module that introduces top-down perspective via generating query-specific visual attention maps. A cross-modal object matching (COM) module is further introduced, which exploits the recently emerged image-text matching pretrained model, CLIP, to predict the target objects from a bottom-up perspective. The top-down and bottom-up predictions are then integrated via a similarity funsion (SF) module. We also propose a knowledge adaptation matching (KAM) module that leverages unpaired training data to adapt pretrained knowledge to the target dataset and task. Experiments show that our framework outperforms previous works by 6.55% and 9.94% on two popular grounding datasets.