Self-paced Multi-grained Cross-modal Interaction Modeling for Referring Expression Comprehension

TL;DR

This paper introduces a self-paced, multi-grained cross-modal interaction framework for referring expression comprehension, leveraging transformer-based attention and adaptive learning to improve localization accuracy across diverse visual and linguistic data.

Contribution

It proposes a novel self-paced learning approach combined with multi-grained cross-modal attention for enhanced reasoning in REC tasks.

Findings

Outperforms state-of-the-art on multiple datasets

Effectively utilizes multi-grained information in visual and linguistic modalities

Adaptive learning improves performance on hard examples

Abstract

As an important and challenging problem in vision-language tasks, referring expression comprehension (REC) generally requires a large amount of multi-grained information of visual and linguistic modalities to realize accurate reasoning. In addition, due to the diversity of visual scenes and the variation of linguistic expressions, some hard examples have much more abundant multi-grained information than others. How to aggregate multi-grained information from different modalities and extract abundant knowledge from hard examples is crucial in the REC task. To address aforementioned challenges, in this paper, we propose a Self-paced Multi-grained Cross-modal Interaction Modeling framework, which improves the language-to-vision localization ability through innovations in network structure and learning mechanism. Concretely, we design a transformer-based multi-grained cross-modal attention, which effectively utilizes the inherent multi-grained information in visual and linguistic encoders. Furthermore, considering the large variance of samples, we propose a self-paced sample informativeness learning to adaptively enhance the network learning for samples containing abundant multi-grained information. The proposed framework significantly outperforms state-of-the-art methods on widely used datasets, such as RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame datasets, demonstrating the effectiveness of our method.