Hire: Hybrid-modal Interaction with Multiple Relational Enhancements for Image-Text Matching

TL;DR

This paper introduces Hire, a hybrid-modal interaction framework with relational enhancements for image-text matching, leveraging explicit and implicit relationship modeling to improve contextual understanding and achieve state-of-the-art results.

Contribution

The paper proposes a novel hybrid-modal interaction method with multiple relational enhancements, combining explicit spatial-semantic graph reasoning and implicit relationship modeling for improved image-text matching.

Findings

Achieves new state-of-the-art results on MS-COCO and Flickr30K datasets.

Demonstrates the effectiveness of explicit and implicit relationship modeling in image-text matching.

Improves contextual object representation through graph-based reasoning and cross-modal attention.

Abstract

Image-text matching (ITM) is a fundamental problem in computer vision. The key issue lies in jointly learning the visual and textual representation to estimate their similarity accurately. Most existing methods focus on feature enhancement within modality or feature interaction across modalities, which, however, neglects the contextual information of the object representation based on the inter-object relationships that match the corresponding sentences with rich contextual semantics. In this paper, we propose a Hybrid-modal Interaction with multiple Relational Enhancements (termed \textit{Hire}) for image-text matching, which correlates the intra- and inter-modal semantics between objects and words with implicit and explicit relationship modelling. In particular, the explicit intra-modal spatial-semantic graph-based reasoning network is designed to improve the contextual representation of visual objects with salient spatial and semantic relational connectivities, guided by the explicit relationships of the objects' spatial positions and their scene graph. We use implicit relationship modelling for potential relationship interactions before explicit modelling to improve the fault tolerance of explicit relationship detection. Then the visual and textual semantic representations are refined jointly via inter-modal interactive attention and cross-modal alignment. To correlate the context of objects with the textual context, we further refine the visual semantic representation via cross-level object-sentence and word-image-based interactive attention. Extensive experiments validate that the proposed hybrid-modal interaction with implicit and explicit modelling is more beneficial for image-text matching. And the proposed \textit{Hire} obtains new state-of-the-art results on MS-COCO and Flickr30K benchmarks.