Bridging the Modality Gap: Dimension Information Alignment and Sparse Spatial Constraint for Image-Text Matching

TL;DR

This paper introduces DIAS, a novel method for image-text matching that aligns modality information and applies sparse spatial constraints, significantly improving performance by bridging the modality gap.

Contribution

The paper proposes a new approach called DIAS that aligns cross-modal embeddings dimension-wise and incorporates sparse spatial constraints to enhance image-text matching.

Findings

Achieves 4.3%-10.2% rSum improvements on Flickr30k and MSCOCO.

Effectively bridges the modality gap with dimension alignment and spatial constraints.

Outperforms existing contrastive learning models in image-text matching.

Abstract

Many contrastive learning based models have achieved advanced performance in image-text matching tasks. The key of these models lies in analyzing the correlation between image-text pairs, which involves cross-modal interaction of embeddings in corresponding dimensions. However, the embeddings of different modalities are from different models or modules, and there is a significant modality gap. Directly interacting such embeddings lacks rationality and may capture inaccurate correlation. Therefore, we propose a novel method called DIAS to bridge the modality gap from two aspects: (1) We align the information representation of embeddings from different modalities in corresponding dimension to ensure the correlation calculation is based on interactions of similar information. (2) The spatial constraints of inter- and intra-modalities unmatched pairs are introduced to ensure the effectiveness of semantic alignment of the model. Besides, a sparse correlation algorithm is proposed to select strong correlated spatial relationships, enabling the model to learn more significant features and avoid being misled by weak correlation. Extensive experiments demonstrate the superiority of DIAS, achieving 4.3\%-10.2\% rSum improvements on Flickr30k and MSCOCO benchmarks.