Descriptive Image-Text Matching with Graded Contextual Similarity

TL;DR

This paper introduces DITM, a novel image-text matching approach that models graded contextual similarity using sentence descriptiveness, enabling more flexible and accurate matching beyond binary supervision.

Contribution

The work proposes a new method leveraging sentence descriptiveness scores to improve image-text matching, addressing the limitations of binary supervision and capturing many-to-many relationships.

Findings

Outperforms state-of-the-art on MS-COCO, Flickr30K, and CxC datasets.

Enhances hierarchical reasoning capabilities of the model.

Effectively models complex image-text relationships with graded similarity.

Abstract

Image-text matching aims to build correspondences between visual and textual data by learning their pairwise similarities. Most existing approaches have adopted sparse binary supervision, indicating whether a pair of images and sentences matches or not. However, such sparse supervision covers a limited subset of image-text relationships, neglecting their inherent many-to-many correspondences; an image can be described in numerous texts at different descriptive levels. Moreover, existing approaches overlook the implicit connections from general to specific descriptions, which form the underlying rationale for the many-to-many relationships between vision and language. In this work, we propose descriptive image-text matching, called DITM, to learn the graded contextual similarity between image and text by exploring the descriptive flexibility of language. We formulate the descriptiveness score of each sentence with cumulative term frequency-inverse document frequency (TF-IDF) to balance the pairwise similarity according to the keywords in the sentence. Our method leverages sentence descriptiveness to learn robust image-text matching in two key ways: (1) to refine the false negative labeling, dynamically relaxing the connectivity between positive and negative pairs, and (2) to build more precise matching, aligning a set of relevant sentences in a generic-to-specific order. By moving beyond rigid binary supervision, DITM enhances the discovery of both optimal matches and potential positive pairs. Extensive experiments on MS-COCO, Flickr30K, and CxC datasets demonstrate the effectiveness of our method in representing complex image-text relationships compared to state-of-the-art approaches. In addition, DITM enhances the hierarchical reasoning ability of the model, supported by the extensive analysis on HierarCaps benchmark.