Maximal Matching Matters: Preventing Representation Collapse for Robust Cross-Modal Retrieval

TL;DR

This paper introduces a set-based approach with maximal pair assignment similarity and novel loss functions to improve cross-modal image-text retrieval, effectively capturing diverse relationships and preventing set collapse.

Contribution

It proposes a new set-based representation method with specialized loss functions to enhance semantic diversity and prevent collapse in cross-modal retrieval.

Findings

Achieves state-of-the-art results on MS-COCO and Flickr30k datasets.

Effectively captures diverse cross-modal relationships.

Prevents set collapse and enhances semantic richness.

Abstract

Cross-modal image-text retrieval is challenging because of the diverse possible associations between content from different modalities. Traditional methods learn a single-vector embedding to represent semantics of each sample, but struggle to capture nuanced and diverse relationships that can exist across modalities. Set-based approaches, which represent each sample with multiple embeddings, offer a promising alternative, as they can capture richer and more diverse relationships. In this paper, we show that, despite their promise, these set-based representations continue to face issues including sparse supervision and set collapse, which limits their effectiveness. To address these challenges, we propose Maximal Pair Assignment Similarity to optimize one-to-one matching between embedding sets which preserve semantic diversity within the set. We also introduce two loss functions to further enhance the representations: Global Discriminative Loss to enhance distinction among embeddings, and Intra-Set Divergence Loss to prevent collapse within each set. Our method achieves state-of-the-art performance on MS-COCO and Flickr30k without relying on external data.