Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval

TL;DR

This paper introduces L2RM, a novel framework using Optimal Transport to rematch mismatched pairs in cross-modal retrieval, thereby improving robustness against noisy data and enhancing retrieval accuracy.

Contribution

L2RM is the first to leverage semantic similarity among unpaired samples for rematching mismatched pairs using a partial OT approach with a self-supervised cost function.

Findings

L2RM significantly improves robustness against mismatched pairs.

The method outperforms existing models on three benchmark datasets.

Rematching reduces the negative impact of noisy data in cross-modal retrieval.

Abstract

Collecting well-matched multimedia datasets is crucial for training cross-modal retrieval models. However, in real-world scenarios, massive multimodal data are harvested from the Internet, which inevitably contains Partially Mismatched Pairs (PMPs). Undoubtedly, such semantical irrelevant data will remarkably harm the cross-modal retrieval performance. Previous efforts tend to mitigate this problem by estimating a soft correspondence to down-weight the contribution of PMPs. In this paper, we aim to address this challenge from a new perspective: the potential semantic similarity among unpaired samples makes it possible to excavate useful knowledge from mismatched pairs. To achieve this, we propose L2RM, a general framework based on Optimal Transport (OT) that learns to rematch mismatched pairs. In detail, L2RM aims to generate refined alignments by seeking a minimal-cost transport plan across different modalities. To formalize the rematching idea in OT, first, we propose a self-supervised cost function that automatically learns from explicit similarity-cost mapping relation. Second, we present to model a partial OT problem while restricting the transport among false positives to further boost refined alignments. Extensive experiments on three benchmarks demonstrate our L2RM significantly improves the robustness against PMPs for existing models. The code is available at https://github.com/hhc1997/L2RM.