ReCon: Enhancing True Correspondence Discrimination through Relation Consistency for Robust Noisy Correspondence Learning

TL;DR

ReCon introduces a relation consistency learning framework that improves true correspondence discrimination in multimodal datasets by enforcing dual relation constraints, effectively reducing mismatched pair errors and enhancing robustness.

Contribution

The paper proposes a novel relation consistency learning framework, ReCon, which leverages dual relation constraints to improve true correspondence discrimination in noisy multimodal data.

Findings

ReCon outperforms state-of-the-art methods on benchmark datasets.

Dual relation constraints significantly improve matching accuracy.

ReCon effectively filters out mismatched pairs in multimodal datasets.

Abstract

Can we accurately identify the true correspondences from multimodal datasets containing mismatched data pairs? Existing methods primarily emphasize the similarity matching between the representations of objects across modalities, potentially neglecting the crucial relation consistency within modalities that are particularly important for distinguishing the true and false correspondences. Such an omission often runs the risk of misidentifying negatives as positives, thus leading to unanticipated performance degradation. To address this problem, we propose a general Relation Consistency learning framework, namely ReCon, to accurately discriminate the true correspondences among the multimodal data and thus effectively mitigate the adverse impact caused by mismatches. Specifically, ReCon leverages a novel relation consistency learning to ensure the dual-alignment, respectively of, the cross-modal relation consistency between different modalities and the intra-modal relation consistency within modalities. Thanks to such dual constrains on relations, ReCon significantly enhances its effectiveness for true correspondence discrimination and therefore reliably filters out the mismatched pairs to mitigate the risks of wrong supervisions. Extensive experiments on three widely-used benchmark datasets, including Flickr30K, MS-COCO, and Conceptual Captions, are conducted to demonstrate the effectiveness and superiority of ReCon compared with other SOTAs. The code is available at: https://github.com/qxzha/ReCon.