Disentangled Noisy Correspondence Learning

TL;DR

DisNCL is a novel information-theoretic framework that improves cross-modal retrieval by effectively disentangling meaningful content from noise in noisy correspondence data, leading to better alignment and retrieval accuracy.

Contribution

DisNCL introduces a new approach using information bottlenecks for robust feature disentanglement and soft matching for noisy multi-modal data, advancing cross-modal retrieval methods.

Findings

Achieves 2% average recall improvement over baselines.

Learns meaningful modality-invariant and exclusive subspaces.

Demonstrates robustness to noisy correspondences through experiments.

Abstract

Cross-modal retrieval is crucial in understanding latent correspondences across modalities. However, existing methods implicitly assume well-matched training data, which is impractical as real-world data inevitably involves imperfect alignments, i.e., noisy correspondences. Although some works explore similarity-based strategies to address such noise, they suffer from sub-optimal similarity predictions influenced by modality-exclusive information (MEI), e.g., background noise in images and abstract definitions in texts. This issue arises as MEI is not shared across modalities, thus aligning it in training can markedly mislead similarity predictions. Moreover, although intuitive, directly applying previous cross-modal disentanglement methods suffers from limited noise tolerance and disentanglement efficacy. Inspired by the robustness of information bottlenecks against noise, we introduce DisNCL, a novel information-theoretic framework for feature Disentanglement in Noisy Correspondence Learning, to adaptively balance the extraction of MII and MEI with certifiable optimal cross-modal disentanglement efficacy. DisNCL then enhances similarity predictions in modality-invariant subspace, thereby greatly boosting similarity-based alleviation strategy for noisy correspondences. Furthermore, DisNCL introduces soft matching targets to model noisy many-to-many relationships inherent in multi-modal input for noise-robust and accurate cross-modal alignment. Extensive experiments confirm DisNCL's efficacy by 2% average recall improvement. Mutual information estimation and visualization results show that DisNCL learns meaningful MII/MEI subspaces, validating our theoretical analyses.