PC$^2$: Pseudo-Classification Based Pseudo-Captioning for Noisy Correspondence Learning in Cross-Modal Retrieval

TL;DR

This paper introduces PC$^2$, a novel framework for cross-modal retrieval that uses pseudo-classification and pseudo-captioning to handle noisy data pairs, improving robustness and semantic understanding in noisy correspondence learning.

Contribution

The paper proposes a new PC$^2$ framework combining pseudo-classification and pseudo-captioning to address noisy correspondence in cross-modal retrieval, along with a new real-world NCL dataset called NoW.

Findings

PC$^2$ outperforms existing methods on simulated NCL datasets.

The NoW dataset provides a realistic benchmark for NCL.

Empirical results demonstrate significant improvements in retrieval accuracy.

Abstract

In the realm of cross-modal retrieval, seamlessly integrating diverse modalities within multimedia remains a formidable challenge, especially given the complexities introduced by noisy correspondence learning (NCL). Such noise often stems from mismatched data pairs, which is a significant obstacle distinct from traditional noisy labels. This paper introduces Pseudo-Classification based Pseudo-Captioning (PC$^2$) framework to address this challenge. PC$^2$ offers a threefold strategy: firstly, it establishes an auxiliary "pseudo-classification" task that interprets captions as categorical labels, steering the model to learn image-text semantic similarity through a non-contrastive mechanism. Secondly, unlike prevailing margin-based techniques, capitalizing on PC$^2$'s pseudo-classification capability, we generate pseudo-captions to provide more informative and tangible supervision for each mismatched pair. Thirdly, the oscillation of pseudo-classification is borrowed to assistant the correction of correspondence. In addition to technical contributions, we develop a realistic NCL dataset called Noise of Web (NoW), which could be a new powerful NCL benchmark where noise exists naturally. Empirical evaluations of PC$^2$ showcase marked improvements over existing state-of-the-art robust cross-modal retrieval techniques on both simulated and realistic datasets with various NCL settings. The contributed dataset and source code are released at https://github.com/alipay/PC2-NoiseofWeb.