Robust Noisy Correspondence Learning via Self-Drop and Dual-Weight

TL;DR

This paper introduces a novel self-drop and dual-weight method for robustly learning from noisy cross-modal data, effectively reducing the impact of noisy pairs and emphasizing significant clean samples, leading to improved stability and performance.

Contribution

The paper proposes a new data partitioning and weighting strategy that enhances robustness against noisy correspondences in cross-modal learning tasks.

Findings

The approach outperforms prior methods on noisy datasets.

It maintains stable performance under high noise ratios.

Effective in vision-language pre-training scenarios.

Abstract

Many researchers collect data from the internet through crowd-sourcing or web crawling to alleviate the data-hungry challenge associated with cross-modal matching. Although such practice does not require expensive annotations, it inevitably introduces mismatched pairs and results in a noisy correspondence problem. Current approaches leverage the memorization effect of deep neural networks to distinguish noise and perform re-weighting. However, briefly lowering the weight of noisy pairs cannot eliminate the negative impact of noisy correspondence in the training process. In this paper, we propose a novel self-drop and dual-weight approach, which achieves elaborate data processing by qua-partitioning the data. Specifically, our approach partitions all data into four types: clean and significant, clean yet insignificant, vague, and noisy. We analyze the effect of noisy and clean data pairs and find that for vision-language pre-training models, a small number of clean samples is more valuable than a majority of noisy ones. Based on this observation, we employ self-drop to discard noisy samples to effectively mitigate the impact of noise. In addition, we adopt a dual-weight strategy to ensure that the model focuses more on significant samples while appropriately leveraging vague samples. Compared to the prior works, our approach is more robust and demonstrates relatively more stable performance on noisy datasets, especially under a high noise ratio. Extensive experiments on three widely used datasets, including Flickr30K, MS-COCO, and Conceptual Captions, validate the effectiveness of our approach.