Neighbor-aware Instance Refining with Noisy Labels for Cross-Modal Retrieval

TL;DR

This paper introduces NIRNL, a novel framework for cross-modal retrieval that effectively handles noisy labels by neighbor-aware instance refining, improving robustness and performance on benchmark datasets.

Contribution

The paper proposes a new robust learning framework, NIRNL, combining CMP and neighbor-aware instance refining to better utilize data and mitigate noise in cross-modal retrieval.

Findings

Achieves state-of-the-art results on benchmark datasets.

Demonstrates robustness under high noise rates.

Effectively partitions data into pure, hard, and noisy subsets.

Abstract

In recent years, Cross-Modal Retrieval (CMR) has made significant progress in the field of multi-modal analysis. However, since it is time-consuming and labor-intensive to collect large-scale and well-annotated data, the annotation of multi-modal data inevitably contains some noise. This will degrade the retrieval performance of the model. To tackle the problem, numerous robust CMR methods have been developed, including robust learning paradigms, label calibration strategies, and instance selection mechanisms. Unfortunately, they often fail to simultaneously satisfy model performance ceilings, calibration reliability, and data utilization rate. To overcome the limitations, we propose a novel robust cross-modal learning framework, namely Neighbor-aware Instance Refining with Noisy Labels (NIRNL). Specifically, we first propose Cross-modal Margin Preserving (CMP) to adjust the relative distance between positive and negative pairs, thereby enhancing the discrimination between sample pairs. Then, we propose Neighbor-aware Instance Refining (NIR) to identify pure subset, hard subset, and noisy subset through cross-modal neighborhood consensus. Afterward, we construct different tailored optimization strategies for this fine-grained partitioning, thereby maximizing the utilization of all available data while mitigating error propagation. Extensive experiments on three benchmark datasets demonstrate that NIRNL achieves state-of-the-art performance, exhibiting remarkable robustness, especially under high noise rates.