Dual-view Curricular Optimal Transport for Cross-lingual Cross-modal Retrieval

TL;DR

This paper introduces Dual-view Curricular Optimal Transport (DCOT), a novel method for cross-lingual cross-modal retrieval that effectively handles noisy pseudo-parallel data using optimal transport and curriculum learning, improving robustness and generalization.

Contribution

The paper proposes a dual-view optimal transport framework with curriculum learning to better model noisy cross-lingual and cross-modal correspondences in retrieval tasks.

Findings

DCOT outperforms baseline methods on multilingual image-text and video-text datasets.

The approach demonstrates robustness to noisy pseudo-parallel data.

It generalizes well to out-of-domain data.

Abstract

Current research on cross-modal retrieval is mostly English-oriented, as the availability of a large number of English-oriented human-labeled vision-language corpora. In order to break the limit of non-English labeled data, cross-lingual cross-modal retrieval (CCR) has attracted increasing attention. Most CCR methods construct pseudo-parallel vision-language corpora via Machine Translation (MT) to achieve cross-lingual transfer. However, the translated sentences from MT are generally imperfect in describing the corresponding visual contents. Improperly assuming the pseudo-parallel data are correctly correlated will make the networks overfit to the noisy correspondence. Therefore, we propose Dual-view Curricular Optimal Transport (DCOT) to learn with noisy correspondence in CCR. In particular, we quantify the confidence of the sample pair correlation with optimal transport theory from both the cross-lingual and cross-modal views, and design dual-view curriculum learning to dynamically model the transportation costs according to the learning stage of the two views. Extensive experiments are conducted on two multilingual image-text datasets and one video-text dataset, and the results demonstrate the effectiveness and robustness of the proposed method. Besides, our proposed method also shows a good expansibility to cross-lingual image-text baselines and a decent generalization on out-of-domain data.