Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation

TL;DR

This paper introduces OPTICAL, a novel optimal transport-based distillation method that improves cross-lingual retrieval for low-resource languages by leveraging monolingual models and minimal data.

Contribution

The paper proposes a new distillation approach using optimal transport to transfer knowledge from high-resource to low-resource languages in cross-lingual retrieval.

Findings

OPTICAL outperforms strong baselines on low-resource languages.

The method requires only minimal bitext data for training.

Significant improvements are achieved in low-resource language retrieval tasks.

Abstract

Benefiting from transformer-based pre-trained language models, neural ranking models have made significant progress. More recently, the advent of multilingual pre-trained language models provides great support for designing neural cross-lingual retrieval models. However, due to unbalanced pre-training data in different languages, multilingual language models have already shown a performance gap between high and low-resource languages in many downstream tasks. And cross-lingual retrieval models built on such pre-trained models can inherit language bias, leading to suboptimal result for low-resource languages. Moreover, unlike the English-to-English retrieval task, where large-scale training collections for document ranking such as MS MARCO are available, the lack of cross-lingual retrieval data for low-resource language makes it more challenging for training cross-lingual retrieval models. In this work, we propose OPTICAL: Optimal Transport distillation for low-resource Cross-lingual information retrieval. To transfer a model from high to low resource languages, OPTICAL forms the cross-lingual token alignment task as an optimal transport problem to learn from a well-trained monolingual retrieval model. By separating the cross-lingual knowledge from knowledge of query document matching, OPTICAL only needs bitext data for distillation training, which is more feasible for low-resource languages. Experimental results show that, with minimal training data, OPTICAL significantly outperforms strong baselines on low-resource languages, including neural machine translation.