Language Agnostic Multilingual Information Retrieval with Contrastive Learning

TL;DR

This paper introduces a contrastive learning approach to enhance multilingual information retrieval by leveraging parallel and non-parallel corpora, achieving strong zero-shot performance across languages with less computational cost.

Contribution

It proposes a novel semantic and language contrastive loss framework that improves cross-lingual transfer in multilingual IR using limited parallel data.

Findings

Significant improvement over prior models in zero-shot multilingual retrieval.

Effective with few parallel sentences, suitable for low-resource languages.

Compatible as an add-on to existing models and tasks.

Abstract

Multilingual information retrieval (IR) is challenging since annotated training data is costly to obtain in many languages. We present an effective method to train multilingual IR systems when only English IR training data and some parallel corpora between English and other languages are available. We leverage parallel and non-parallel corpora to improve the pretrained multilingual language models' cross-lingual transfer ability. We design a semantic contrastive loss to align representations of parallel sentences that share the same semantics in different languages, and a new language contrastive loss to leverage parallel sentence pairs to remove language-specific information in sentence representations from non-parallel corpora. When trained on English IR data with these losses and evaluated zero-shot on non-English data, our model demonstrates significant improvement to prior work on retrieval performance, while it requires much less computational effort. We also demonstrate the value of our model for a practical setting when a parallel corpus is only available for a few languages, but a lack of parallel corpora resources persists for many other low-resource languages. Our model can work well even with a small number of parallel sentences, and be used as an add-on module to any backbones and other tasks.