C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval

TL;DR

This paper introduces C3, a method that enhances cross-language ad-hoc retrieval by continued pretraining with contrastive weak supervision using comparable Wikipedia articles, leading to improved retrieval performance.

Contribution

The paper proposes a novel continued pretraining approach leveraging contrastive weak supervision with multilingual Wikipedia data for better cross-language retrieval.

Findings

Improved retrieval effectiveness on cross-language tasks.

Contrastive weak supervision enhances multilingual model pretraining.

Pretraining with Wikipedia articles benefits downstream retrieval performance.

Abstract

Pretrained language models have improved effectiveness on numerous tasks, including ad-hoc retrieval. Recent work has shown that continuing to pretrain a language model with auxiliary objectives before fine-tuning on the retrieval task can further improve retrieval effectiveness. Unlike monolingual retrieval, designing an appropriate auxiliary task for cross-language mappings is challenging. To address this challenge, we use comparable Wikipedia articles in different languages to further pretrain off-the-shelf multilingual pretrained models before fine-tuning on the retrieval task. We show that our approach yields improvements in retrieval effectiveness.