Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models

TL;DR

This paper presents ColBERT-X, a cross-language dense retrieval model leveraging XLM-R, trained via zero-shot and translate-train methods, significantly outperforming traditional lexical baselines in multilingual document ranking tasks.

Contribution

It introduces ColBERT-X, a novel cross-language dense retrieval model using transformer encoders, and demonstrates effective training strategies for multilingual information retrieval.

Findings

Significant improvements over lexical CLIR baselines.

Effective zero-shot and translate-train training methods.

Statistically significant results across multiple languages.

Abstract

The advent of transformer-based models such as BERT has led to the rise of neural ranking models. These models have improved the effectiveness of retrieval systems well beyond that of lexical term matching models such as BM25. While monolingual retrieval tasks have benefited from large-scale training collections such as MS MARCO and advances in neural architectures, cross-language retrieval tasks have fallen behind these advancements. This paper introduces ColBERT-X, a generalization of the ColBERT multi-representation dense retrieval model that uses the XLM-RoBERTa (XLM-R) encoder to support cross-language information retrieval (CLIR). ColBERT-X can be trained in two ways. In zero-shot training, the system is trained on the English MS MARCO collection, relying on the XLM-R encoder for cross-language mappings. In translate-train, the system is trained on the MS MARCO English queries coupled with machine translations of the associated MS MARCO passages. Results on ad hoc document ranking tasks in several languages demonstrate substantial and statistically significant improvements of these trained dense retrieval models over traditional lexical CLIR baselines.