Parameter-efficient Zero-shot Transfer for Cross-Language Dense Retrieval with Adapters

TL;DR

This paper introduces a parameter-efficient method using adapters for zero-shot cross-language dense retrieval, demonstrating improved transfer from monolingual models and analyzing the limitations of language adapter replacement at inference.

Contribution

It proposes a novel adapter-based transfer approach for cross-language retrieval, showing it outperforms full fine-tuning and analyzing the inefficacy of language adapter replacement during inference.

Findings

Adapters improve cross-language transfer in dense retrieval.

Replacing language adapters at inference is suboptimal for retrieval.

Monolingual models with adapters outperform full fine-tuning in CLIR.

Abstract

A popular approach to creating a zero-shot cross-language retrieval model is to substitute a monolingual pretrained language model in the retrieval model with a multilingual pretrained language model such as Multilingual BERT. This multilingual model is fined-tuned to the retrieval task with monolingual data such as English MS MARCO using the same training recipe as the monolingual retrieval model used. However, such transferred models suffer from mismatches in the languages of the input text during training and inference. In this work, we propose transferring monolingual retrieval models using adapters, a parameter-efficient component for a transformer network. By adding adapters pretrained on language tasks for a specific language with task-specific adapters, prior work has shown that the adapter-enhanced models perform better than fine-tuning the entire model when transferring across languages in various NLP tasks. By constructing dense retrieval models with adapters, we show that models trained with monolingual data are more effective than fine-tuning the entire model when transferring to a Cross Language Information Retrieval (CLIR) setting. However, we found that the prior suggestion of replacing the language adapters to match the target language at inference time is suboptimal for dense retrieval models. We provide an in-depth analysis of this discrepancy between other cross-language NLP tasks and CLIR.