Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages

TL;DR

This paper introduces PARC, a retrieval-augmented prompting method that enhances zero-shot cross-lingual transfer for low-resource languages by leveraging semantically similar high-resource language sentences, improving performance across multiple tasks.

Contribution

The paper proposes the PARC pipeline, a novel retrieval-augmented prompting approach that significantly boosts zero-shot performance on low-resource languages in multilingual models.

Findings

PARC improves zero-shot performance by up to 16.3% on low-resource languages.

PARC outperforms finetuning baseline by 3.7%.

Performance correlates with language similarity and pretraining data amount.

Abstract

Multilingual Pretrained Language Models (MPLMs) have shown their strong multilinguality in recent empirical cross-lingual transfer studies. In this paper, we propose the Prompts Augmented by Retrieval Crosslingually (PARC) pipeline to improve the zero-shot performance on low-resource languages (LRLs) by augmenting the context with semantically similar sentences retrieved from a high-resource language (HRL) as prompts. PARC improves the zero-shot performance on three downstream tasks (binary sentiment classification, topic categorization and natural language inference) with multilingual parallel test sets across 10 LRLs covering 6 language families in both unlabeled settings (+5.1%) and labeled settings (+16.3%). PARC-labeled also outperforms the finetuning baseline by 3.7%. We find a significant positive correlation between cross-lingual transfer performance on one side, and the similarity between the high- and low-resource languages as well as the amount of low-resource pretraining data on the other side. A robustness analysis suggests that PARC has the potential to achieve even stronger performance with more powerful MPLMs.