Soft Language Prompts for Language Transfer

TL;DR

This paper introduces soft language prompts for cross-lingual transfer in NLP, demonstrating that combining soft prompts with task adapters often yields better results than other methods across multiple languages.

Contribution

It is the first to apply soft prompts for language transfer and systematically compares their effectiveness with adapters in multilingual NLP tasks.

Findings

Soft language prompts outperform many configurations in cross-lingual transfer.

Combining soft prompts with task adapters often yields superior results.

A comprehensive analysis across 16 languages, including low-resource ones.

Abstract

Cross-lingual knowledge transfer, especially between high- and low-resource languages, remains challenging in natural language processing (NLP). This study offers insights for improving cross-lingual NLP applications through the combination of parameter-efficient fine-tuning methods. We systematically explore strategies for enhancing cross-lingual transfer through the incorporation of language-specific and task-specific adapters and soft prompts. We present a detailed investigation of various combinations of these methods, exploring their efficiency across 16 languages, focusing on 10 mid- and low-resource languages. We further present to our knowledge the first use of soft prompts for language transfer, a technique we call soft language prompts. Our findings demonstrate that in contrast to claims of previous work, a combination of language and task adapters does not always work best; instead, combining a soft language prompt with a task adapter outperforms most configurations in many cases.