Soft Language Prompts for Language Transfer
Ivan Vykopal, Simon Ostermann, Mari\'an \v{S}imko

TL;DR
This paper introduces soft language prompts for cross-lingual transfer in NLP, demonstrating that combining soft prompts with task adapters often yields better results than other methods across multiple languages.
Contribution
It is the first to apply soft prompts for language transfer and systematically compares their effectiveness with adapters in multilingual NLP tasks.
Findings
Soft language prompts outperform many configurations in cross-lingual transfer.
Combining soft prompts with task adapters often yields superior results.
A comprehensive analysis across 16 languages, including low-resource ones.
Abstract
Cross-lingual knowledge transfer, especially between high- and low-resource languages, remains challenging in natural language processing (NLP). This study offers insights for improving cross-lingual NLP applications through the combination of parameter-efficient fine-tuning methods. We systematically explore strategies for enhancing cross-lingual transfer through the incorporation of language-specific and task-specific adapters and soft prompts. We present a detailed investigation of various combinations of these methods, exploring their efficiency across 16 languages, focusing on 10 mid- and low-resource languages. We further present to our knowledge the first use of soft prompts for language transfer, a technique we call soft language prompts. Our findings demonstrate that in contrast to claims of previous work, a combination of language and task adapters does not always work best;…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsNatural Language Processing Techniques
MethodsAdapter
