TL;DR
XeroAlign is a simple zero-shot alignment method for cross-lingual transformers that improves multilingual NLP performance by encouraging similar embeddings across languages using translated data.
Contribution
It introduces XeroAlign, a novel task-specific alignment technique for cross-lingual transformers like XLM-R, achieving state-of-the-art zero-shot results.
Findings
XLM-RA outperforms baseline models on multilingual NLP tasks.
XLM-RA surpasses XLM-R trained with labeled data in text classification.
XLM-RA matches state-of-the-art on cross-lingual paraphrasing.
Abstract
The introduction of pretrained cross-lingual language models brought decisive improvements to multilingual NLP tasks. However, the lack of labelled task data necessitates a variety of methods aiming to close the gap to high-resource languages. Zero-shot methods in particular, often use translated task data as a training signal to bridge the performance gap between the source and target language(s). We introduce XeroAlign, a simple method for task-specific alignment of cross-lingual pretrained transformers such as XLM-R. XeroAlign uses translated task data to encourage the model to generate similar sentence embeddings for different languages. The XeroAligned XLM-R, called XLM-RA, shows strong improvements over the baseline models to achieve state-of-the-art zero-shot results on three multilingual natural language understanding tasks. XLM-RA's text classification accuracy exceeds that of…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsXLM-R
