Self-Augmentation Improves Zero-Shot Cross-Lingual Transfer

TL;DR

The paper introduces SALT, a simple self-augmentation method using code-switching and embedding mixup to enhance zero-shot cross-lingual transfer in multilingual models without external data.

Contribution

It presents SALT, a novel self-augmentation technique that improves cross-lingual transferability of multilingual models without relying on external alignment resources.

Findings

Improves zero-shot transfer on XNLI and PAWS-X datasets.

Enhances transferability without external data.

Effective distillation of cross-lingual knowledge.

Abstract

Zero-shot cross-lingual transfer is a central task in multilingual NLP, allowing models trained in languages with more sufficient training resources to generalize to other low-resource languages. Earlier efforts on this task use parallel corpora, bilingual dictionaries, or other annotated alignment data to improve cross-lingual transferability, which are typically expensive to obtain. In this paper, we propose a simple yet effective method, SALT, to improve the zero-shot cross-lingual transfer of the multilingual pretrained language models without the help of such external data. By incorporating code-switching and embedding mixup with self-augmentation, SALT effectively distills cross-lingual knowledge from the multilingual PLM and enhances its transferability on downstream tasks. Experimental results on XNLI and PAWS-X show that our method is able to improve zero-shot cross-lingual transferability without external data. Our code is available at https://github.com/luka-group/SALT.