Zero-shot Cross-lingual Transfer without Parallel Corpus

TL;DR

This paper introduces a novel zero-shot cross-lingual transfer method that leverages a pre-trained model with bilingual alignment and self-training, achieving state-of-the-art results without relying on parallel corpora or translation models.

Contribution

It proposes a new approach combining bilingual task fitting and self-training modules for effective zero-shot transfer in low-resource languages.

Findings

Achieved new state-of-the-art results on multiple NLP tasks.

No dependency on parallel corpus or translation models.

Effective transfer in low-resource language scenarios.

Abstract

Recently, although pre-trained language models have achieved great success on multilingual NLP (Natural Language Processing) tasks, the lack of training data on many tasks in low-resource languages still limits their performance. One effective way of solving that problem is to transfer knowledge from rich-resource languages to low-resource languages. However, many previous works on cross-lingual transfer rely heavily on the parallel corpus or translation models, which are often difficult to obtain. We propose a novel approach to conduct zero-shot cross-lingual transfer with a pre-trained model. It consists of a Bilingual Task Fitting module that applies task-related bilingual information alignment; a self-training module generates pseudo soft and hard labels for unlabeled data and utilizes them to conduct self-training. We got the new SOTA on different tasks without any dependencies on the parallel corpus or translation models.