Can Monolingual Pretrained Models Help Cross-Lingual Classification?

TL;DR

This paper explores methods to enhance zero-shot cross-lingual classification by transferring knowledge from monolingual pretrained models to multilingual models, outperforming standard multilingual fine-tuning on benchmark tasks.

Contribution

The paper introduces two novel approaches for knowledge transfer from monolingual to multilingual models to improve cross-lingual classification performance.

Findings

Our methods outperform vanilla multilingual fine-tuning on benchmarks.

Knowledge transfer from monolingual models enhances zero-shot cross-lingual transfer.

Experimental results demonstrate significant improvements over baseline methods.

Abstract

Multilingual pretrained language models (such as multilingual BERT) have achieved impressive results for cross-lingual transfer. However, due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. In this work, we present two approaches to improve zero-shot cross-lingual classification, by transferring the knowledge from monolingual pretrained models to multilingual ones. Experimental results on two cross-lingual classification benchmarks show that our methods outperform vanilla multilingual fine-tuning.