Multilingual Encoder Knows more than You Realize: Shared Weights Pretraining for Extremely Low-Resource Languages

TL;DR

This paper introduces a novel framework that reuses weights between encoder and decoder in multilingual models, significantly improving text generation for extremely low-resource languages by leveraging shared semantic spaces.

Contribution

The paper proposes a shared weights pretraining framework for multilingual models, enabling effective low-resource language generation and demonstrating its success on Chinese minority languages.

Findings

XLM-SWCM outperforms larger models on downstream tasks.

Shared weights improve low-resource language generalization.

Framework is effective across multiple low-resource languages.

Abstract

While multilingual language models like XLM-R have advanced multilingualism in NLP, they still perform poorly in extremely low-resource languages. This situation is exacerbated by the fact that modern LLMs such as LLaMA and Qwen support far fewer languages than XLM-R, making text generation models non-existent for many languages in the world. To tackle this challenge, we propose a novel framework for adapting multilingual encoders to text generation in extremely low-resource languages. By reusing the weights between the encoder and the decoder, our framework allows the model to leverage the learned semantic space of the encoder, enabling efficient learning and effective generalization in low-resource languages. Applying this framework to four Chinese minority languages, we present XLM-SWCM, and demonstrate its superior performance on various downstream tasks even when compared with much larger models.