Larger-Scale Transformers for Multilingual Masked Language Modeling

TL;DR

This paper introduces two larger multilingual masked language models, XLM-R XL and XXL, which outperform previous models on cross-lingual and English benchmarks, demonstrating the benefits of increased capacity for multilingual understanding.

Contribution

The paper presents two significantly larger multilingual models, XLM-R XL and XXL, achieving improved performance across multiple benchmarks and languages compared to prior models.

Findings

XLM-R XL and XXL outperform previous models on XNLI by 1.8% and 2.4%.

Models handle 99 more languages than previous models.

Larger capacity models improve performance on both high-resource and low-resource languages.

Abstract

Recent work has demonstrated the effectiveness of cross-lingual language model pretraining for cross-lingual understanding. In this study, we present the results of two larger multilingual masked language models, with 3.5B and 10.7B parameters. Our two new models dubbed XLM-R XL and XLM-R XXL outperform XLM-R by 1.8% and 2.4% average accuracy on XNLI. Our model also outperforms the RoBERTa-Large model on several English tasks of the GLUE benchmark by 0.3% on average while handling 99 more languages. This suggests pretrained models with larger capacity may obtain both strong performance on high-resource languages while greatly improving low-resource languages. We make our code and models publicly available.