The Less the Merrier? Investigating Language Representation in Multilingual Models

TL;DR

This paper investigates how multilingual models represent different languages, especially low-resource ones, revealing community-centered models perform better for certain languages and offering insights to improve multilingual NLP.

Contribution

It provides an analysis of language support and representation in multilingual models, highlighting the benefits of community-centered approaches for low-resource languages.

Findings

Community-centered models outperform others for low-resource languages.

Models' representations vary across language families and dialects.

Performance on downstream tasks depends on language support and representation.

Abstract

Multilingual Language Models offer a way to incorporate multiple languages in one model and utilize cross-language transfer learning to improve performance for different Natural Language Processing (NLP) tasks. Despite progress in multilingual models, not all languages are supported as well, particularly in low-resource settings. In this work, we investigate the linguistic representation of different languages in multilingual models. We start by asking the question which languages are supported in popular multilingual models and which languages are left behind. Then, for included languages, we look at models' learned representations based on language family and dialect and try to understand how models' learned representations for~(1) seen and~(2) unseen languages vary across different language groups. In addition, we test and analyze performance on downstream tasks such as text generation and Named Entity Recognition. We observe from our experiments that community-centered models -- models that focus on languages of a given family or geographical location and are built by communities who speak them -- perform better at distinguishing between languages in the same family for low-resource languages. Our paper contributes to the literature in understanding multilingual models and their shortcomings and offers insights on potential ways to improve them.