How does a Multilingual LM Handle Multiple Languages?

TL;DR

This paper critically evaluates multilingual language models like BLOOM and Qwen2, highlighting their strengths in high-resource languages and limitations with low-resource languages, through semantic, syntactic, and cross-lingual tasks.

Contribution

It provides a comprehensive analysis of MLMs' capabilities and limitations across languages, proposing evaluation methods to improve multilingual NLP models.

Findings

Models perform well on high-resource languages.

Struggle with low-resource language understanding.

Cross-lingual transfer is limited for low-resource languages.

Abstract

Multilingual language models have significantly advanced due to rapid progress in natural language processing. Models like BLOOM 1.7B, trained on diverse multilingual datasets, aim to bridge linguistic gaps. However, their effectiveness in capturing linguistic knowledge, particularly for low-resource languages, remains an open question. This study critically examines MLMs capabilities in multilingual understanding, semantic representation, and cross-lingual knowledge transfer. While these models perform well for high-resource languages, they struggle with less-represented ones. Additionally, traditional evaluation methods often overlook their internal syntactic and semantic encoding. This research addresses key limitations through three objectives. First, it assesses semantic similarity by analyzing multilingual word embeddings for consistency using cosine similarity. Second, it examines BLOOM-1.7B and Qwen2 through Named Entity Recognition and sentence similarity tasks to understand their linguistic structures. Third, it explores cross-lingual knowledge transfer by evaluating generalization from high-resource to low-resource languages in sentiment analysis and text classification. By leveraging linguistic probing, performance metrics, and visualizations, this study provides insights into the strengths and limitations of MLMs. The findings aim to enhance multilingual NLP models, ensuring better support for both high- and low-resource languages, thereby promoting inclusivity in language technologies.