Are Multilingual Models Effective in Code-Switching?

TL;DR

This paper evaluates the effectiveness of multilingual language models in code-switching tasks, analyzing their performance, speed, and parameter efficiency across different language pairs and tasks.

Contribution

It provides an empirical assessment of multilingual models' capabilities in code-switching, highlighting limitations and proposing meta-embeddings as a more efficient alternative.

Findings

Pre-trained multilingual models may not produce high-quality representations for code-switching.

Meta-embeddings achieve comparable performance with fewer parameters.

Multilingual models' effectiveness varies across language pairs and tasks.

Abstract

Multilingual language models have shown decent performance in multilingual and cross-lingual natural language understanding tasks. However, the power of these multilingual models in code-switching tasks has not been fully explored. In this paper, we study the effectiveness of multilingual language models to understand their capability and adaptability to the mixed-language setting by considering the inference speed, performance, and number of parameters to measure their practicality. We conduct experiments in three language pairs on named entity recognition and part-of-speech tagging and compare them with existing methods, such as using bilingual embeddings and multilingual meta-embeddings. Our findings suggest that pre-trained multilingual models do not necessarily guarantee high-quality representations on code-switching, while using meta-embeddings achieves similar results with significantly fewer parameters.